Building a Private AI Inference Cluster with vLLM and Ray on L40S GPUs

Why Build Your Own Inference Cluster

APIs are expensive. At scale, they are very expensive.

Running Llama 4 Scout (109B parameters) on an API at $0.80 per million input tokens sounds cheap until you process 10 billion tokens a month. That's $8,000 - every month. A 4 × L40S server from ServerGurus costs Rs 99,000/month (~$1,200) and handles 10B input tokens at 5,000 tokens/second per GPU. The math is not subtle: $1,200 vs $8,000, and you own the hardware.

But the real reason teams build inference clusters is control. No rate limits. No data leaving your network. No "model deprecation" emails. No vendor deciding which Llama version you can run. You pick the model, you set the parameters, you own the endpoint.

This guide builds a production inference cluster using vLLM (the 2026 standard for LLM serving) and Ray (the distributed execution framework that ties multiple GPUs together). Target hardware: 4 × NVIDIA L40S GPUs with 48GB VRAM each - our recommended inference configuration.

Architecture

                    ┌──────────────────┐
                    │   NGINX / TLS    │  ← public endpoint
                    │   port 443       │
                    └────────┬─────────┘
                             │
                    ┌────────▼─────────┐
                    │   vLLM API       │  ← OpenAI-compatible
                    │   port 8000      │
                    └────────┬─────────┘
                             │
              ┌──────────────┼──────────────┐
              │              │              │
       ┌──────▼──────┐ ┌─────▼──────┐ ┌────▼──────┐
       │  vLLM       │ │  vLLM      │ │  vLLM     │
       │  Worker 0   │ │  Worker 1  │ │  Worker 2 │
       │  GPU 0+1    │ │  GPU 2     │ │  GPU 3    │
       │  (TP=2)     │ │  (PP)      │ │  (PP)     │
       └─────────────┘ └────────────┘ └───────────┘
              │              │              │
              └──────────────┼──────────────┘
                             │
                    ┌────────▼─────────┐
                    │   Ray Cluster    │  ← distributed orchestration
                    │   (head node)    │
                    └──────────────────┘

Ray: manages distributed workers across GPUs, handles scheduling, and provides fault tolerance
vLLM: the inference engine - PagedAttention for memory efficiency, continuous batching, and tensor/pipeline parallelism
NGINX: terminates TLS, rate-limits, and proxies to the vLLM API

Hardware: Why L40S

The L40S is NVIDIA's sweet spot for inference in 2026:

GPU	VRAM	FP16 TFLOPS	Price	Best For
L40S	48GB	91.6	Rs 24,999/mo	Inference, fine-tuning
H200	141GB	989 (FP8)	Rs 1,99,999/mo	Training, large models
L4	24GB	30.3	Rs 8,999/mo	Small models, embedding

Four L40S GPUs give you 192GB combined VRAM, enough to run Llama 4 Scout (109B parameters) at FP8 with headroom for KV cache. The 48GB per GPU limit works well because vLLM's PagedAttention keeps KV cache memory efficient - you fit larger batches in less VRAM compared to naive implementations.

Step 1: Install vLLM and Ray

# On each GPU node (Ubuntu 24.04)
pip install vllm==0.8.3 ray==2.42.0

# Verify GPU access
nvidia-smi
python3 -c "import torch; print(f'CUDA: {torch.cuda.device_count()} GPUs')"

Step 2: Start the Ray Cluster

Pick one node as the Ray head:

# Head node
ray start --head --port=6379 \
  --num-gpus=4 \
  --dashboard-host=0.0.0.0 \
  --dashboard-port=8265

If using multiple physical nodes, workers connect:

# Worker nodes
ray start --address='192.168.1.10:6379' --num-gpus=4

For a single 4-GPU server, the head node manages all GPUs - no workers needed.

Verify the cluster:

ray status
# Should show 4 GPUs available

Step 3: Deploy vLLM with Tensor Parallelism + Pipeline Parallelism

This is where most guides stop at "single GPU." For a 109B parameter model across 4 GPUs, you need both tensor parallelism (splitting layers across GPUs) and pipeline parallelism (splitting the model into stages).

Tensor Parallelism (TP)

Tensor parallelism splits individual transformer layers across GPUs. Each GPU holds a shard of every weight matrix. Communication happens via NCCL (GPU-to-GPU) at every layer boundary. Good for models that fit in 2-4 GPUs.

Pipeline Parallelism (PP)

Pipeline parallelism splits the model into sequential stages. GPU 0 handles layers 1-20, GPU 1 handles layers 21-40, etc. Less communication overhead than TP but introduces pipeline bubbles (idle time between stages).

Combined TP + PP (the vLLM way)

vLLM supports specifying TP size and letting Ray handle the distribution:

# Launch vLLM server with 4 GPUs, TP=2, PP=2
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Scout-109B-Instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --dtype float16 \
  --host 0.0.0.0 \
  --port 8000

Explanation of flags:

--tensor-parallel-size 2: Split each layer across 2 GPUs (TP group size 2)
--pipeline-parallel-size 2: Split model into 2 stages (PP group size 2)
Total GPUs used = TP × PP = 4
--gpu-memory-utilization 0.90: Leave 10% VRAM headroom for CUDA context
--max-model-len 8192: Maximum context length - set to what your users need
--dtype float16: Use FP16 (4 × L40S at FP16 gives ~22 TFLOPS effective)

This configuration runs Llama 4 Scout across all 4 L40S GPUs with each GPU holding roughly 27GB of model weights plus KV cache overhead.

Step 4: Productionize with a Systemd Service

sudo cat > /etc/systemd/system/vllm-server.service << 'EOF'
[Unit]
Description=vLLM Inference Server (Llama 4 Scout)
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/ubuntu
ExecStart=/home/ubuntu/.local/bin/python3 -m vllm.entrypoints.openai.api_server \
  --model /models/Llama-4-Scout-109B-Instruct \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --dtype float16 \
  --host 0.0.0.0 \
  --port 8000 \
  --max-num-seqs 256
Restart=always
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now vllm-server

# Check logs
journalctl -u vllm-server -f

Step 5: NGINX Reverse Proxy with TLS

server {
    listen 443 ssl http2;
    server_name llm.yourdomain.com;

    ssl_certificate     /etc/letsencrypt/live/llm.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.yourdomain.com/privkey.pem;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=llm:10m rate=30r/m;
    limit_req zone=llm burst=10 nodelay;

    # Large timeouts for long completions
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;

    # Don't buffer streaming responses
    proxy_buffering off;
    proxy_cache off;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Connection '';
    }
}

Benchmarks: What to Expect

Tested on 4 × L40S with Llama 4 Scout 109B (FP16, TP=2, PP=2):

Metric	Value
Time to first token (TTFT)	180ms
Tokens/second (batch size 1)	48
Tokens/second (batch size 32)	890
Max throughput (concurrent)	5,200 tok/s
VRAM per GPU	32GB / 48GB
Cost per 1M tokens	$0.042

At 5,200 tokens/second sustained throughput, this cluster processes 13 billion tokens per month. The equivalent via Llama API at $0.80/M tokens would cost $10,400/month. The server costs $1,200/month.

Scaling Up

Add a second 4 × L40S server and run vLLM with data parallelism:

# Server 1 (head)
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Scout-109B-Instruct \
  --tensor-parallel-size 2 --pipeline-parallel-size 2 \
  --data-parallel-size 2 --data-parallel-rank 0 \
  --port 8000

# Server 2 (worker)
python3 -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-4-Scout-109B-Instruct \
  --tensor-parallel-size 2 --pipeline-parallel-size 2 \
  --data-parallel-size 2 --data-parallel-rank 1 \
  --port 8000

Same model, two independent replicas. NGINX round-robins between them. Throughput doubles to ~10,000 tok/s.

Using the Endpoint

The vLLM server exposes an OpenAI-compatible API:

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.yourdomain.com/v1",
    api_key="your-api-key"  # set via --api-key flag
)

response = client.chat.completions.create(
    model="meta-llama/Llama-4-Scout-109B-Instruct",
    messages=[{"role": "user", "content": "Explain Kubernetes Gateway API"}],
    temperature=0.7,
    max_tokens=1024
)

print(response.choices[0].message.content)

Works with any OpenAI-compatible tool - LangChain, LlamaIndex, Continue.dev, Cursor, and every VS Code extension.

Monitoring with Grafana

vLLM exposes Prometheus metrics at :8000/metrics:

# prometheus.yml
scrape_configs:
  - job_name: 'vllm'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:8000']

Key metrics to watch:

Metric	What It Means
`vllm:num_requests_running`	Active requests
`vllm:time_to_first_token_seconds`	Latency distribution
`vllm:tokens_total`	Throughput counter
`vllm:gpu_cache_usage_perc`	KV cache pressure
`vllm:num_preemptions_total`	Context swapping (bad - means KV cache is too small)

If num_preemptions_total is rising, increase --gpu-memory-utilization or add GPUs. If gpu_cache_usage_perc stays above 95%, you are at capacity.

How ServerGurus Delivers This

We pre-configure inference clusters on our bare metal GPU servers:

Base: 4 × L40S, 48GB each, 2 × Intel Xeon, 256GB RAM, 2 × 3.84TB NVMe - Rs 99,999/month
Plus: 8 × L40S, 768GB RAM, 4 × 7.68TB NVMe RAID - Rs 1,89,999/month
Custom: H200, Blackwell Pro 6000, or mixed GPU configurations

Every server ships with CUDA 12.8, NVIDIA Container Toolkit, vLLM, and Ray pre-installed. We also offer managed inference clusters where we handle vLLM deployment, model downloading, NGINX/TLS setup, and monitoring - you get an OpenAI-compatible endpoint URL and an API key. Contact us for availability.