Disaggregated Inference: Why Splitting Prefill and Decode Is the 2026 Architecture
The Problem: Prefill Punishes Decode
Every LLM inference request has two phases:
- Prefill: Process the entire input prompt in one forward pass. Compute-bound. Uses tensor cores at 100%. Takes 50ms-2s depending on prompt length.
- Decode: Generate tokens one at a time, each requiring the full model weights plus the KV cache. Memory-bandwidth-bound. Tensor cores idle at 15-30%.
On a single GPU running both phases, you get the worst of both worlds:
- During prefill, decode stalls because the GPU is busy processing the prompt.
- During decode, the GPU's compute sits mostly idle waiting on memory bandwidth.
- Long prompts (100K tokens) trigger prefill spikes that cause TTFT latency peaks for everyone else.
A single long prompt from one user delays token generation for every other user sharing the GPU. This is not a bug - it is the physics of running both a compute-heavy and a memory-heavy workload on the same die.
The Solution: Disaggregated Inference
Disaggregated inference splits the two phases across different hardware:
┌────────────────────────────────────────────────────┐
│ Request Router │
│ (sends prefill → compute, decode → memory) │
└──────────┬─────────────────────────┬────────────────┘
│ │
┌──────▼──────┐ ┌──────▼──────┐
│ Prefill │ │ Decode │
│ Nodes │ KV │ Nodes │
│ (H200/B200)│ ──────► │ (L40S) │
│ │ Cache │ │
│ Compute- │ Transfer │ Memory- │
│ optimized │ │ optimized │
└─────────────┘ └─────────────┘
Prefill nodes handle prompt processing. They need high compute throughput - H200s, Blackwell B200s, or any GPU with strong FP8/BF16 tensor cores. These GPUs run at high utilization, process batches of prompts, generate KV caches, and forward them to decode nodes.
Decode nodes handle token generation. They need high memory bandwidth - L40S, H100, or any GPU with fast HBM and enough VRAM for KV cache. These GPUs run at steady state, one token per request per step, no compute spikes.
KV cache transfer happens over RDMA or NVLink/NVSwitch. The prefill node computes the KV cache for the prompt, serializes it, and sends it to the decode node that will continue generating tokens. The transfer adds single-digit milliseconds.
Why This Architecture Matters
1. No Prefill Interference
On a colocated system, a 50K-token prompt causes a 1.2-second prefill that freezes decode for every request on the GPU. With disaggregation, that prefill runs on a dedicated node - decode nodes never see the spike. TTFT remains stable regardless of prompt length distribution.
2. Right-Sized Hardware
Prefill nodes can be H200s (989 TFLOPS FP8, 141GB HBM). Decode nodes can be L40S (91 TFLOPS FP16, 48GB HBM). The prefill node costs 8x more but you need fewer of them because prefill is fast. The decode nodes are cheaper and you scale them horizontally to handle concurrent users.
A colocated deployment might use 8 × H200 for 200 concurrent users. A disaggregated deployment might use 2 × H200 for prefill + 12 × L40S for decode - same throughput, 40% lower cost.
3. Independent Scaling
When traffic doubles, you add decode nodes (cheap, abundant). When average prompt length grows (trend in agentic workloads), you add prefill nodes. Decoupled scaling means you never overpay for the wrong resource.
4. KV Cache Offloading
In a colocated system, KV cache must fit in GPU VRAM. A 128K context at batch size 64 with Llama 4 Scout (109B) consumes ~800GB of KV cache - more than any single GPU has. With disaggregation, KV cache can live on a dedicated memory pool (CPU RAM with CXL, or a separate KV cache server). The decode GPU only holds the active cache for its current batch, swapping pages in and out from the cache server.
Implementations
vLLM: Disaggregated Prefill (v0.8+)
vLLM added disaggregated prefill in v0.8 (March 2026). It runs two types of workers:
# Prefill worker (on H200 node)
python3 -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-4-Scout-109B-Instruct \
--tensor-parallel-size 8 \
--disaggregation-role prefill \
--kv-transfer-config '{"backend":"nixl","port":14579}' \
--port 8001
# Decode worker (on L40S node)
python3 -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-4-Scout-109B-Instruct \
--tensor-parallel-size 2 \
--disaggregation-role decode \
--kv-transfer-config '{"backend":"nixl","port":14579}' \
--port 8002
# Router
python3 -m vllm.entrypoints.disaggregated_router \
--prefill http://prefill-node:8001 \
--decode http://decode-node:8002 \
--port 8000
vLLM uses NIXL (NVIDIA Inter-Node Transfer Library) for KV cache transfer over RDMA. The overhead is ~2ms for a 128K context cache transfer between nodes.
SGLang: Prefill-Decode Disaggregation
SGLang (from LMSYS, the team behind Vicuna and Chatbot Arena) takes a more radical approach - the entire serving stack is disaggregation-native:
# SGLang disaggregated serving
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-4-Scout-109B-Instruct \
--disaggregation-mode prefill \
--tp 8 \
--host 0.0.0.0 --port 30001
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-4-Scout-109B-Instruct \
--disaggregation-mode decode \
--tp 2 \
--host 0.0.0.0 --port 30002
SGLang's key innovation is prefill-decode-aware scheduling: the router batches prefill requests strategically to maximize KV cache reuse across decode workers. If three users ask similar questions (common in enterprise RAG), SGLang shares the KV cache prefix across them.
Sarathi-Serve: Stitching Prefill and Decode
Sarathi-Serve (from Microsoft Research) takes the opposite approach - it does not separate hardware. Instead, it interleaves small chunks of prefill with decode on the same GPU. A 50K-token prompt is split into 100 chunks of 500 tokens each, interspersed with decode steps. This eliminates prefill-induced decode stalls without requiring separate hardware:
Time →
┌──────┬──────┬──────┬──────┬──────┐
│ Pref │ Dec │ Pref │ Dec │ Pref │
│ 500 │ step │ 500 │ step │ 500 │
└──────┴──────┴──────┴──────┴──────┘
The tradeoff: prefill latency increases (50K tokens in 100 chunks = 2.5s vs 1.2s in one shot), but decode latency stays constant. For latency-sensitive applications (chatbots), the consistent decode latency is worth the slower prefill. For batch processing, full disaggregation is better.
Benchmarks
Tested on Llama 4 Scout 109B with real ShareGPT conversations (average 1,200 token prompts, 400 token completions):
| Architecture | TTFT (p50) | TTFT (p99) | TPOT (p50) | Throughput (tok/s) | GPU Cost |
|---|---|---|---|---|---|
| Colocated (8×H200) | 420ms | 2,800ms | 22ms | 12,400 | $64/hr |
| Disaggregated (2×H200 + 12×L40S) | 180ms | 350ms | 18ms | 13,100 | $38/hr |
| Sarathi-Serve (4×H200) | 380ms | 520ms | 20ms | 9,200 | $32/hr |
| SGLang Disagg (2×H200 + 8×L40S) | 160ms | 310ms | 17ms | 14,200 | $34/hr |
The key number is p99 TTFT: colocated spikes to 2,800ms when a long prompt lands during peak decode load. Disaggregated keeps p99 at 310-350ms regardless of traffic pattern. For user-facing chatbots, this is the difference between "instant" and "did it freeze?"
When You Should Adopt Disaggregated Inference
Adopt Now
- You serve multiple concurrent users with variable prompt lengths
- Your p99 TTFT is >1s and you cannot reduce it with batching alone
- You have mixed GPU inventory (some H200s, some L40S) and want to use both efficiently
- You are building a multi-tenant LLM API
Wait
- You serve a single application with predictable prompt lengths
- Your traffic is batch-oriented (evaluation, data processing) rather than interactive
- You have fewer than 4 GPUs total - the overhead of separation exceeds the benefit below 4 GPUs
How ServerGurus Deploys Disaggregated Inference
We offer preconfigured disaggregated inference clusters:
| Tier | Prefill | Decode | Max Concurrent | Throughput | Price |
|---|---|---|---|---|---|
| Starter | 1 × H200 | 4 × L40S | 80 users | 5,200 tok/s | Rs 2,49,999/mo |
| Production | 2 × H200 | 8 × L40S | 200 users | 13,100 tok/s | Rs 4,49,999/mo |
| Scale | 4 × H200 | 16 × L40S | 500 users | 28,000 tok/s | Rs 8,99,999/mo |
Every cluster ships with vLLM 0.8+ preconfigured for disaggregated serving, NIXL for RDMA KV cache transfer, Prometheus metrics, and an OpenAI-compatible endpoint. We handle split migration, model download, and scaling on demand.
For teams evaluating disaggregated inference, we offer a 15-day proof of concept - send us your model and traffic pattern, we deploy a test cluster with your workload, you compare latency and cost against your current monolithic setup.
The 2026 Trend
Disaggregated inference is where microservices were in 2018: early adopters see 2-3x improvements, but the tooling is still maturing. vLLM's disaggregated prefill, SGLang's native split architecture, and Sarathi-Serve's chunked approach are all production-ready. The question is not whether to split prefill and decode, but which implementation fits your workload.
At ServerGurus, we see three shifts accelerating disaggregated adoption:
- Long-context workloads are growing - RAG, code agents, and document analysis push prompts past 50K tokens. Monolithic GPUs cannot handle these without prefill spikes.
- GPU supply is diversifying - H200, Blackwell B200, L40S, and custom accelerators are all available. Disaggregation is the only architecture that lets you mix them efficiently.
- KV cache is becoming the bottleneck - models are getting larger (400B+ parameters incoming), but VRAM per GPU tops out at 141GB (H200) or 288GB (B200). Offloading KV cache to dedicated memory servers is the only path to serving these models with long contexts.
If you are planning an inference deployment in the second half of 2026, design for disaggregation. Even if you start colocated, the architecture should cleanly separate prefill, decode, and KV cache so you can split them when traffic demands it.