How Much Network Does Your AI Inference Server Actually Need? RDMA, RoCE, and When Gigabit Is Fine
The Short Answer
| Inference Setup | Minimum Network | Recommended Network |
|---|---|---|
| Single GPU, single model | 1 Gigabit | 10 Gigabit |
| Multi-GPU (TP=2-4, single node) | NVLink (GPU-to-GPU) | NVLink + 10GbE for API |
| Multi-node inference (data parallel) | 25 Gigabit | 100 Gigabit |
| Disaggregated prefill/decode | 100 Gigabit RoCE v2 | 200Gb/s InfiniBand NDR |
| Training (for context) | 200Gb/s InfiniBand | 400Gb/s InfiniBand NDR400 |
If you run a single GPU or tensor parallelism on one node, your network is almost irrelevant. The GPU does all the work internally. If you split inference across multiple nodes or disaggregate prefill and decode, the network becomes the bottleneck. Here is how to figure out where you fall.
Inference Is Not Training
AI training saturates network links. A 200B parameter model training across 8 nodes with tensor parallelism + data parallelism + pipeline parallelism sends gigabytes of gradients every optimizer step. Every 100ms, 8 nodes exchange 400MB each. That's 32Gb/s sustained, bidirectional. You need InfiniBand. You need RDMA. You need fabric-level congestion control.
Inference is different.
During inference, the model weights are loaded into GPU VRAM once. The only data that moves during operation is:
- Input tokens - a few hundred bytes per request
- KV cache - grows with context length, transferred only during disaggregated prefill→decode
- Output tokens - a few bytes each, streamed to the API consumer
That's it. No gradients. No optimizer states. No all-reduce collectives. The network traffic for a monolithic inference server is almost entirely HTTP requests coming in and JSON responses going out. Gigabit handles thousands of concurrent inference requests.
The Network Budget of a Single-GPU Inference Server
An Llama 4 Scout (109B) server with 1 × H200 generating tokens at 50 tok/s per request:
| Metric | Per Request | At 100 Concurrent Users |
|---|---|---|
| Input payload (1,200 tokens) | 4.8KB | 480KB/s ingress |
| Output tokens (400 tokens) | 1.6KB | 160KB/s egress |
| Total per request | 6.4KB | 640KB/s |
A 1Gb/s link handles ~125MB/s - about 20,000 concurrent inference requests at these sizes. You will run out of GPU VRAM and compute long before you run out of network bandwidth on a single-GPU inference server.
Verdict: Gigabit is fine. 10GbE is generous. 25GbE is wasted money for single-GPU inference.
When the Network Matters: Multi-Node Inference
Three scenarios push inference networking requirements up:
1. Data Parallelism (Model Replicas)
Each node runs a complete copy of the model. A load balancer distributes requests. Nodes don't talk to each other - they talk to the load balancer and to clients.
┌─────────┐
│ NGINX │ ← 10GbE or higher (aggregating 100+ concurrent streams)
└───┬─┬─┬──┘
┌──────┘ │ └──────┐
┌────▼───┐ ┌──▼──┐ ┌───▼────┐
│ Node 0 │ │Node1│ │ Node 2 │ ← Gigabit to each node
│ 4×L40S │ │4×L40│ │ 4×L40S │
└────────┘ └─────┘ └────────┘
Each node independently serves requests. No inter-node traffic. The load balancer sits at the edge and needs enough bandwidth to aggregate all client connections. At 100 concurrent users pushing 640KB/s total, even the load balancer stays under 1Gb/s.
Verdict: 1GbE per node. 10GbE for the load balancer. No RDMA needed.
2. Tensor Parallelism (Single Node)
Tensor parallelism splits layers across GPUs within the same server. GPUs communicate over NVLink (900GB/s on H200, 1.8TB/s on B200) or PCIe (64GB/s on Gen5 x16). The network never touches this traffic.
┌─────────────────────────────────┐
│ Node │
│ ┌─────┐ ┌─────┐ ┌─────┐ │
│ │GPU 0│◄─┤GPU 1│◄─┤GPU 2│ │ ← NVLink 900 GB/s
│ └─────┘ └─────┘ └─────┘ │
│ TP group size = 4 │
└─────────────────────────────────┘
│
10GbE API port
Verdict: Network is irrelevant for tensor parallelism. The interconnects that matter are NVLink and PCIe. The network port just serves the API.
3. Pipeline Parallelism (Multi-Node)
Pipeline parallelism splits model layers across nodes. Node 0 handles layers 1-20, Node 1 handles layers 21-40. Activations flow between nodes.
┌──────────┐ ┌──────────┐
│ Node 0 │──────► Node 1 │
│ Layers 1-20│ act. │Layers 21-40│
│ 4×H200 │ │ 4×H200 │
└──────────┘ └──────────┘
Each activation tensor is batch_size × sequence_length × hidden_size. For Llama 4 Scout (hidden_size=8192) at batch_size=32 and sequence_length=8192:
activation_size = 32 × 8192 × 8192 × 2 bytes (BF16) = 4GB per pipeline stage boundary
At 50 decode steps per second, that's 200GB/s flowing between nodes. For comparison:
| Interconnect | Bandwidth | Handles pipeline parallelism? |
|---|---|---|
| 25GbE | 3.1 GB/s | No |
| 100GbE | 12.5 GB/s | No |
| 200Gb/s InfiniBand NDR | 25 GB/s | Barely - needs tensor parallelism to reduce activation size |
| NVLink (same node) | 900 GB/s | Yes |
This is why virtually nobody runs pure pipeline parallelism across nodes for inference. It saturates even 200Gb/s links. The practical solution is tensor parallelism within nodes and data parallelism across nodes.
Disaggregated Inference: The Network Tax
Disaggregated inference (covered in our disaggregated inference post) separates prefill and decode across different nodes. This introduces a new network bottleneck: KV cache transfer.
┌──────────────┐ KV Cache ┌──────────────┐
│ Prefill │──────────────► Decode │
│ 2×H200 │ 32GB/req │ 8×L40S │
└──────────────┘ └──────────────┘
For a 128K context Llama 4 Scout request:
kv_cache_size = 2 layers × 128K tokens × 2 (K+V) × 8 heads × 128 dim × 2 bytes
≈ 32GB per request
At 10 prefill requests per second, that's 320GB/s - you need 2.5Tb/s of network bandwidth. Even with RDMA and NVLink fabric, this is impractical for real-time transfer.
The fix: KV cache is not transferred all at once. vLLM and SGLang use RDMA (NIXL in vLLM, NVSHMEM in SGLang) to stream KV cache pages incrementally - only the active pages move, and they move page-by-page rather than in one 32GB burst. This drops the bandwidth requirement from 320GB/s to ~8GB/s (64Gb/s) - manageable on 100GbE RoCE v2.
RDMA vs RoCE v2: What You Actually Need
| Technology | Bandwidth | Latency | Cost (switch + NIC per node) | Use Case |
|---|---|---|---|---|
| InfiniBand NDR200 | 200Gb/s | 0.6μs | $8,000+ | Training, disaggregated infer at scale |
| InfiniBand NDR | 400Gb/s | 0.5μs | $12,000+ | Large training clusters |
| RoCE v2 (100GbE) | 100Gb/s | 1.5μs | $2,500 | Disaggregated inference, KV cache transfer |
| RoCE v2 (25GbE) | 25Gb/s | 2.5μs | $800 | Multi-node data-parallel inference |
| TCP/IP (100GbE) | 100Gb/s | 10-20μs | $2,500 | Not recommended for GPU-to-GPU |
| TCP/IP (10GbE) | 10Gb/s | 10-20μs | $200 | API serving, small data-parallel clusters |
| TCP/IP (1GbE) | 1Gb/s | 50-100μs | $30 | Single-node inference, management |
RDMA (Remote Direct Memory Access) lets one GPU write directly into another GPU's memory over the network - no CPU involvement, no kernel TCP stack, no data copies. Latency drops from 10-20μs (TCP) to 0.6-1.5μs (RDMA).
RoCE v2 (RDMA over Converged Ethernet) runs RDMA over standard Ethernet switches. You need RoCE-capable NICs (ConnectX-6 or newer) and a switch that supports PFC (Priority Flow Control) and ECN (Explicit Congestion Notification). Practically, this means a Arista 7050 or NVIDIA Spectrum switch - not a $100 Netgear.
InfiniBand is a separate fabric with its own switches, cables, and subnet manager. No Ethernet frames. Sub-microsecond latency. The gold standard for training clusters. Overkill for inference.
What ServerGurus Ships
Every GPU server includes:
| GPU Tier | Default Networking | Upgrade Options |
|---|---|---|
| Single L40S | 10GbE SFP+ | 25GbE |
| 4 × L40S | 25GbE SFP28 | 100GbE RoCE |
| 4 × H200 | 100GbE RoCE v2 (ConnectX-7) | 200Gb/s InfiniBand NDR |
| 8 × H200 / B200 | 200Gb/s InfiniBand NDR | 400Gb/s NDR400 |
For most inference customers, the 25GbE on a 4 × L40S node handles everything: API traffic, data-parallel replication, and model downloads. The 100GbE RoCE upgrade ($299/month) adds disaggregated inference capability - KV cache streaming between prefill and decode nodes.
If you are building a training cluster, every server ships with InfiniBand. If you are building an inference cluster, start with 25GbE and monitor. Upgrade to 100GbE RoCE only when you disaggregate.
The Networking Decision Tree
Are you training a model?
├─ Yes → InfiniBand. Do not overthink this.
└─ No (inference only)
├─ Single node, all GPUs in one box?
│ └─ Yes → 10GbE is plenty. 1GbE works for light use.
│ GPU-to-GPU communication is NVLink. Network is API only.
│
├─ Multi-node, data parallel (each node has full model)?
│ └─ Yes → 25GbE per node. No RDMA needed.
│ Nodes don't talk to each other. Load balancer needs bandwidth.
│
└─ Multi-node, disaggregated prefill/decode?
└─ Yes → 100GbE RoCE v2 minimum.
KV cache streaming needs RDMA. TCP won't cut it.
Budget $2,500 per node for NIC + switch port.
The 2026 Reality
The gap between "what GPU vendors recommend" and "what inference actually uses" is wide. NVIDIA's reference architecture ships ConnectX-7 400Gb/s InfiniBand on every DGX. For inference, that NIC sits 98% idle.
The only inference workload that saturates a fast network is disaggregated prefill/decode with KV cache transfer. If you are not running that architecture, do not buy InfiniBand for inference. Buy more GPUs instead. A $12,000 InfiniBand NIC+switch port buys you an entire additional L40S GPU.
For everything else - single GPU, tensor parallelism within a node, data parallelism across nodes - standard Ethernet at 10-25GbE handles inference traffic with room to spare. The network is not your bottleneck. Your GPU VRAM is.