NVIDIA Released an Open-Weight LLM That Does 6x the Work on the Same GPU
NVIDIA dropped Nemotron-Labs-Diffusion on July 10. It is an 8B parameter language model that delivers 6.82 accepted tokens per forward pass. For comparison, the existing Eagle3 speculative decoding approach on Qwen3-8B gives you 2.75 tokens per pass. That is 2.5x more throughput from the same GPU.
The model is on Hugging Face under Apache 2.0. You can download it, deploy it, and get 6x the inference throughput compared to standard autoregressive decoding on the same hardware. Here is how it works and how to run it.
What Makes Nemotron Different
Most language models generate one token at a time. You give them a prompt, they predict the next token, append it, and repeat. This is slow because the GPU does a full forward pass for every single token.
Speculative decoding speeds this up by using a small draft model to predict several tokens at once, then having the main model verify them. The draft model is fast but dumb. The main model is slow but smart. Together they can generate multiple tokens per forward pass. The bottleneck is that you need two separate models loaded in GPU memory.
Nemotron eliminates the draft model entirely. A single checkpoint handles three modes:
- Autoregressive mode - standard token-by-token generation
- Diffusion mode - generates multiple tokens in parallel using a diffusion process
- Self-speculation mode - the model drafts tokens and verifies them using the same weights and a shared KV cache
The result is a single model that does what previously required a draft model plus a verifier model, but with half the memory footprint and no coordination overhead between two separate inference engines.
Real Numbers
From the paper, Nemotron-8B versus Qwen3-8B with Eagle3 speculative decoding:
| Metric | Qwen3-8B + Eagle3 | Nemotron-8B Diffusion |
|---|---|---|
| Accepted tokens per forward pass | 2.75 | 6.82 |
| GPU memory (two models vs one) | 32 GB | 16 GB |
| Inference throughput (requests/sec) | baseline | 2.4x |
On a single NVIDIA L40S with 48 GB VRAM, you can serve roughly 2.4x more requests per second with Nemotron compared to a similarly sized model using traditional speculative decoding. On an H100, the gap widens further because the GPU has more compute headroom that the dual-model approach cannot use.
How to Deploy Nemotron on Your GPU Server
Nemotron is available on Hugging Face under nvidia/Nemotron-Labs-Diffusion-8B. Here is the deployment process:
Step 1: Clone the model
pip install huggingface_hub
huggingface-cli download nvidia/Nemotron-Labs-Diffusion-8B --local-dir ./nemotron-8b
Step 2: Install dependencies
pip install torch transformers accelerate
Step 3: Serve with vLLM or TGI
The model uses a standard transformer architecture and is compatible with vLLM for high-throughput serving:
pip install vllm
vllm serve ./nemotron-8b --host 0.0.0.0 --port 8000 --max-model-len 8192
Step 4: Call the API
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
response = client.chat.completions.create(
model="./nemotron-8b",
messages=[{"role": "user", "content": "Explain speculative decoding"}],
max_tokens=500
)
The model handles the rest. You do not need to configure diffusion mode or self-speculation. The inference engine detects the tri-mode capability and switches automatically.
What This Means for Your GPU Budget
The industry has spent two years throwing more GPUs at the inference throughput problem. Bigger clusters. More H100s. Higher bills. Nemotron flips the equation: get more output from the GPU you already have.
If you currently run a 4-GPU inference cluster serving a Qwen-sized model, Nemotron at 8B gives you comparable output quality with 2.4x the throughput. That means you could potentially serve the same traffic with 2 GPUs instead of 4. Or keep all 4 GPUs and handle 2.4x more users.
For the 86% of enterprises running GPUs at half capacity or less, this is not a nice-to-have. It is a cost reduction sitting on Hugging Face right now.
The Catch
Nemotron is an 8B model. It is good at coding, reasoning, and general text generation. It is not a 70B or 405B frontier model. If your workload requires deep reasoning across long contexts, you still need a larger model.
But for the vast majority of production inference workloads (chatbots, summarization, classification, code completion, RAG pipelines), an 8B model with 6.82 tokens per pass is more than enough. And at 2.4x the throughput of alternatives, the cost per request is hard to beat.
GPU Infrastructure That Makes This Practical
Deploying Nemotron on a shared cloud GPU gives you variable performance. You are sharing the GPU with other tenants. Your 2.4x throughput advantage shrinks when the hypervisor schedules someone else's workload on your GPU.
Dedicated GPU servers give you the full hardware. Our L40S and H100 nodes run your models with no noisy neighbors, no vCPU overcommit, and no hypervisor tax on CUDA operations. You get the full 6.82 tokens per pass that the paper promises.
If you are running inference at scale and your per-token cost matters, a dedicated GPU server running Nemotron pays for itself within a quarter compared to shared GPU instances at the same throughput.
Check GPU cloud plans and run Nemotron on dedicated hardware.