How NVIDIA Slashed DeepSeek V4 Inference Costs by 5x on Blackwell GPUs (Full Optimization Breakdown)

This article breaks down exactly which optimizations NVIDIA deployed, what the DeepSeek V4 model family looks like under the hood, which inference providers are already using these improvements, and how NVIDIA frames the economic logic behind its laser focus on cost-per-token.

Software Optimizations Behind the 5x Cost Reduction

NVIDIA organizes its inference optimization stack into three layers: production operations, application acceleration, and infrastructure access . Across these layers, the specific techniques that delivered the 5x improvement include:

NVIDIA Dynamo Inference Framework

Dynamo is an open-source distributed serving framework that disaggregates inference phases across different GPUs. It separates prefill from decode, intelligently routes requests to the right GPU to avoid redundant computation, and extends GPU memory via NVLink-based caching to cost-effective storage tiers . Dynamo supports SGLang, TensorRT-LLM, and vLLM, and integrates natively with these open-source engines . The framework can boost the number of requests served by up to 7x on NVIDIA Blackwell, as demonstrated in the SemiAnalysis InferenceX benchmark .

TensorRT-LLM Optimizations

NVIDIA achieved a 5x reduction in cost-per-token through TensorRT-LLM optimization alone within two months of Blackwell launch, with no hardware change . At datacenter scale, a 5x reduction in cost-per-token represents a fivefold improvement in the revenue-generating capacity of the same infrastructure investment .

Disaggregated Serving

By separating the prefill (input processing) and decode (token generation) phases across different GPUs, disaggregated serving eliminates resource contention and allows each phase to be optimized independently for its specific needs . This is a core feature of the NVIDIA Dynamo framework .

Large-Scale Expert Parallelism

DeepSeek V4 uses a Mixture-of-Experts (MoE) architecture with 384 distributed experts . Optimized routing dispatches tokens across these experts with reduced redundant computation, improving efficiency across the GPU cluster .

NVLink-Based Parallel Communication

NVIDIA's high-speed GPU interconnect enables efficient all-to-all communication across experts, which is critical for MoE models where expert parallelism requires frequent data exchanges between GPUs .

NVFP4 Precision

Using 4-bit floating-point precision for inference reduces memory bandwidth and compute requirements without significant accuracy loss . For DeepSeek-V3.2, NVIDIA's NVFP4 quantization reduced the memory footprint by 1.7x compared to the original FP8 format (415 GB vs. 690 GB), leading to significant boosts in throughput and cost efficiency .

Multi-Token Prediction (MTP)

MTP generates multiple tokens per forward pass, increasing throughput. The first MTP support for DeepSeek V4 came on Day 3 from SGLang . Using MTP, SGLang later exceeded 12K tokens/second per GPU on GB300 NVL72 hardware .

Optimized CUDA Kernels

Not all optimizations came from NVIDIA alone. SemiAnalysis had to fix NVIDIA's open-source mHC (manifold-constrained hyper-connection) kernel launch code for DeepSeek V4's novel architecture, because TensorRT-LLM initially did not work well with the model . This community contribution was essential for production-quality inference.

SGLang Serving Recipes

LMSYS Org achieved a verified 5x throughput increase on NVIDIA GB300 NVL72 hardware using SGLang, jumping from approximately 2,200 to 11,200 tokens per second per GPU at roughly 50 tokens per second per user . NVIDIA's Dynamo support matrix specifically lists lmsysorg/sglang:deepseek-v4-blackwell as a supported configuration .

Combined, these optimizations deliver up to 20x higher throughput per GPU on Blackwell .

DeepSeek V4 Model Family Specifications

DeepSeek V4 was released on April 24, 2026, under the MIT license as a two-tier model family .

Sources:

The hybrid attention architecture combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), achieving just 27% of the single-token inference FLOPs of DeepSeek-V3.2 at 1M-token context . This efficiency is what makes million-token agentic context tracts computationally viable.

Inference Providers Already Using These Optimizations

Several providers and inference engines have deployed NVIDIA's software optimizations for DeepSeek V4 on Blackwell:

• LMSYS Org (SGLang) — achieved a verified 5x throughput increase on NVIDIA GB300 NVL72 hardware (June 23, 2026) . NVIDIA's Dynamo support matrix lists lmsysorg/sglang:deepseek-v4-blackwell as a supported configuration . SGLang also exceeded 12K tok/s per GPU on the same hardware .
• NVIDIA NIM — NVIDIA's own inference microservice stack offers DeepSeek-V4-Pro endpoints through the NVIDIA API catalog .
• vLLM — the open-source inference engine shipped DeepSeek V4 support within days of launch, integrated with NVIDIA Dynamo for distributed inference on GB200 .
• TensorRT-LLM — NVIDIA's own optimized runtime, though early adoption had issues with mHC kernel code that SemiAnalysis helped fix .
• DeepInfra — offers DeepSeek V4 Pro endpoints at $0.435/M input and $0.87/M output tokens .
• Together AI, Fireworks AI, Baseten — NVIDIA's blog notes broad deployment through GPU-accelerated cloud endpoints, with these providers reducing cost-per-token by up to 10x compared to Hopper .

Providers like Together AI and Baseten reduced cost-per-token by up to 90% compared with Hopper after adopting Blackwell, according to NVIDIA .

NVIDIA's Economic Framing for Agentic AI Workloads

NVIDIA positions cost-per-token as the single most important metric for inference total cost of ownership (TCO) — explicitly rejecting older metrics like cost-per-GPU-hour or FLOPS-per-dollar . Jensen Huang declared that "NVIDIA's cost per token is the lowest in the world" in April 2026, framing it as "a direct result of architecture excellence and extreme co-design" .

The reasoning behind this metric shift is directly tied to agentic AI:

Agentic Workloads Are Compute-Explosive on Inference

As AI shifts from one-shot answers to multi-step reasoning — planning, retrieving context, invoking tools, reflecting, and self-correcting — the number of tokens generated per query can multiply by 100x to 1,000x . A single multi-step agent task can cost $0.10 to $1.00 in inference compute . Gartner's March 2026 analysis confirmed that agentic AI models require 5–30x more tokens per task than standard chatbots .

Inference Is Now the Dominant Cost Center

Industry estimates suggest 55–80% of enterprise AI GPU spend goes to inference, not training . Deloitte estimates that inference accounts for approximately two-thirds of all AI compute in 2026, up from one-third in 2023 . Inference also accounts for 80 to 90 percent of the lifetime cost of a production AI system .

Software Optimization Unlocks Massive Value Without New Hardware

NVIDIA explicitly frames this as a strategic advantage: "NVIDIA achieved a 5x reduction in cost per token through TensorRT-LLM optimization alone within two months of Blackwell launch, with no hardware change" . At datacenter scale, a 5x reduction in cost-per-token directly determines whether agentic AI workloads become economically viable . NVIDIA's inference software keeps driving down token costs long after AI infrastructure is deployed .

Cost-per-Token Is the "Only TCO Metric That Matters"

NVIDIA argues that cost-per-token is the only metric that directly accounts for hardware performance, software optimization, ecosystem support, and real-world utilization . The company publishes "lowest token cost" as its headline Blackwell value proposition . The NVIDIA B200 achieves two cents per million tokens on GPT-OSS-120B, and the architecture lowered cost-per-million-tokens by 15x versus the prior generation .

In summary, NVIDIA's message is clear: agentic AI requires dramatically more inference tokens per task; software-level inference optimizations on Blackwell can cut those token costs by 5x without new hardware, directly determining whether large-scale agentic deployments are profitable .