Google Open-Sources TPU Raiden: A Direct Challenger to NVIDIA's NIXL for KV-Cache Transfer in Disaggregated LLM Inference

Offloading primitives. Raiden provides the underlying mechanisms to move KV-cache data out of TPU memory to external storage tiers, supporting hierarchical cache management similar to what NIXL does with NVMe and RDMA backends S. This is essential for extending the effective cache capacity beyond expensive on-chip memory.

TPU-native transport. Unlike NVIDIA's GPUDirect RDMA stack, Raiden operates over Google's TPU interconnect fabric (ICI) and is optimized for TPU v5e and newer hardware . This means it is designed from the ground up for Google's accelerator architecture rather than being adapted from a GPU-centric approach.

TPU Raiden vs. NVIDIA NIXL: A Feature Comparison

The following table summarizes how the two libraries compare across key dimensions:

The Broader Industry Trend: Open-Sourcing Inference Infrastructure

TPU Raiden's release is part of a clear and accelerating industry shift toward open-sourcing the inference software stack.

Both hyperscaler vendors are racing to open-source their stacks. NVIDIA open-sourced NIXL alongside the Dynamo inference framework at GTC 2025 . Google has been steadily externalizing its TPU software: first through vLLM TPU integration, then through the Kubernetes-native llm-d distributed inference framework, and now with Raiden .

Disaggregated inference has become the standard serving architecture. Separating prefill and decode stages improves time-to-first-token latency and resource utilization — but it makes fast KV-cache transfer the critical performance bottleneck . Both Raiden and NIXL exist to solve precisely this problem .

Vendor lock-in is being fought at the data-movement layer. NIXL is described as "vendor-agnostic" and supports AWS EFA, not just NVIDIA's own interconnects . Raiden similarly plugs into open frameworks (vLLM, SGLang, llm-d). Both libraries are designed to make their respective hardware ecosystems more attractive to developers rather than forcing proprietary APIs S.

The ecosystem is converging on pluggable KV-connector interfaces. vLLM's KVConnector API now accepts NIXL and Mooncake connectors, and the architecture is designed to accommodate any backend — including Raiden — allowing operators to swap the transport layer without changing the inference engine . This pluggability is critical for multi-accelerator environments where hardware flexibility matters.

Why TPU Raiden Matters for AI Infrastructure

TPU Raiden addresses a fundamental scaling challenge. As large language models grow, the KV cache they produce during inference becomes enormous — often exceeding the on-chip memory of individual accelerators. Moving this data between prefill and decode nodes efficiently is the difference between a responsive inference system and one bogged down by data-transfer bottlenecks S.

By open-sourcing Raiden, Google is not just matching NVIDIA's NIXL play — it is signaling that TPU-based inference is a serious contender for production AI workloads. The library gives TPU operators the same class of tools that GPU operators have had since NIXL's release: fine-grained control over how KV-cache data moves between accelerators, memory hierarchies, and external storage tiers.

For the broader AI industry, Raiden's release means that disaggregated inference on TPUs is now a viable option with proper tooling. Developers deploying Google's TPU hardware for production inference can leverage the same disaggregated architecture patterns that have become standard on NVIDIA GPU clusters, without being locked into proprietary data-movement APIs.