Huawei OceanStor M900: How Its Shared KV Cache Works

Introduced at HUAWEI CONNECT 2026, OceanStor M900 is Huawei’s context-memory storage system for SuperPoD AI inference. Its job is to keep more reusable inference context available across accelerators as prompts, conversations and agent workflows grow. 19

Why inference needs a shared KV cache

A key–value (KV) cache holds information computed while a model processes a prompt. Reusing a suitable cached prefix can spare an inference system from processing that prefix again. Long contexts and repeated agent interactions make that cache more valuable—but also harder to fit entirely in memory close to each neural processing unit (NPU). Huawei positions the M900 as a way to expand and share that cache across a SuperPoD. 136

How UnifiedBus and storage tiers fit together

The M900 uses Huawei’s UnifiedBus network to pool KV-cache data across on-chip memory, DRAM and SSDs. Huawei says a single cluster can provide 64 PB of KV-cache capacity, moving the amount accessible to an NPU from gigabytes into the terabyte range. That is shared cluster storage, not 64 PB of memory inside an NPU. More retained context could improve reuse when requests have cacheable prefixes, but the benefit depends on what the workload actually reuses. 3617

To make the SSD tier practical for inference, Huawei describes a one-hop path between a SuperPoD NPU and SSDs. Its design integrates CPU, network-controller and NAND-controller functions to avoid some conventional protocol-conversion and CPU-forwarding steps. Huawei reports access latency of roughly 60 microseconds and 40 TB/s of aggregate bandwidth per cluster. Those are stated architecture and system figures, not a guarantee for every request. 362832

The claimed payoff—and the evidence limit

If a request can reuse cached work, it may reach its first generated token sooner; less repeated computation could also free capacity for more tokens overall. Huawei claims that, in a typical AI coding scenario, the architecture doubled inference-cluster token throughput and halved time to first token. Those workload-specific claims should not be read as universal performance gains. 2833

Huawei also describes KV-aware placement that manages how long cached data stays on different storage media. The company cites SSD endurance of up to 24 drive writes per day and a 16-fold lifespan improvement, with a three-year stability target. The intended economic case is less recomputation and fewer storage replacements, lowering inference cost per token. The provided evidence does not independently establish those performance, lifetime or cost outcomes in production SuperPoDs. 6833