How Huawei OceanStor M900 Tackles the KV-Cache Memory Wall

Long-context and agentic inference can generate more reusable key-value (KV) cache than a SuperPoD can keep close to its processors. Huawei’s OceanStor M900 addresses that capacity constraint by adding a shared, SSD-backed cache tier beneath on-chip memory and DRAM. The goal is to retain and reuse more context without treating SSDs as a replacement for the NPU’s fastest memory. 81012

How pooling expands available cache

Huawei says its Lingqu, or UnifiedBus, interconnect pools KV cache across a SuperPoD and supports tiering between on-chip memory, DRAM and SSDs. It claims up to 64 PB of capacity per cluster and says the KV-cache capacity available to each NPU rises from gigabytes to terabytes. That per-NPU figure describes access to shared, tiered capacity—not an increase in physical on-chip memory. 810

This makes M900 an inference-focused context-storage system rather than another compute chip. It complements NPUs and depends on the interconnect for pooled access; it does not replace either. Huawei describes it as designed for agent-heavy, longer-context workloads, while industry reporting identifies Atlas 960 SuperPoDs as a target setting. 126

The claimed access and inference gains

Huawei attributes a one-hop NPU-to-SSD path to a design integrating CPU, network-controller and NAND-controller functions. It says the path avoids CPU forwarding and protocol conversion, cutting access latency by 90% to 60 microseconds. Huawei also claims 40 TB/s of aggregate cluster bandwidth, or 1.5 times its stated comparison solution; that is a cluster-wide figure, not bandwidth guaranteed to each NPU. 8

For a “typical AI programming” workload, Huawei reports double the inference cluster’s token throughput and half the time to first token. Those figures should not be read as a universal speedup for every agentic or long-context request. 814

Endurance, cost and the evidence gap

M900’s KV-Aware scheduling predicts cache-data lifetime and manages placement across storage media, according to Huawei. The company claims support for up to 24 drive writes per day, 16-fold longer SSD-media life and three years of stable operation. It argues that fewer replacements and less maintenance can lower inference costs, but the cited material does not establish a measured cost-per-token saving. 8

The central distinction is between more cache capacity within reach and proven faster, cheaper inference at scale. Independent tests would need to establish usable capacity and cache-hit rates on real workloads; latency and sustained bandwidth under contention; token gains against a specified baseline; and SSD wear and total cost over time. The available third-party coverage reports Huawei’s performance figures rather than independently validating them. 614