Supermicro, IBM, and NVIDIA launched a jointly validated KV cache storage solution that persists inference context on shared flash instead of recomputing it on GPUs. Futurum examines what the underlying white paper proves about shared storage as a production memory tier for large-scale AI inference.
What Is Covered in This Article:
- Joint Supermicro, IBM, and NVIDIA KV cache storage solution launched November 4, 2026
- 56x faster time-to-first-token at 130K-token prompts versus GPU recompute
- 22x throughput gain from 0.19 to 4.26 requests per second across 200 concurrent requests
- White paper codification of NVIDIA Dynamo’s five-tier KV cache memory hierarchy, including the CMX flash appliance
- A noisy-neighbor stress test sustaining 3.6 requests per second against 200 GB/s of competing network traffic
- Competitive field spanning WEKA, VAST Data, DDN, Lightbits, and NVIDIA’s own pooled flash tier
The News: Supermicro (NASDAQ: SMCI) announced a key-value (KV) cache storage solution for large-scale AI inference developed with IBM and NVIDIA on November 4, 2026. The solution combines Supermicro Petascale all-flash storage servers on AMD EPYC 9005 processors, IBM Storage Scale Erasure Code Edition (ECE), the NVIDIA Dynamo inference framework, and NVIDIA Spectrum-X Ethernet with BlueField-3 DPUs. The architecture persists previously computed KV states on shared storage so inference platforms reuse long context across requests, sessions, and GPU nodes instead of recomputing it, targeting agentic AI, coding assistants, long-context retrieval-augmented generation, and multi-turn applications. In joint testing against a 120-billion-parameter model, persistent KV cache delivered up to 56x faster time-to-first-token (TTFT) at 130,000-token prompts. As input grew from 30,000 to 130,000 tokens, TTFT rose by 0.377 seconds with the cache versus 29.55 seconds without it.
“Enterprises need a fundamentally better way to scale AI inference,” said Sam Werner, GM, IBM Storage. “Agentic AI and long-context applications depend on reusing context efficiently across many steps,” said Shruti Koparkar, AI Inference Product Lead at NVIDIA. The companies documented the full benchmark study in an IBM Redbook, Context Without Limits.
Supermicro, IBM, and NVIDIA Cut Inference Latency 56x With KV Cache on Shared Storage
Analyst Take: The white paper beneath the press release is the more consequential document because it converts “storage as memory” from a vendor pitch into a jointly engineered reference architecture with a published methodology and stress conditions designed to resemble production. The economic motive shows up in Futurum’s AI Supply Chain Market Model, which puts high-bandwidth memory at 39.0% of accelerator cost of goods in 2026, rising to 45.6% in 2027. Every gigabyte of context parked in HBM displaces active computation on the most expensive silicon in the data center. Moving reusable context to erasure-coded flash becomes irresistible once the latency math works, and the benchmarks argue that it now does.
The White Paper Codifies NVIDIA’s Five-Tier Memory Hierarchy, Including a Tier Few Have Named
The underlying Redbook lays out NVIDIA Dynamo’s KV cache hierarchy in five tiers, HBM, system DRAM, local NVMe, pooled flash through NVIDIA’s Inference Context Memory Storage appliance (CMX), and shared enterprise storage persisting for days to months. The pooled flash designation is among the clearest public placements of CMX inside the hierarchy to date, and it defines exactly where NVIDIA’s own appliance ends and partner storage begins. The paper’s case for shared storage rests on failure domains and capacity ceilings.
Each of the first three tiers is bounded by a single node’s physical limits and a node failure causes the cached context to be lost. IBM Storage Scale ECE aggregates the storage cluster into a single namespace accessible to multiple GPU clusters at once, which the paper brands “Context Caching.” The test cluster, 8 Supermicro Petascale nodes in an 8+2P erasure code, presented 644 TB of usable capacity, roughly 9 billion tokens of KV cache for the gpt-oss-120b test model. Inference context is becoming a durable data asset with its own storage tier rather than scratch space that evaporates with the session.
The Methodology Answers the Standard Objections to Storage Benchmarks
The test design is where this validation separates itself from the KV cache claims that have crowded 2026. The team disabled the hottest tiers by setting their capacity to zero in Dynamo’s configuration, forcing all 1.4 million tokens to use shared storage on every retrieval. The 56x TTFT gain at 130K tokens, 0.57 seconds against 32.14 seconds of GPU recompute, is therefore a measurement of the slowest tier alone, with TTFT nearly flat between 0.193 and 0.57 seconds across every prompt length. The throughput run started with a cold cache, so the 22x gain, 0.19 to 4.26 requests per second with total time for 200 requests falling 95%, includes the penalty of early uncached requests. A third benchmark added 200 GB/s of competing network I/O on a 3-switch spine-leaf topology, and throughput fell only 18% to 3.6 requests per second. The partners also disclosed headroom, since the 8-node tier can serve up to 315 GB/s of reads while client networking in the test topped out at 80 GB/s.
All inference results came from a single client node running four NVIDIA RTX PRO 6000 Blackwell GPUs, with gpt-oss-120b quantized to MXFP4 on one GPU. That choice proves out enterprise-grade hardware, and it means behavior under hundreds of concurrent GPU nodes remains a projection from disclosed headroom rather than a measured result.
Shared Storage Is Now a Contested Tier That NVIDIA Also Occupies
The shared storage tier the paper validates is filling up with competitors. WEKA has integrated its Augmented Memory Grid with Dynamo and NIXL, VAST Data announced a DPU-native architecture for shared KV cache, Lightbits claims 16x session density with Inferra, and DDN and Hammerspace are pursuing the same socket. Futurum assessed NVIDIA’s push to standardize this layer in its coverage of open AI storage at FMS 2026.
IBM’s differentiation is specific. Storage Scale ECE descends from GPFS, a parallel file system with decades of multi-tenant production history, it runs over Spectrum-X Ethernet rather than proprietary fabrics, and erasure coding across nodes delivers the fault tolerance per-node tiers lack. For Supermicro, the solution builds a storage attach story onto an AI server franchise that produced breakout GPU platform revenue in Q2 FY 2026.
The unresolved tension sits one tier up. NVIDIA’s CMX and partner storage default to a layered outcome, pooled flash for minutes-to-hours reuse and shared storage for longer persistence. The collision would come if NVIDIA sizes CMX into the persistence range or publishes reference designs that stop at pooled flash. The direction of travel favors tiering. Inference demand is restructuring the memory hierarchy from HBM outward, a shift Futurum traced in Intel’s D-Series bet that HBM becomes optional for agentic inference, and every tier that absorbs context cheaply extends the effective capacity of the GPUs above it.
What to Watch:
- Whether named enterprise deployments of the reference architecture appear by mid-2027
- Whether NVIDIA positions CMX pooled flash alongside shared storage or grows it into the persistence tier
- Whether WEKA, VAST Data, or DDN publish Dynamo-integrated benchmarks that match the 56x TTFT claim
- Whether the results repeat on HGX-class GPU clusters at production concurrency beyond a single client node
- Whether IBM Storage Scale attach shows up in Supermicro storage revenue in FY 2027 results
Read the complete press release on the KV cache solution on the Supermicro website.
Sources
Declaration of generative AI and AI-assisted technologies in the writing process: This content has been generated with the support of artificial intelligence technologies. Due to the fast pace of content creation and the continuous evolution of data and information, The Futurum Group and its analysts strive to ensure the accuracy and factual integrity of the information presented. However, the opinions and interpretations expressed in this content reflect those of the individual author/analyst. The Futurum Group makes no guarantees regarding the completeness, accuracy, or reliability of any information contained herein. Readers are encouraged to verify facts independently and consult relevant sources for further clarification.
Disclosure: Futurum is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of Futurum as a whole.
Read the full Futurum Group Disclosure.
Other Insights From Futurum:
IBM Quantum Safe Treats PQC Migration as More Than Just a Project
IBM Bets on Vault’s Installed Base to Secure AI Agent Identity
IBM Q3 FY 2026: Software Growth and Deal Recovery Lift Execution
Author Information
Brendan is Research Director, Semiconductors, Supply Chain, and Emerging Tech. He advises clients on strategic initiatives and leads the Futurum Semiconductors Practice. He is an experienced tech industry analyst who has guided tech leaders in identifying market opportunities spanning edge processors, generative AI applications, and hyperscale data centers.
Before joining Futurum, Brendan consulted with global AI leaders and served as a Senior Analyst in Emerging Technology Research at PitchBook. At PitchBook, he developed market intelligence tools for AI, highlighted by one of the industry’s most comprehensive AI semiconductor market landscapes encompassing both public and private companies. He has advised Fortune 100 tech giants, growth-stage innovators, global investors, and leading market research firms. Before PitchBook, he led research teams in tech investment banking and market research.
Brendan is based in Seattle, Washington. He has a Bachelor of Arts Degree from Amherst College.
