More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

NVIDIA says GPU memory pressure keeps rising as models grow larger, contexts run longer and agentic workloads multiply, while memory supply tightens and prices climb — a combination the company expects to continue through the near term. Its stated answer is not simply packing more memory onto the accelerator, but designing GPUs, CPUs, networking, storage and software together so each layer of the AI factory extracts more work from the memory it already has.

That framing rests on how an AI factory actually runs: on a memory and storage hierarchy. GPU high-bandwidth memory sits at the top as the fastest, most expensive and most contested tier, where model weights and KV cache compete for limited capacity. Below it, CPU memory, local SSDs and shared storage each trade speed for more capacity at a lower cost per bit. How much useful work a byte performs, NVIDIA argues, depends on shrinking what must be stored, placing it on the right tier and bringing it back to the GPU before it is needed.

The company claims that approach is already paying off. From Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3%, while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX — a spread that suggests utilization, not raw capacity, is driving the gains. NVIDIA also says it was the first to adopt HBM3e and HBM4 technologies into an accelerated computing stack.

The pressure behind those numbers comes from demand side growth in the workload itself. Frontier models now carry trillions of parameters, and KV cache — which holds the context each active request needs so the model avoids reprocessing the same information — is expanding alongside them. Agentic trajectories carry prior context across repeated model and tool calls, in some cases processing up to 3 million tokens, so working memory per request grows over time. Retrieval-augmented generation compounds the effect by injecting relevant enterprise information into prompts. During inference, HBM primarily holds both model parameters and KV cache, concentrating that pressure at the top of the hierarchy.

NVIDIA's own framing notes the limit of the obvious fix: capacity delivers only what bandwidth can move. More HBM per GPU helps, but without the bandwidth and placement logic to keep it fed, added capacity translates poorly into useful work — which is why the company argues for codesign rather than a memory-stacking arms race.

What to watch next is whether near-term supply and pricing trends actually ease, and how much of the remaining gap is closed by new memory generations versus software that places and evicts data more intelligently. Open questions include whether lower tiers such as local SSDs can absorb agentic and RAG context fast enough, and whether the modest capacity gains reported between Blackwell Ultra and Vera Rubin can sustain throughput-per-dollar improvements as trajectory lengths keep growing.