From the source
NVIDIA is describing how it designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory as larger models, longer contexts and agentic applications raise GPU memory pressure while supply tightens and prices rise.
- The company says the approach is not to pack more memory onto the GPU but to manage data across a memory and storage hierarchy, where HBM is the fastest, most expensive and most contested tier and model weights and KV cache compete for its limited capacity.
- NVIDIA says that from Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3% while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX.
- It points to the memory wall, the point at which memory bandwidth cannot supply data as fast as compute can use it, as the reason extra HBM capacity can no longer be put to work while still adding cost and drawing power.
- The techniques it describes include the NVFP4 format, which reduces the space required for weights compared to FP8, the lowest precision possible on the NVIDIA Hopper platform, and applying NVFP4 to the KV cache so more context stays resident in HBM.
- Within an NVL72 system, model weights can be distributed across up to 72 GPUs, leaving more HBM for KV cache and concurrent users, while NVIDIA Dynamo moves inactive context to NVIDIA CMX context memory storage and returns it before reuse.
