# NVIDIA — More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

- Company: NVIDIA (nvidia.com)
- Announced: 2026-11-05T16:00:00+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://blogs.nvidia.com/blog/memory-full-stack-codesign/
- Record: https://forck.live/items/32164-more-performance-from-every-byte-how-full-stack-codesign-solves-ai-s-memory
- Subject: AI platform
- Models affected: FLUX.2 [klein], Inkling, Beam 1.0, Meta Muse Spark, Step 5 Preview, NVIDIA Nemotron 3.5 Super VL

NVIDIA is describing how it designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory as larger models, longer contexts and agentic applications raise GPU memory pressure while supply tightens and prices rise. The company says the approach is not to pack more memory onto the GPU but to manage data across a memory and storage hierarchy, where HBM is the fastest, most expensive and most contested tier and model weights and KV cache compete for its limited capacity. NVIDIA says that from Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU rose roughly 3% while agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX. It points to the memory wall, the point at which memory bandwidth cannot supply data as fast as compute can use it, as the reason extra HBM capacity can no longer be put to work while still adding cost and drawing power. The techniques it describes include the NVFP4 format, which reduces the space required for weights compared to FP8, the lowest precision possible on the NVIDIA Hopper platform, and applying NVFP4 to the KV cache so more context stays resident in HBM. Within an NVL72 system, model weights can be distributed across up to 72 GPUs, leaving more HBM for KV cache and concurrent users, while NVIDIA Dynamo moves inactive context to NVIDIA CMX context memory storage and returns it before reuse.

## Evidence

Verbatim from https://blogs.nvidia.com/blog/memory-full-stack-codesign/:

> Larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise — a trend expected to continue through the near term.
> 
> NVIDIA works closely with memory partners to rapidly adopt the latest memory technologies. NVIDIA was first to adopt HBM3e and HBM4 technologies into an accelerated computing stack.
> 
> But the approach isn’t to simply pack more memory onto the GPU. It’s full-stack codesign. NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory.
> 
> That matters because an AI factory runs on a memory and storage hierarchy. GPU high-bandwidth memory (HBM) is the fastest, most expensive and most contested tier — with model weights and KV cache competing for limited capacity. Below it sit CPU memory, local SSDs and shared storage, each trading speed for more capacity at a lower cost per bit.
> 
> How much useful work a byte does depends on how well the whole hierarchy is used: shrinking what must be stored, placing it on the right tier and bringing it back to the GPU before it’s needed.

---

Record: https://forck.live/items/32164-more-performance-from-every-byte-how-full-stack-codesign-solves-ai-s-memory
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md