Serving DeepSeek-V4: why million-token context is an inference systems problem

Together AI discusses serving implications of DeepSeek-V4, focusing on its 1M-token context window and architectural changes with compressed sparse attention mechanisms.

They describe how V4's token-axis KV cache compression improves serving capacity but requires sophisticated memory management across multiple cache types (CSA, HCA, SWA).

The post shares early bring-up results on NVIDIA HGX B200 hardware, showing increased KV-cache capacity from 1.2M to 3.7M tokens on a single node through cache policy optimizations.