Together AI· Inference platformStory of the weekModel update

Serving DeepSeek-V4: why million-token context is an inference systems problem

Together AI discusses serving implications of DeepSeek-V4, focusing on its 1M-token context window and architectural changes with compressed sparse attention mechanisms. They describe how V4's token-axis KV cache compression improves serving capacity but requires sophisticated memory management across multiple cache types (CSA, HCA, SWA). The post shares early bring-up results on NVIDIA HGX B200 hardware, showing increased KV-cache capacity from 1.2M to 3.7M tokens on a single node through cache policy optimizations.

From the source

V4's architecture creates the opportunity for long-context efficiency, but the realized capacity depends on how the inference engine stores, recomputes, and evicts the different cache types.

together.ai