DeepSeek open-sources DeepInfer, the inference framework behind its production services

DeepSeek has open-sourced DeepInfer, the inference framework behind its own production services, under the Apache 2.0 license on GitHub.

• The company calls reaching a massive user base at a reasonable price "the core problem an inference framework has to solve," and says one cluster now serves 155 trillion tokens a day, with revenue at off-peak pricing working out to $38 per GPU per hour.
• DeepInfer supports only DeepSeek-V4.1-Flash for now, loading the updated 1017 version by default, and runs on Nvidia's SM100 Blackwell GPUs and Huawei's Ascend 950 neural processing units; Hopper and other older Nvidia architectures are not on the list.
• Two features that keep outputs identical from run to run work only on Nvidia hardware, and all published benchmarks, which are DeepSeek's own, ran on Nvidia chips.
• Speculative decoding runs through DSpark, where a draft model proposes five tokens at a time and a confidence head predicts which the main model will accept, trimming draft tokens checked per step by about 30% and, DeepSeek estimated, adding 4% to 7% throughput at production load.
• DeepSeek has not released its key-value cache management and load-balancing code, saying the two are tightly coupled to internal infrastructure, and splitting prefill and decode across separate machines is left to operators.