From the source
From the source
Baseten built a Qwen 3.6 engine that beat vLLM by up to 90% on decode speed using MetaInfer skills.
Baseten used the MetaInfer skills-only framework to build a custom inference engine for Qwen-3.6-35B-A3B on a single B200.
The engine achieved up to 90% faster single-stream decode speed and 2.33x faster time to first token (TTFT) compared to vLLM 0.25.1, enabling new ultra-low-latency use cases for the model.
From the source
With Qwen-3.6-35B-A3B in NVFP4 precision on a single B200, our LLM-generated inference engine outperformed vLLM 0.25.1 (latest at the time of the experiment) by up to 90% on single-stream decode speed, and with 2.33x faster TTFT, opening up new ultra-low-latency use cases for the model that had not been practical before.
baseten.co
