From the source
Ant Ling reports that Ling-3.0-flash reaches 0.78 ms mean TPOT at batch 1 on four
NVIDIA B200 GPUs, after systems work on its speculative-decoding path with SGLang.
- The company frames batch-1 latency as a whole-stack problem, since at one request in flight there is no concurrency to amortize launch overhead and every microsecond on the decode critical path is visible to interactive agents and coding assistants.
- Ling-3.0-flash is a 42-layer hybrid-attention MoE model with 35 KDA linear-attention layers and 7 MLA full-attention layers, 512 routed experts plus one shared expert, run with tensor parallelism across four GPUs in bf16, with roughly 63 GB of weights per rank.
- In the final 8K profile, MLA attention accounts for only about 244 µs of a decode step, so the company says the hard part is cold MoE-weight reads, dense weight bandwidth, hundreds of small kernels and host-to-GPU handoffs.
- Traces showed the GPU busy for only about two-thirds of each batch-1 step, which Ant Ling traced to a KDA attention backend declaring CPU-side sequence lengths necessary; setting needs_cpu_seq_lens to False removed a device-to-host copy and synchronization and recovered roughly 485 µs of median host stall per step.
- The company then fused metadata chains and captured device-side metadata refills into a small CUDA Graph, keeping the capture boundary conservative because a FlashInfer-style plan() can rebuild internal state.
