Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

Amazon contributed a push-based NIXL connector to vLLM that moves post-prefill KV transfer coordination and setup off the critical path for disaggregated prefill/decode inference.

• In pull mode, the router relay, decode-side block allocation and NIXL READ run serially after prefill, extending time-to-first-token.
• Push mode has the decode instance register its target blocks in advance while the producer is still computing KV, so the transfer starts as soon as the producer's blocks are ready.
• The setup work runs on the prefill side and on a background thread, keeping those cycles off the decode instance's token-generation phase.
• On a 1P1D deployment of Amazon EC2 P5en instances connected with EFA, Amazon says push mode delivered up to 3x better TTFT and up to 30 percent better TPOT than pull mode for the Qwen3-32B model, with results depending on workload and configuration.
• Amazon built and validated the feature on AWS Trainium clusters connected with EFA, then contributed it to vLLM as an accelerator-agnostic feature; the reported results come from a GPU cluster with EFA.