Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

Amazon's post explains how bi-directional KV cache transfer in vLLM lets prefill and decode nodes reuse previously computed context across turns of a multi-turn agentic conversation.

• Conventional disaggregated serving moves KV cache only from prefill to decode, so the prefill node cannot retrieve the previous response's entries and must recompute them, while the decode node holds session state and raises the chance of cache eviction on prefill nodes.
• The feature returns previously computed KV from the decode node to the prefill node on later turns, governed by a configurable KV recompute threshold, and is coordinated by a KV-cache-aware router that maps a client conversation_id to the decode node's kv_transfer_params.
• Amazon says it built and validated the feature on AWS Trainium clusters connected with EFA and contributed it to vLLM as an accelerator-agnostic capability.
• In multi-turn benchmarks on a 1P1D deployment using Amazon EC2 P5en instances, bi-directional KV transfer reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, with results varying by workload and configuration.
• The instances use AWS Elastic Fabric Adapter, which supports GPUDirect RDMA and Neuron Direct RDMA to move KV blocks directly between GPU or Trainium device memory.