From the source
Amazon SageMaker Inference introduced prefix-aware routing, a new routing strategy that sends requests sharing the same prompt prefix to the same instance to improve KV cache reuse.
- In benchmarks on Llama 3.1 70B, it reduced P50 TTFT by up to 77% and increased throughput by up to 16%.





