From the source
Amazon announced SageMaker HyperPod Inference Gateway, a Kubernetes-native GPU-aware routing system that deploys as an EKS managed addon to optimize inference request placement on GPU clusters.
- The system reduces first-token latency by up to 82% through real-time GPU metrics and intelligent routing algorithms that consider KV cache utilization, queue depth, LoRA adapter residency, and prefix cache hit rates, with no changes required to model servers or client applications.





