From the source
Lead story
Ask your AI
Top stories
Models & availability
Latest
Amazon· Bedrock / NovaIntroducing Amazon SageMaker HyperPod Inference Gateway
GPU-aware routing system reduces first-token latency by up to 82% with single addon install.
Amazon announced SageMaker HyperPod Inference Gateway, a Kubernetes-native GPU-aware routing system that deploys as an EKS managed addon to optimize inference request placement on GPU clusters.
The system reduces first-token latency by up to 82% through real-time GPU metrics and intelligent routing algorithms that consider KV cache utilization, queue depth, LoRA adapter residency, and prefix cache hit rates, with no changes required to model servers or client applications.
From the source
It is a Kubernetes-native, GPU-aware routing system that deploys as a single EKS managed addon on your existing HyperPod infrastructure. It uses real-time GPU signals to place every inference request on the best-suited pod, delivering lower latency with no changes to your model servers or client applications.
aws.amazon.com
Reported by
Other stories
Amazon · 2 OctSweep thousands of leases for compliance using Amazon Quick and the Adjudicated Query patternAmazon · 2 OctAdd secure Web Search to Claude Desktop with Amazon Bedrock AgentCoreAmazon · 2 OctFine-tune a search agent with multi-turn RL on Amazon SageMaker AIAmazon · 1 OctServe live, governed data in AI-built apps with Amazon Quick