From the source
Amazon SageMaker HyperPod Inference now supports
NVIDIA Dynamo, letting teams run distributed LLM serving with disaggregated prefill and decode on managed GPU infrastructure.
- The integration addresses a concrete limit: GLM-5.3's FP8 weights exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance before KV cache and runtime overhead are counted.
- Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM.
- HyperPod supplies the supporting infrastructure, including high-performance Elastic Fabric Adapter networking, coordinated GPU scheduling, and node health monitoring and recovery.
- The add-on installs Dynamo custom resource definitions plus the HyperPod Inference Operator, Dynamo operator, Grove, and the Kubernetes AI Scheduler, which uses gang scheduling to place multi-node worker pods together.
- Two configuration paths are offered: bringing an existing DynamoGraphDeployment, or defining an endpoint through InferenceEndpointConfig with a dynamoSpec section, with vLLM as the default engine.






