# Amazon — Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

- Company: Amazon (amazon.com)
- Announced: 2026-11-17T23:33:26+00:00
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference/
- Record: https://forck.live/items/37667-managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference
- Subject: Bedrock / Nova
- Models affected: GLM-5.3

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, letting teams run distributed LLM serving with disaggregated prefill and decode on managed GPU infrastructure. The integration addresses a concrete limit: GLM-5.3's FP8 weights exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance before KV cache and runtime overhead are counted. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supplies the supporting infrastructure, including high-performance Elastic Fabric Adapter networking, coordinated GPU scheduling, and node health monitoring and recovery. The add-on installs Dynamo custom resource definitions plus the HyperPod Inference Operator, Dynamo operator, Grove, and the Kubernetes AI Scheduler, which uses gang scheduling to place multi-node worker pods together. Two configuration paths are offered: bringing an existing DynamoGraphDeployment, or defining an endpoint through InferenceEndpointConfig with a dynamoSpec section, with vLLM as the default engine.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference/:

> Serving large language models (LLMs) efficiently requires sufficient GPU memory and careful coordination of prompt processing and token generation. GLM-5.3’s FP8 weights, for example, exceed the 640 GB of GPU memory on a single ml.p5.48xlarge instance, before accounting for KV cache and runtime overhead. Long prompts can also slow token generation when both share the same GPUs.
> 
> Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo , combining Dynamo’s distributed inference capabilities with the infrastructure management of HyperPod to address these challenges. Dynamo is an open source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines such as vLLM, SGLang, and TensorRT-LLM. HyperPod supports these deployments with high-performance Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, and node health monitoring and recovery. In this post, we illustrate multi-node and disaggregated serving through a deployment example, share benchmark results, and outline best practices for optimizing inference on HyperPod.
> 
> How the integration works

## Follow-ups

- Unite.AI (reported, 17 Nov): AWS Adds Managed NVIDIA Dynamo to SageMaker HyperPod Inference — https://forck.live/items/37683-aws-adds-managed-nvidia-dynamo-to-sagemaker-hyperpod-inference

---

Record: https://forck.live/items/37667-managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
