# Unite.AI — AWS Adds Managed NVIDIA Dynamo to SageMaker HyperPod Inference

- Company: Unite.AI (unite.ai)
- Announced: 2026-11-17T23:51:05+00:00
- Category: availability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.unite.ai/aws-adds-managed-nvidia-dynamo-to-sagemaker-hyperpod-inference/
- Record: https://forck.live/items/37683-aws-adds-managed-nvidia-dynamo-to-sagemaker-hyperpod-inference
- Subject: Reporting
- Models affected: GLM-5.3, Qwen3-235B-A22B-FP8, Qwen3.5-397B-A17B-FP8

AWS says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for managed multi-node and disaggregated model serving, with benchmark results showing 2.05 times the highest qualifying aggregated output-token throughput per GPU under its latency targets. The addition targets two constraints in serving large language models: fitting weights and the reusable KV cache into GPU memory, and keeping long prompt processing from slowing token generation when both share GPUs. AWS illustrates the memory pressure with GLM-5.3, whose FP8 weights alone exceed the 640 GB of GPU memory on one ml.p5.48xlarge instance before KV cache and runtime overhead. Dynamo is an open-source distributed inference framework that works with vLLM, SGLang and TensorRT-LLM, and its disaggregated topology routes requests through a frontend to prefill and decode workers that scale independently but both load full model weights. In a Qwen3-235B-A22B-FP8 comparison on 16 H200 GPUs, disaggregated serving reached 207 output tokens per second per GPU at concurrency 80 against 101 at concurrency 32 for the best aggregated result, so AWS notes the gain applies to latency-constrained throughput at different concurrency levels. A separate Qwen3.5-397B-A17B-FP8 test found a two-node setup reached 126.4 output tokens per second per GPU, or 99.1% of single-node throughput, at concurrency 32 with high prefix reuse. Users can bring their own DynamoGraphDeployment manifest or use the managed InferenceEndpointConfig path, which defaults to vLLM and supports an OpenAI-compatible interface through SageMaker endpoints.

## Evidence

Verbatim from https://www.unite.ai/aws-adds-managed-nvidia-dynamo-to-sagemaker-hyperpod-inference/:

> Dynamo's disaggregated topology lets prefill and decode pools scale independently while both still load full model weights.
> 
> AWS's 2.05 times throughput gain compares latency-constrained results at different concurrency levels rather than unrestricted maxima.
> 
> Managed HyperPod automates coordination and recovery but leaves topology and GPU allocation trade-offs to measured workload behavior.
> 
> Amazon Web Services announced on November 17, 2026, that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for managed multi-node and disaggregated model serving, alongside benchmark results showing 2.05 times the highest qualifying aggregated output-token throughput per GPU under AWS’s latency targets.
> 
> The addition addresses two constraints in serving large language models: fitting model weights and the reusable key-value (KV) cache into GPU memory, and keeping long prompt processing from slowing token generation when both share GPUs. AWS illustrates the memory pressure with GLM-5.3, whose 8-bit floating-point (FP8) weights alone exceed the 640 GB of GPU memory on one ml.p5.48xlarge instance before KV cache and runtime overhead are included.

---

Record: https://forck.live/items/37683-aws-adds-managed-nvidia-dynamo-to-sagemaker-hyperpod-inference
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
