From the source
From the source
From the source
From the source
Dynamo's disaggregated topology lets prefill and decode pools scale independently while both still load full model weights. AWS's 2.05 times throughput gain compares latency-constrained results at different concurrency levels rather than unrestricted maxima.
Managed HyperPod automates coordination and recovery but leaves topology and GPU allocation trade-offs to measured workload behavior. Amazon Web Services announced on November 17, 2026, that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for managed multi-node and disaggregated model serving, alongside benchmark results showing 2.05 times the highest qualifying aggregated output-token throughput per GPU under AWS’s latency targets.
The addition addresses two constraints in serving large language models: fitting model weights and the reusable key-value (KV) cache into GPU memory, and keeping long prompt processing from slowing token generation when both share GPUs. AWS illustrates the memory pressure with GLM-5.3, whose 8-bit floating-point (FP8) weights alone exceed the 640 GB of GPU memory on one ml.p5.48xlarge instance before KV cache and runtime overhead are included. …
unite.ai