# Amazon — Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters

- Company: Amazon (amazon.com)
- Announced: 2026-11-20T22:35:24+00:00
- Category: infrastructure-release
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint-across-regions-and-clusters/
- Record: https://forck.live/items/39465-introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint
- Subject: Bedrock / Nova

Amazon introduced the SageMaker HyperPod Global Inference Gateway, a SageMaker-managed gateway that routes inference requests across every registered HyperPod cluster, in one Region or across Regions. It answers a capacity problem: during a traffic peak, requests can queue on a saturated cluster while GPUs on another cluster, sometimes in another AWS Region, sit idle. The gateway picks the best-suited cluster and carries the request over the AWS backbone, with automatic failover and authentication through SigV4 and IAM, while each cluster's existing Inference Gateway keeps choosing the pod. Setup takes two steps: turning on the spec.globalRouting block in a cluster's InferenceGatewayConfig to publish it, then calling CreateGlobalInferenceGateway with the clusters to route across, which returns one endpoint. A cluster is reachable only after it is published; listing an unpublished cluster in a gateway fails with a message telling you what to fix. AWS operates the routing fleet, private connectivity, certificates and validation, and health and failover, while customers decide which clusters to publish, which to group, and how to weight routing. Bagelcode's Jinhyung Park said beta tests routed requests across two AWS Regions through a single endpoint and kept serving them when one cluster could no longer serve the model.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint-across-regions-and-clusters/:

> During a traffic peak, a request can sit in a queue on a saturated HyperPod cluster while GPUs on another cluster, sometimes in another AWS Region, sit idle. You have already paid for that capacity, but your requests can’t reach it.
> 
> In Introducing Amazon SageMaker HyperPod Inference Gateway , we showed how the Inference Gateway gives each Amazon SageMaker HyperPod cluster a single, GPU-aware endpoint. It reads real-time signals such as key-value (KV) cache utilization and queue depth to place every request on the best-suited pod. That solves routing inside one cluster, but not across clusters.
> 
> Today, SageMaker HyperPod introduces the Global Inference Gateway. It is one SageMaker-managed gateway that routes across every cluster you register, in one Region or across Regions. The gateway picks the cluster and carries the request over the AWS backbone, with automatic failover and authentication through AWS Signature Version 4 (SigV4) and AWS Identity and Access Management (IAM). Inside each cluster, the Inference Gateway keeps picking the pod, unchanged. The Global Inference Gateway is the second tier of the architecture we described in part 1.

---

Record: https://forck.live/items/39465-introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
