From the source
Amazon introduced the SageMaker HyperPod Global Inference Gateway, a SageMaker-managed gateway that routes inference requests across every registered HyperPod cluster, in one Region or across Regions.
- It answers a capacity problem: during a traffic peak, requests can queue on a saturated cluster while GPUs on another cluster, sometimes in another AWS Region, sit idle.
- The gateway picks the best-suited cluster and carries the request over the AWS backbone, with automatic failover and authentication through SigV4 and IAM, while each cluster's existing Inference Gateway keeps choosing the pod.
- Setup takes two steps: turning on the spec.globalRouting block in a cluster's InferenceGatewayConfig to publish it, then calling CreateGlobalInferenceGateway with the clusters to route across, which returns one endpoint.
- A cluster is reachable only after it is published; listing an unpublished cluster in a gateway fails with a message telling you what to fix.
- AWS operates the routing fleet, private connectivity, certificates and validation, and health and failover, while customers decide which clusters to publish, which to group, and how to weight routing.
- Bagelcode's Jinhyung Park said beta tests routed requests across two AWS Regions through a single endpoint and kept serving them when one cluster could no longer serve the model.





