# Amazon — SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

- Company: Amazon (amazon.com)
- Announced: 2026-11-06T18:01:00+00:00
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu-utilization-for-model-endpoints/
- Record: https://forck.live/items/32733-sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu
- Subject: Bedrock / Nova

Amazon SageMaker Inference now supports GPU partitioning through NVIDIA Multi-Instance GPU, letting customers run several models on one accelerator with hardware-enforced isolation instead of one model per whole GPU. The change answers a cost problem: an ml.p4d.24xlarge holds eight A100s and therefore eight models, whether each needs 40 GB or 14 GB, so teams either under-served a model or paid for capacity it never used. Each partition gets a fixed share of compute units and a dedicated block of memory, and a different model can sit in each partition. Partitioning works only on endpoints that use inference components, and single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned. MIG splits one GPU into as many as seven instances depending on the accelerator, and the layout must be chosen when the endpoint is created because partitions must be idle to change it. Amazon says SageMaker handles GPU configuration, placement and scaling, work that previously meant running a Kubernetes stack; Salesforce helped define requirements and validate the initial implementation and is preparing to test it on a representative production workload.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu-utilization-for-model-endpoints/:

> When you deploy a model to a GPU endpoint, the smallest unit of GPU you can request is one whole accelerator.
> 
> This constraint can cost you twice. First, you can’t host more models on an instance than it has accelerators. An ml.p4d.24xlarge has eight A100s, so it holds eight models, whether each one needs 40 GB or 14 GB. Second, and less obvious, it makes every deployment come down to a choice of which full GPU to deploy to. As your concurrency and prompt lengths grow, a full GPU on an older instance stops being enough, but a full GPU on a newer one is more than your model can use. You end up either under-served or paying for capability you never reach.
> 
> Today, Amazon SageMaker Inference supports GPU partitioning, which uses NVIDIA Multi-Instance GPU (MIG) to divide a physical GPU into independent partitions. Each partition receives a fixed share of the GPU’s compute units and a dedicated block of its memory, with isolation enforced in hardware. That changes both constraints: you can place a different model in each partition, and you can choose a fraction of a newer accelerator instead of a whole one of either generation.
> 
> This post covers both.

---

Record: https://forck.live/items/32733-sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
