From the source
Amazon SageMaker Inference now supports GPU partitioning through
NVIDIA Multi-Instance GPU, letting customers run several models on one accelerator with hardware-enforced isolation instead of one model per whole GPU.
- The change answers a cost problem: an ml.p4d.24xlarge holds eight A100s and therefore eight models, whether each needs 40 GB or 14 GB, so teams either under-served a model or paid for capacity it never used.
- Each partition gets a fixed share of compute units and a dedicated block of memory, and a different model can sit in each partition.
- Partitioning works only on endpoints that use inference components, and single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned.
- MIG splits one GPU into as many as seven instances depending on the accelerator, and the layout must be chosen when the endpoint is created because partitions must be idle to change it.
- Amazon says SageMaker handles GPU configuration, placement and scaling, work that previously meant running a Kubernetes stack; Salesforce helped define requirements and validate the initial implementation and is preparing to test it on a representative production workload.






