From the source
Amazon's second 2026 SageMaker Inference retrospective walks through five stages of running models in production, following one hypothetical team from a single 70B-parameter support assistant to a larger reasoning model across two Regions.
- Inference recommendations, launched in April as an API, gained four updates that put it in SageMaker AI Studio, the SageMaker Python SDK v3, and coding agents.
- The Studio workflow, added in July 2026, lets teams pick a use-case profile and an optimization goal, then benchmarks candidate configurations on real GPU infrastructure using
NVIDIA AIPerf and ranks them by TTFT, inter-token latency, throughput, and cost.
- The SDK integration, in version 3.17.0 and later, benchmarks a live endpoint against production traffic and deploys the top-ranked configuration in a single call, and supports parallel jobs comparing serving frameworks such as LMI and vLLM.
- September 2026 added LoRA adapter support, accepting adapters as Model Package ARNs or S3 URIs and deploying them as Inference Components so adapters can change without disrupting the base model.
- A second September update added an SLA-based concurrency search that finds the highest concurrency an endpoint can sustain without breaching latency or error targets.






