# Amazon — Amazon SageMaker AI now detects latency drift, idle GPUs, and missed optimizations on your inference endpoints

- Company: Amazon (amazon.com)
- Announced: 2026-11-19T21:28:18+00:00
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed-optimizations-on-your-inference-endpoints/
- Record: https://forck.live/items/38916-amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed
- Subject: Bedrock / Nova

Amazon SageMaker AI now monitors generative AI inference endpoints and flags when a change could improve performance or lower cost, using its Generative AI optimized inference recommendations feature with the agentic experience in SageMaker Studio. Monitoring is on by default for endpoints that use vLLM serving containers and GPU instances, needs no setup, and can be turned off per endpoint. SageMaker AI says it looks only at operational signals such as latency, GPU utilization and the endpoint's serving configuration, and does not read requests or responses. It reports three kinds of findings: latency drift, meaning a week-over-week rise in median end-to-end latency or time to first token; unused capacity, meaning GPUs that sit mostly idle; and configuration gaps, such as a missing performance improvement or security patch. Findings arrive through AWS Health, one notification per endpoint, and each states what was noticed, why it matters and a recommended action. Opening the notification leads to the agentic experience in SageMaker Studio with the endpoint and its recent traffic filled in, where benchmarks run on separate compute and nothing changes until a result is applied. Monitoring carries no GPU cost, while benchmarks use standard SageMaker AI charges or an existing ML reservation.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed-optimizations-on-your-inference-endpoints/:

> Every inference endpoint is well tuned the day it goes into production. Teams choose an instance type, serving configuration, and scaling settings that suit the model and its traffic at launch. With the endpoint serving smoothly, they understandably move on to the next priority. The workload, however, rarely stays still. Traffic patterns evolve, prompts grow longer, serving engines such as vLLM ship new releases, SageMaker inference develops new optimization techniques, and new fine-tuned checkpoints change the resources your model needs. Over time, a well-tuned endpoint quietly drifts. Latency and time to first token edge upward. A configuration that was never quite ideal keeps missing available performance gains. A GPU provisioned for demand that never materialized sits idle week after week. Because nothing has failed, no alarm sounds, and the drift goes unnoticed until users feel it or the bill reflects it.
> 
> Amazon SageMaker AI now monitors your generative AI inference endpoints and tells you when a change could improve performance or lower cost by using Generative AI optimized inference recommendations feature with Agentic experience in Amazon SageMaker Studio .

---

Record: https://forck.live/items/38916-amazon-sagemaker-ai-now-detects-latency-drift-idle-gpus-and-missed
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
