From the source
Amazon SageMaker AI now monitors generative AI inference endpoints and flags when a change could improve performance or lower cost, using its Generative AI optimized inference recommendations feature with the agentic experience in SageMaker Studio.
- Monitoring is on by default for endpoints that use vLLM serving containers and GPU instances, needs no setup, and can be turned off per endpoint.
- SageMaker AI says it looks only at operational signals such as latency, GPU utilization and the endpoint's serving configuration, and does not read requests or responses.
- It reports three kinds of findings: latency drift, meaning a week-over-week rise in median end-to-end latency or time to first token; unused capacity, meaning GPUs that sit mostly idle; and configuration gaps, such as a missing performance improvement or security patch.
- Findings arrive through AWS Health, one notification per endpoint, and each states what was noticed, why it matters and a recommended action.
- Opening the notification leads to the agentic experience in SageMaker Studio with the endpoint and its recent traffic filled in, where benchmarks run on separate compute and nothing changes until a result is applied.
- Monitoring carries no GPU cost, while benchmarks use standard SageMaker AI charges or an existing ML reservation.






