# Amazon — Stop guessing auto scaling targets for TTS on Amazon SageMaker AI

- Company: Amazon (amazon.com)
- Announced: 2026-10-22T16:30:06+00:00
- Category: not stated
- Coverage: not counted
- Announcement: no
- Group: routine
- Source: https://aws.amazon.com/blogs/machine-learning/stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai/
- Record: https://forck.live/items/26724-stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai
- Subject: Bedrock / Nova
- Models affected: Kokoro-82M

Amazon's guide shows how to replace guessed auto scaling targets for self-hosted text-to-speech endpoints on Amazon SageMaker AI with values measured against a live deployment. A wrong target fails quietly: too conservative leaves GPU instances mostly idle, while too aggressive lets each instance hold more concurrent requests than it can serve inside its latency promise. The walkthrough deploys Kokoro-82M, an 82-million-parameter TTS model, in a custom container on a SageMaker AI real-time endpoint using AWS CDK stacks. It derives the scale-out threshold, scale-in threshold and both cooldowns from two measurements: Q_max, the highest concurrency one instance sustains before breaching the SLO, and T_total, how long a scale-out takes end to end. The policy scales on ConcurrentRequestsPerModel, a native SageMaker AI metric, rather than a custom metric the container must emit. For the Kokoro-82M run, the measured ratio between the client-measured and CloudWatch-reported thresholds was exactly 1.0 at every rung of the concurrency ladder. The sample container pins an earlier PyTorch with a known advisory, PYSEC-2026-2286, fixed in torch 2.10.0, which the post says should be reviewed and upgraded before any non-proof-of-value use.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai/:

> We replace guesswork with two measurements taken against the live endpoint. Q_max is the highest concurrency one instance sustains before breaching the SLO, and T_total is how long a scale-out takes end to end. We derive the scale-out threshold, scale-in threshold, and both cooldowns from those two numbers by arithmetic.

---

Record: https://forck.live/items/26724-stop-guessing-auto-scaling-targets-for-tts-on-amazon-sagemaker-ai
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md