From the source
- A wrong target fails quietly: too conservative leaves GPU instances mostly idle, while too aggressive lets each instance hold more concurrent requests than it can serve inside its latency promise.
- The walkthrough deploys Kokoro-82M, an 82-million-parameter TTS model, in a custom container on a SageMaker AI real-time endpoint using AWS CDK stacks.
- It derives the scale-out threshold, scale-in threshold and both cooldowns from two measurements: Q_max, the highest concurrency one instance sustains before breaching the SLO, and T_total, how long a scale-out takes end to end.
- The policy scales on ConcurrentRequestsPerModel, a native SageMaker AI metric, rather than a custom metric the container must emit.
- For the Kokoro-82M run, the measured ratio between the client-measured and CloudWatch-reported thresholds was exactly 1.0 at every rung of the concurrency ladder.
- The sample container pins an earlier PyTorch with a known advisory, PYSEC-2026-2286, fixed in torch 2.10.0, which the post says should be reviewed and upgraded before any non-proof-of-value use.
