From the source
Amazon SageMaker AI added KV cache-aware routing, a routing strategy for real-time endpoints that sends each request to an instance already holding a long cached match for its prompt and not too busy.
- It is aimed at teams serving large language models with multi-turn conversations, long shared prompts or heavy load, where requests often begin with the same text.
- The container reports its cache contents through an environment variable, and the endpoint configuration sets RoutingStrategy to KV_AWARE; clients keep calling the endpoint with the same APIs.
- SageMaker AI keeps a managed cache index of that state, holding only one-way truncated hashes of cached content mapped to the instance or inference component copy that holds it, and never prompts, token IDs, request payloads or KV tensors.
- The endpoint weighs how much of a prompt each candidate has cached against how busy it is, skipping candidates that are too busy and falling back to load-aware routing when no candidate has a useful match.
- Because the index matches on tokens rather than a fixed number of bytes, a multi-turn history can match in full as it grows, unlike prefix-aware routing, which hashes the start of a request and cannot tell when an instance has evicted that cache.
- KV-aware routing is active only with at least two instances, or at least two copies of an inference component; with a single instance the setting has no effect.






