# Amazon — Amazon SageMaker AI real-time endpoints now support KV cache aware routing

- Company: Amazon (amazon.com)
- Announced: 2026-11-17T21:11:51+00:00
- Category: capability-change
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware-routing/
- Record: https://forck.live/items/37567-amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware-routing
- Subject: Bedrock / Nova

Amazon SageMaker AI added KV cache-aware routing, a routing strategy for real-time endpoints that sends each request to an instance already holding a long cached match for its prompt and not too busy. It is aimed at teams serving large language models with multi-turn conversations, long shared prompts or heavy load, where requests often begin with the same text. The container reports its cache contents through an environment variable, and the endpoint configuration sets RoutingStrategy to KV_AWARE; clients keep calling the endpoint with the same APIs. SageMaker AI keeps a managed cache index of that state, holding only one-way truncated hashes of cached content mapped to the instance or inference component copy that holds it, and never prompts, token IDs, request payloads or KV tensors. The endpoint weighs how much of a prompt each candidate has cached against how busy it is, skipping candidates that are too busy and falling back to load-aware routing when no candidate has a useful match. Because the index matches on tokens rather than a fixed number of bytes, a multi-turn history can match in full as it grows, unlike prefix-aware routing, which hashes the start of a request and cannot tell when an instance has evicted that cache. KV-aware routing is active only with at least two instances, or at least two copies of an inference component; with a single instance the setting has no effect.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware-routing/:

> Today, Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing), a new routing strategy for real-time endpoints. It uses the cache state that the model container reports. It sends each request to an instance that already holds a long cached match for its prompt and isn’t too busy. It’s for teams that serve large language models (LLMs) with multi-turn conversations, long shared prompts, or heavy load. You turn it on in two places: add one environment variable to your model so the container reports its cache contents, and set RoutingStrategy to KV_AWARE in the endpoint configuration. Your clients call the endpoint with the same APIs as before.
> 
> Here’s the problem it solves: LLM applications such as chat assistants, Retrieval Augmented Generation (RAG), and agents with long system prompts send many requests that start with the same text. A model keeps the work it did on a prompt in GPU memory, the KV cache, and skips that work when the same start reaches the same instance again. In part 1 of this series , we introduced prefix-aware routing for Amazon SageMaker AI, which sends requests that start the same way to the same instance.

---

Record: https://forck.live/items/37567-amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware-routing
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
