From the source
Amazon published a distributed KV caching skill in the Agent Toolkit for AWS that lets coding agents evaluate and set up a shared KV cache for self-hosted LLM inference.
- Self-hosting often wastes GPU work because small, isolated caches on each server cannot hold or share the growing context that agents and chat applications resend, so a server picking up a resumed request recomputes the key-value attention states for every token.
- Amazon says distributed KV caching with Amazon ElastiCache can cut self-hosted LLM inference costs by up to 40% and time to first token by up to 80%.
- The skill works with Kiro, Claude Code,
OpenAI Codex and other agents that support Agent Skills, and walks through diagnosing a slow first token, sizing and pricing the cache, setting it up, and fixing a failed transfer.
- It runs a fit check with a read-only describe-instances call, estimates savings from the customer's own traffic with the KVTrace tool at no additional charge, and stops if the savings do not cover the cache's cost.
- Setup adds the LMCache connector to an existing vLLM or SGLang engine and sets the fleet to one KV ID mechanism and an explicit KV data type to avoid key mismatches.
- The flows require a coding agent that supports Agent Skills, GPU instances that support EFA RDMA write such as p5, LMCache 0.5.8rc1 or later with vLLM 0.11.1 or later or SGLang 0.5.13 or later, and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes.






