# Amazon — Cut self-hosted LLM cost and latency with the distributed KV caching agent skill

- Company: Amazon (amazon.com)
- Announced: 2026-11-21T00:14:26+00:00
- Category: developer-tool-release
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://aws.amazon.com/blogs/machine-learning/cut-self-hosted-llm-cost-and-latency-with-the-distributed-kv-caching-agent-skill/
- Record: https://forck.live/items/39486-cut-self-hosted-llm-cost-and-latency-with-the-distributed-kv-caching-agent-skill
- Subject: Bedrock / Nova

Amazon published a distributed KV caching skill in the Agent Toolkit for AWS that lets coding agents evaluate and set up a shared KV cache for self-hosted LLM inference. Self-hosting often wastes GPU work because small, isolated caches on each server cannot hold or share the growing context that agents and chat applications resend, so a server picking up a resumed request recomputes the key-value attention states for every token. Amazon says distributed KV caching with Amazon ElastiCache can cut self-hosted LLM inference costs by up to 40% and time to first token by up to 80%. The skill works with Kiro, Claude Code, OpenAI Codex and other agents that support Agent Skills, and walks through diagnosing a slow first token, sizing and pricing the cache, setting it up, and fixing a failed transfer. It runs a fit check with a read-only describe-instances call, estimates savings from the customer's own traffic with the KVTrace tool at no additional charge, and stops if the savings do not cover the cache's cost. Setup adds the LMCache connector to an existing vLLM or SGLang engine and sets the fleet to one KV ID mechanism and an explicit KV data type to avoid key mismatches. The flows require a coding agent that supports Agent Skills, GPU instances that support EFA RDMA write such as p5, LMCache 0.5.8rc1 or later with vLLM 0.11.1 or later or SGLang 0.5.13 or later, and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes.

## Evidence

Verbatim from https://aws.amazon.com/blogs/machine-learning/cut-self-hosted-llm-cost-and-latency-with-the-distributed-kv-caching-agent-skill/:

> Customers increasingly self-host large language models (LLMs) on GPUs for control over cost, accuracy, and latency. However, self-hosting often means paying for redundant GPU work because small, isolated caches on each server can’t hold or share the growing context AI agents and chat applications resend. Consider a legal AI assistant that can accumulate hundreds of thousands of tokens of case history in each conversation. When a lawyer returns to a thread after lunch, responses from the LLM can take minutes. The GPU server that picks up the request has none of that history in its memory. So, the GPU recomputes the key-value (KV) attention states for all of those tokens, a step called prefill, before it writes a word. The same pattern slows down resumed coding agents, agentic AI assistants, and document Q&A assistants. Distributed KV caching with Amazon ElastiCache can cut self-hosted LLM inference costs by up to 40% and time to first token (TTFT) by up to 80%. Distributed KV caching extends GPU-local caches with a shared tier beside your GPU fleet so any server can reuse KV state for days across sessions, routing shifts, and server failures.

---

Record: https://forck.live/items/39486-cut-self-hosted-llm-cost-and-latency-with-the-distributed-kv-caching-agent-skill
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
