# Hugging Face — Unlocking Longer Generation with Key-Value Cache Quantization

- Company: Hugging Face (huggingface.co)
- Announced: 2024-05-16T00:00:00+00:00
- Category: capability-change
- Subject: Platform
- Source: https://huggingface.co/blog/kv-cache-quantization
- Record: https://forck.live/items/1889-unlocking-longer-generation-with-key-value-cache-quantization

Hugging Face announces a new KV Cache Quantization feature in the Transformers library that reduces memory usage for long-context text generation by quantizing the key-value cache, enabling longer generations on consumer GPUs with minimal quality loss.

## Evidence

Verbatim from https://huggingface.co/blog/kv-cache-quantization:

> KV Cache Quantization reduces memory usage for long-context text generation in LLMs with minimal impact on quality, offering customizable trade-offs between memory efficiency and generation speed.

---

Record: https://forck.live/items/1889-unlocking-longer-generation-with-key-value-cache-quantization
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md