Unlocking Longer Generation with Key-Value Cache Quantization

Hugging Face announces a new KV Cache Quantization feature in the Transformers library that reduces memory usage for long-context text generation by quantizing the key-value cache, enabling longer generations on consumer GPUs with minimal quality loss.