# Hugging Face — Differential Transformer V2

- Company: Hugging Face (huggingface.co)
- Announced: 2026-01-20T03:20:57+00:00
- Category: research-paper
- Subject: Platform
- Source: https://huggingface.co/blog/microsoft/diff-attn-v2
- Record: https://forck.live/items/1562-differential-transformer-v2

Microsoft introduces DIFF V2, a differential attention mechanism that doubles query heads without increasing key-value heads, enabling faster decoding and eliminating the need for custom attention kernels, while removing per-head RMSNorm to improve numerical stability.

## Evidence

Verbatim from https://huggingface.co/blog/microsoft/diff-attn-v2:

> DIFF V2 introduces additional query heads compared to the baseline Transformer, but does not increase the number of key-value (KV) heads. Since LLM decoding is typically memory-bound, this design allows DIFF V2 to achieve decoding speeds on par with standard Transformer.

---

Record: https://forck.live/items/1562-differential-transformer-v2
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md