Differential Transformer V2

Microsoft introduces DIFF V2, a differential attention mechanism that doubles query heads without increasing key-value heads, enabling faster decoding and eliminating the need for custom attention kernels, while removing per-head RMSNorm to improve numerical stability.

From the source
DIFF V2 introduces additional query heads compared to the baseline Transformer, but does not increase the number of key-value (KV) heads. Since LLM decoding is typically memory-bound, this design allows DIFF V2 to achieve decoding speeds on par with standard Transformer.
huggingface.co

Visit source ↗huggingface.co

Announced: 20 Jan 2026 03:20 UTC
Detected: 03 Aug 2026 21:12 UTC (+195d after)

Other stories

Hugging Face · 28 AugThe Open ASR Leaderboard Adds Its First Global South LanguageCapability change

Google Research · 27 AugPlanetary prediction engine: Automating global models via Earth AIResearch paper

OpenAI · 27 AugBetter answers, broader thinking: What students gain from ChatGPT and critical-thinking trainingResearch paper

LG AI Research · 27 Aug

[ACL 2026] ReSQL: A Self-Improving Text-to-SQL Framework through Retrieval-Augmented Error Reasoning

Research paper

xAI · 29 AugGrok Bot now works with XCapability change

GitHub · 28 AugGitHub Copilot in Visual Studio — August update