From the source
From the source
Microsoft introduces DIFF V2, a differential attention mechanism that doubles query heads without increasing key-value heads, enabling faster decoding and eliminating the need for custom attention kernels, while removing per-head RMSNorm to improve numerical stability.
From the source
DIFF V2 introduces additional query heads compared to the baseline Transformer, but does not increase the number of key-value (KV) heads. Since LLM decoding is typically memory-bound, this design allows DIFF V2 to achieve decoding speeds on par with standard Transformer.
huggingface.co
