Differential Transformer
Source: https://arxiv.org/abs/2410.05258 Author: Microsoft Research Date: 2024-10-09
Summary
Microsoft Research paper on the Differential Transformer — an attention mechanism that computes attention as the difference between two attention maps, reducing noise and improving focus on relevant tokens.
Key Claims
- Standard attention problem: attention weights spread across many tokens, including irrelevant ones (“attention noise”).
- Differential attention: compute two softmax attention maps and subtract them. The difference cancels noise, amplifying signal.
- Results: differential attention achieves better performance than standard attention at similar scale, particularly on long-context tasks.
- Efficiency: differential attention has the same computational complexity as standard attention.
- Implications: may reduce context rot by focusing attention more precisely on relevant information.
Concepts
- Mechanistic Interpretability — understanding how attention actually works
- Context Engineering — better attention = better context utilization