Attention Normalizes the Wrong Norm
Source: https://convergentthinking.sh/posts/attention-normalizes-the-wrong-norm/ Author: unknown (convergentthinking.sh) Date: 2025
Summary
Standard softmax attention uses L1 normalization (weights sum to 1), but preserving output variance requires L2 normalization (weights’ L2 norm = 1). As sequence length grows, L1 normalization causes output variance to deteriorate, forcing models to learn position-specific compensations that don’t generalize to longer sequences. Proposed fix: “p-softmax” with p=2.
Key Claims
- Softmax = L1 normalization of attention weights
- L1 normalization: output magnitude depends on sequence length → variance deterioration at longer sequences
- Variance preservation requires L2 norm of attention weights = 1
- Consequence: models learn length-specific adjustments during training that don’t generalize
- Fix: p-softmax (normalize to Lp norm); p=2 preserves variance automatically
- Advantage over Scalable Softmax: L2 adapts automatically vs. fixed log N correction factor
- Demonstrated in counting experiments: better length generalization
Concepts
- Scaling & Compute — architectural fix for long-context generalization
- Context Engineering — attention mechanism shapes what context is used