The Deadly Triad: RL Instability from First Principles
Source: https://helenqu.com/blog/posts/deadly_triad Author: Helen Qu Date: 2025-2026
Summary
The “deadly triad” in RL: function approximation + off-policy learning + bootstrapping. Any single component is fine; the combination breaks convergence guarantees. From an optimization perspective: the update matrix fails to maintain symmetric positive definiteness. Each component is essential for practical RL — challenge is stabilization, not elimination.
Key Claims
- Deadly triad: (1) function approximation, (2) off-policy learning, (3) bootstrapping
- Removing ANY one component individually leads to a converging system
- Instability is specifically from the three-way interaction
- Mathematical explanation: update matrix loses symmetric positive definiteness
- All three are essential for practical RL (scaling, data efficiency, computational tractability)
- Solution: stabilization techniques (clipping, target networks, etc.), not removing components
Connection to Other Sources
Connects to RL Collapse: Training-Inference Gap — the practical manifestation of deadly triad instability in LLM RL.
Concepts
- RL Infrastructure — mathematical foundation of RL instability