SFT vs RL as Forward/Reverse KL Divergence
Source: https://x.com (tweet) Author: unknown (ML researcher) Date: 2025-2026
Summary
Derivation showing SFT and RL training objectives are equivalent to forward and reverse KL divergence respectively. Commonly cited in papers but the derivation is often non-trivial. Fills a gap in most practitioners’ understanding.
Key Claims
- SFT objective = forward KL divergence minimization
- RL objective = reverse KL divergence minimization
- The divergence implies: SFT fits to the target distribution; RL avoids the target’s mass-covering
- Common citation but rarely clearly derived — this fills the gap
- Notation: p* = target distribution, p = current policy
Concepts
- RL Infrastructure — theoretical foundation of SFT vs RL
- Synthetic Data — understanding which distribution to fit matters for synthetic data generation