Survey: RL vs Online Policy Distillation vs SFT
Author: @neural_avb | Date: 2026-05-01
Recommendation of a survey comparing reinforcement learning, online policy distillation (OPD), and supervised fine-tuning (SFT) for LLM alignment and capability improvement. Described as worth a full 30-minute read.
Key Claims
- Survey systematically compares RL, OPD, and SFT approaches
- Each method has distinct tradeoffs in sample efficiency, stability, and final quality
- Online policy distillation emerging as middle ground between RL and SFT
- Understanding when to use each is critical for LLM training pipelines
Takeaways
- The post-training landscape is more nuanced than “just do RLHF”
- OPD (distilling from a policy trained with RL) can capture RL benefits without RL instability
- SFT remains strong baseline but has ceiling effects
- Practitioners should read this to choose the right post-training strategy