DFT：监督微调（SFT）= 某种强化学习（RL）？
Source: https://zhuanlan.zhihu.com/p/… Author: Chinese ML researcher (知乎) Date: 2025-09-16
Summary
Mathematical argument that supervised fine-tuning (SFT) is equivalent to a specific form of policy gradient with a degenerate reward function. Proposes DFT (Distributionally Fine-Tuning) as an improvement.
Key Claims
- The SFT-RL equivalence: SFT can be written as a policy gradient where the reward is 1 for tokens matching the training data and 0 elsewhere.
- Problem with this reward: the 1/0 reward is pathological — it creates sharp gradient cliffs and overfits to exact training tokens.
- DFT proposal: replace the binary reward with a softer, distributional reward that allows some variability around training tokens.
- Practical result: DFT models show better generalization than SFT while maintaining similar on-distribution performance.
- The implication: if SFT is just a bad form of RL, then improving SFT requires thinking about reward design, not just data quality.
Concepts
- RL Infrastructure — SFT/RL unification; reward design implications
- Synthetic Data — training data quality as reward quality