Direct Preference Optimization: Your Language Model Is Secretly a Reward Model
Source: https://arxiv.org/abs/2305.18290 Author: Rafailov et al. Date: 2025-08-13 (saved, originally 2023)
Summary
The DPO paper showing that the RLHF fine-tuning process can be simplified: instead of training a separate reward model and running RL, you can directly optimize the LLM on preference pairs. The LLM implicitly learns the reward function.
Key Claims
- RLHF requires: (1) train reward model from preferences, (2) run PPO/RL to optimize policy against reward model. Complex, unstable.
- DPO insight: the optimal RLHF policy has a closed-form solution expressible as a simple classification loss on preference pairs.
- DPO training: just provide (chosen, rejected) pairs; optimize a binary cross-entropy-like loss. No reward model, no RL loop.
- Practical win: DPO is simpler, more stable, and often competitive with PPO-based RLHF.
- Limitation: DPO requires offline preference data — can’t be used online (new rollouts) the way PPO can.
- Relationship to GRPO: DPO is offline; GRPO is online. Both avoid training a separate reward model.
Concepts
- RL Infrastructure — DPO as RL alternative
- AI Alignment — preference optimization as alignment technique
- Reward Hacking — DPO avoids some reward model over-optimization issues