Training Language Models to Self-Correct via Reinforcement Learning
Source: https://arxiv.org/abs/2409.12917 Author: Google DeepMind Date: 2024-09-21
Summary
DeepMind paper on training LLMs to self-correct their own outputs using RL. Key contribution: a training procedure where the model learns when to trust its first answer and when to revise it.
Key Claims
- Self-correction failure: naively prompting models to self-correct often makes outputs worse, not better — models second-guess correct answers.
- RL training approach: train the model to recognize when its first output is wrong (using verifiable tasks), then generate a correction.
- Key insight: self-correction must be learned with RL, not prompted — the model needs to develop calibrated metacognition.
- The “oracle” vs. “self-generated” correction distinction: training on oracle corrections is easier but doesn’t transfer; training on self-generated corrections builds true metacognition.
- Results: RL-trained self-correction improves performance by 15% on math benchmarks.
Concepts
- Test-Time Compute — self-correction is a test-time compute strategy
- RL Infrastructure — RL as the mechanism for learning metacognition
- Reward Hacking — overcorrection is a form of sycophancy/reward hacking