When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Gap
Source: https://yingru.notion.site/rl-collapse Author: Yingru Date: 2025-12-11
Summary
Technical deep-dive on “RL collapse” — when RL training of language models destabilizes and the model loses coherence. Specifically analyzes the training-inference gap as the root cause.
Key Claims
- RL collapse symptoms: model outputs become incoherent, reward drops suddenly, training diverges.
- Root cause: training-inference gap — the policy used for rollout generation differs from the policy being updated. When this gap grows too large, updates become unstable.
- The gap grows because: (1) large learning rates update the policy faster than rollouts can track, (2) long training runs accumulate distribution shift.
- Mitigation: (1) reduce learning rate, (2) more frequent rollout refreshes, (3) clip the KL divergence between old and new policy.
- Key insight: RL collapse is not random — it’s predictable from the ratio of rollout staleness to update size.
Connection to Other Sources
Technical complement to GRPO++ tricks and POLARIS recipe. Both mention KL constraints that prevent the collapse described here.
Concepts
- RL Infrastructure — RL stability is a core infrastructure challenge