Preventing Entropy Collapse in Long RL Runs
Source: https://x.com (tweet) Author: unknown (RL researcher, community Q&A) Date: 2025-2026
Summary
Community Q&A on tricks to prevent entropy collapse during long LLM RL runs. Key trick: optimize Best-of-N (pass@N) rewards to maintain diversity. Thread soliciting community knowledge on other techniques.
Key Claims
- Entropy collapse is a real failure mode in long RL runs
- Primary trick: optimizing Best-of-N rewards (reward diversity, not just quality)
- Community knowledge is fragmented — practitioners share different techniques
- Open question: what other diversity-preservation tricks work at scale
Connection to Other Sources
Connects to RL Collapse and FP16 vs BF16 for RL.
Concepts
- RL Infrastructure — entropy preservation as RL stability technique