GRPO Group Normalization Kills Training Signals
Source: https://x.com (tweet) Author: unknown (researcher) Date: 2025-2026
Summary
Paper breakdown: GRPO’s group normalization kills valuable training signals when computing reward advantages between rollouts. The paper identifies this problem mathematically and designs a fix that scales well.
Key Claims
- GRPO’s group advantage normalization destroys signal quality between rollout groups
- When all rollouts in a group are correct (or all incorrect), the normalized advantage goes to zero — wasted compute
- Paper proposes a fix that preserves cross-group signal
- The fix scales well — not just a patch for small-scale experiments
Connection to Other Sources
Relates to GRPO++ engineering tricks and Cursor Composer 2 GRPO modifications (which also removes length normalization).
Concepts
- RL Infrastructure — normalization choices as critical implementation detail
- Reward Hacking — normalization failure as signal degradation