GRPO++: Tricks for Making RL Actually Work
Source: https://cameronrwolfe.substack.com/p/grpo-tricks Author: Cameron R. Wolfe, Ph.D. Date: 2026-01-23
Summary
Practical guide to making GRPO (Group Relative Policy Optimization) work reliably for training reasoning models. GRPO is the RL algorithm behind DeepSeek-R1 and many subsequent reasoning models. This post covers the engineering tricks that turn theory into practice.
Key Claims
- GRPO compares groups of rollouts (not pairs like DPO) — this makes it more stable than PPO for LLM fine-tuning.
- Critical tricks: (1) normalize rewards within groups, not globally; (2) clip KL divergence to prevent model collapse; (3) use length normalization to avoid verbosity bias.
- Verbosity hack: without length normalization, models learn that longer answers get higher rewards regardless of quality.
- Cold-start problem: GRPO needs a decent base policy; starting from raw pretrained weights leads to reward collapse.
- Batch size matters more than learning rate — larger batches improve diversity of rollouts, stabilizing training.
- The “RL gimmick” critique: GRPO can produce models that look reasoning-capable but have just learned to pattern-match on reasoning-format text.
Entities
- Cameron R. Wolfe — ML researcher, Cameron’s AI Newsletter
Concepts
- RL Infrastructure — GRPO implementation details
- Reward Hacking — verbosity bias is a form of reward hacking
- Test-Time Compute — reasoning models trained with GRPO
- Synthetic Data — rollout generation as synthetic data