Information Bandwidth in Reinforcement Learning
Source: https://yingru.io/rl-bandwidth Author: Yingru LI Date: 2025-10-04
Summary
Information-theoretic analysis of RL training for LLMs. Key question: how much “signal” (information about what to improve) is actually getting into the model per training step?
Key Claims
- Information bandwidth concept: the amount of useful information in each reward signal. A binary reward (correct/incorrect) has low bandwidth; a detailed rubric has high bandwidth.
- The bandwidth bottleneck: most RL training is information-bandwidth limited, not compute-limited. More GPUs don’t help if reward signals are sparse.
- High-bandwidth rewards: process reward models (PRMs) provide feedback at each reasoning step, not just the final answer. This dramatically increases information bandwidth.
- Practical implication: improving reward signal quality has more impact than scaling compute, up to a certain threshold.
- Connection to GRPO: GRPO’s group structure (comparing multiple rollouts) increases effective information bandwidth compared to single-rollout methods.
Concepts
- RL Infrastructure — reward signal quality as the core training bottleneck
- Reward Hacking — low-bandwidth rewards are more gameable
- Test-Time Compute — process rewards enable step-level test-time feedback