Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond In-Context Reasoning?
Source: https://arxiv.org/abs/… Author: Zhiqi Chen et al. Date: 2025-04-23
Summary
Skeptical paper questioning whether RL-trained reasoning models have genuinely improved reasoning capacity, or whether they’ve just learned to format reasoning chains better while relying on the same underlying capabilities.
Key Claims
- The question: does RL training actually improve reasoning ability, or does it improve reasoning-like presentation?
- Experiment: take an RL-trained reasoning model, strip its chain-of-thought, and evaluate it against the base model. If reasoning improved, the base capability should be higher too.
- Finding: mixed results — some genuine improvement in base capability, but much of the benchmark gain comes from extended thinking format, not improved reasoning.
- The implication: RL-trained models that “show their work” aren’t necessarily better reasoners — they’re better at displaying reasoning.
- Counter-evidence: some reasoning errors only appear in the thinking process, which wouldn’t exist if reasoning were superficial.
Connection to Other Sources
Important skeptical counterpoint to claims in Lilian Weng’s thinking survey and GRPO++.
Concepts
- Test-Time Compute — does test-time reasoning actually improve capability?
- RL Infrastructure — RL effectiveness is the question
- Reward Hacking — format gaming is a form of reward hacking