let-verify-step-by-step

Let’s Verify Step by Step

Source: https://arxiv.org/abs/2305.20050 Author: Lightman et al. (OpenAI) Date: 2024-09-14 (saved)

Summary

OpenAI paper introducing process reward models (PRMs) — reward models that evaluate each step of a reasoning chain rather than just the final answer. Foundational for step-level feedback in RL training.

Key Claims

• Outcome reward models (ORM): score only the final answer — binary correct/incorrect.
• Process reward models (PRM): score each intermediate reasoning step — richer feedback signal.
• Key finding: PRM training significantly improves math problem solving vs. ORM — especially on hard problems requiring multiple steps.
• Why: ORM can only distinguish correct from incorrect solutions; PRM identifies exactly where reasoning went wrong.
• Data requirement: PRM requires step-level labels, which are expensive to collect (human annotators must label each step).
• This paper justified the investment in step-level labeling at OpenAI.

Connection to Other Sources

PRMs are the theoretical foundation for RL Information Bandwidth (high-bandwidth rewards). They enable step-level reward in POLARIS.

Entities

• OpenAI

Concepts

• RL Infrastructure — PRM as high-quality reward model
• Test-Time Compute — PRMs enable test-time verification of reasoning steps
• Reward Hacking — PRMs reduce gaming compared to ORMs