Value Functions Are Underrated for LLM Agents
Source: https://x.com (tweet) Author: unknown (referencing John Schulman) Date: 2025-2026
Summary
Recalling John Schulman’s claim that “value functions are underrated.” For token-level LLMs, the horizon is huge, making naive token-level value functions high-variance. Key open problem: how to use value functions effectively for LLM agents.
Key Claims
- John Schulman: “value functions are underrated” — a recurring theme in RL for LLMs
- For token-level LLMs: huge horizon → naively placing value function on tokens = high variance
- Value functions could enable better credit assignment in long-horizon agent tasks
- Still an open problem: how to make value functions work for LLM agents
Connection to Other Sources
Complements Process Reward Models (step-level value functions) and Active Partial Rollouts (value-guided training efficiency).
Concepts
- RL Infrastructure — value functions as unresolved RL component