Policy Gradient Algorithms
Source: https://lilianweng.github.io/posts/2018-04-08-policy-gradient/ Author: Lilian Weng Date: 2025-10-04 (saved, originally 2018)
Summary
Lilian Weng’s comprehensive tutorial on policy gradient methods — the family of RL algorithms used to train language models (PPO, GRPO, etc.). The reference article for understanding the math behind RLHF.
Key Claims
- Policy gradient theorem: the gradient of expected reward with respect to policy parameters can be computed by sampling trajectories and weighting by their rewards.
- REINFORCE: the simplest policy gradient method — collect rollouts, compute rewards, update in the direction of high-reward actions.
- Actor-Critic: separate the policy (actor) from the value estimator (critic) for lower-variance gradient estimates.
- PPO: proximal policy optimization — clip the gradient update to prevent too-large policy changes, ensuring stability.
- Why this matters for LLMs: RLHF/GRPO are policy gradient methods applied to language generation. The math is the same; the scale is very different.
Entities
Concepts
- RL Infrastructure — foundational reference for RL algorithms
- Test-Time Compute — policy gradient underpins test-time RL