On-Policy Distillation
Source: https://thinkingmachines.ai/blog/on-policy-distillation Author: Thinking Machines Lab Date: 2025-10-28
Summary
Technical post on on-policy distillation — training a smaller “student” model by having it learn from a larger “teacher” model’s live generations (not cached data). This approach maintains distribution alignment between teacher and student.
Key Claims
- Standard distillation problem: teacher generates a dataset offline, student learns from it. Problem: student’s distribution diverges from teacher’s, causing distribution mismatch.
- On-policy solution: student generates responses, teacher evaluates and corrects. Student always learns from its own distribution.
- Key benefit: the student learns to fix its own errors, not generic teacher demonstrations.
- Result: on-policy distillation produces smaller models that outperform offline-distilled equivalents on reasoning tasks.
- Connection to RL: on-policy distillation is similar to GRPO/PPO — the student is essentially doing RL with the teacher as the reward model.
Concepts
- Synthetic Data — on-policy data generation as alternative to static datasets
- RL Infrastructure — on-policy distillation as RL-adjacent technique
- Test-Time Compute — reasoning models benefit most from on-policy distillation