On-Policy Distillation for Multi-Turn Tool Use
Source: https://x.com (tweet) Author: unknown (research team) Date: 2025-2026
Summary
On-policy distillation would revolutionize multi-turn tool-use training, but neither Tinker nor TRL supports multi-turn on-policy distillation (only single-turn). Team implemented their own solution. Key insight: on-policy distillation requires the student to generate from its own distribution.
Key Claims
- On-policy distillation critical for multi-turn tool-use training
- Gap in tooling: Tinker and TRL only support single-turn on-policy distillation
- Team built their own multi-turn on-policy distillation implementation
- Validates on-policy distillation theory with a practical implementation gap
Connection to Other Sources
Directly implements On-Policy Distillation concept in multi-turn settings.
Concepts
- RL Infrastructure — on-policy distillation tooling gap
- Synthetic Data — on-policy data generation for distillation