Three Self-Distillation Papers in One Week
Source: Tweet + arxiv papers Author: @novasarc01 Date: 2026-01-29
Summary
Tweet announcing a convergence of self-distillation papers from January 2026 — multiple groups simultaneously publishing on training models from their own outputs.
Papers Mentioned
- “Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models” (arxiv.org/abs/2601.18734)
- “Self-Distillation Enables X” (arxiv.org/abs/2601.19897)
Key Claims
- Self-distillation convergence: multiple groups independently discovering that models can improve by learning from their own high-quality outputs.
- The mechanism: generate outputs → filter for high quality → train on those outputs → repeat.
- This is the “on-policy” variant of on-policy distillation applied to reasoning specifically.
- Suggests self-distillation is becoming a standard technique in the post-training toolkit.
Concepts
- Synthetic Data — self-distillation generates its own training data
- RL Infrastructure — on-policy self-distillation as RL-adjacent technique
- Test-Time Compute — reasoning improvements from self-distillation