Avatarl: Training Language Models from Scratch with Pure Reinforcement Learning
Source: https://tokenbender.com/avatarl Author: tokenbender.com Date: 2025-08-14
Summary
Experiment in training language models purely from RL — no pretraining on human text, no SFT, just RL from scratch. Tests whether RL alone can produce capable language models.
Key Claims
- Experiment: train a language model from random initialization using only RL (no pretraining).
- Finding: pure RL can produce coherent language models, but they require much more compute than supervised pretraining.
- The pretraining shortcut: pretraining on human text gives the model a massive head start — language structure, world knowledge, common patterns. RL would have to rediscover all of this.
- Where pure RL shines: once a base pretrained model exists, pure RL on top can reach capabilities that SFT cannot.
- Lesson: the right approach is hybrid — pretraining for knowledge/structure, RL for capability alignment.
Concepts
- RL Infrastructure — pure RL as a training approach
- Scaling & Compute — compute cost of pure RL vs. pretraining
- Synthetic Data — pretraining data is what makes the hybrid approach efficient