Reinforcement Learning Infrastructure
Overview
The systems engineering challenge of running RL training at scale. Distinct from pretraining infrastructure because RL has two jobs: training (forward/backward/update) and rollout generation (inference on current policy).
Key Insight: Delta Compression
Between consecutive RL checkpoints, 98%+ of bf16 weights are bit-identical. This means:
- Ship ~2% of the model per update instead of 100%
- Cross-region synchronization becomes practical over ordinary networks
- No need for trainer and rollout fleet on the same RDMA fabric
This fundamentally challenges the “mega-cluster” narrative. Source
Async RL
Trade-off: accept a few minutes of policy staleness for much better compute utilization. The delta compression makes this practical by keeping policy movement small and fast.
Implications
If frontier RL doesn’t require mega-clusters, the competitive landscape changes. Teams with fragmented capacity across regions can compete. The bottleneck shifts to algorithms and product, not hardware concentration.
Connections
Vault Notes
- Flow Post-Training Research — Alex’s practical RL/DPO/GRPO work at Kaon/Flow
- Applied Research Engineer JD — Flow’s RL-focused hiring requirements
- 1-Day RL Fundamentals Study Plan — Personal study plan for PPO, GRPO, DAPO