Frontier RL Is Cheaper Than You Think — Fireworks AI
Summary
Challenges the “mega-cluster” narrative for RL training. The key insight: between consecutive RL checkpoints, 98%+ of weights remain bit-identical in bf16. This means you can send compressed deltas (~2% of full model) instead of full 1TB checkpoints to the rollout fleet.
Key Claims
- 98% sparsity between RL checkpoints — post-training updates are extremely fine-grained, most parameters don’t cross the bf16 threshold between steps
- Delta compression cuts transfer by ~94% — a 1024 GiB checkpoint becomes ~20 GiB deltas
- This makes multi-region rollout practical — no need for trainer and inference on the same RDMA fabric
- Async RL trades off-policy staleness for efficiency — a few minutes of policy lag is acceptable if it keeps training and rollout overlapped
- The mega-cluster story is wrong — frontier RL is accessible to teams with fragmented capacity across regions/clouds
Why It Matters
Democratizes frontier RL. If you don’t need a single giant cluster, more teams can compete on algorithms and product rather than infrastructure economics. The bottleneck shifts from hardware concentration to engineering insight.