Making RL Fast
Finbarr Timbers (AI2/OLMo 3) describes three optimizations that made their RL setup 4x faster, saving 750K H100 hours ($1.5M).
Three Optimizations
1. Continuous Batching (+11% throughput)
- Moved from synchronous (static batching) to streaming architecture
- Static batching wastes
(max_seq_len - mean_seq_len) / max_seq_lenof compute — 54% for OLMo 3 (14K avg, 32K max) - Stream examples in/out as previous completions finish
2. Inflight Updates (+117% throughput)
- Implements PipelineRL — pause generation for weight update, don’t drain the queue
- Does not invalidate KV cache after weight update — uses KV cache from older weights
- “It works fine” — handwavy epsilon-delta proof since neural networks compose continuous functions
- Avoids the static batching idleness problem during weight sync
3. Better Thread Synchronization (+39% throughput)
- Decouple actors to start/stop independently (no waiting for stragglers)
- Naive: stop all → update all → restart all (sequential barriers)
- Fixed: each actor stop/update/restart independently, concurrently
- Background prefetch thread to constantly refill inference queue
- Hit Amdahl’s law — synchronization points became the bottleneck as actors got more async
Results
| Configuration | Tokens/s | MBU |
|---|---|---|
| Baseline | 881 | 12.9% |
| + Continuous batching | 975 | 14.3% |
| + Better threading | 1,358 | 19.9% |
| + Inflight updates | 2,949 | 43.2% |
Takeaways
- All three optimizations are systems engineering, not ML research
- The KV cache reuse across weight updates is surprising and pragmatic
- Reference to Cursor Composer 2’s approach (weights to disk) as alternative to broadcast
- These are standard distributed systems techniques applied to RL
Connections
- RL Infrastructure — core topic
- Frontier RL Is Cheaper — delta compression for distributed RL
- Cursor Real-Time RL — similar async RL challenges
- Active Partial Rollouts — complementary RL efficiency technique