RL Infrastructure Is Maturing into Systems Engineering
Thesis
The RL post-training stack is rapidly maturing from research code into proper systems engineering. The key optimizations are not ML innovations but distributed systems techniques — and the gains are enormous (4x for OLMo 3, saving $1.5M).
Evidence
The Optimizations Are Systems Work, Not ML Work
- Making RL Fast (OLMo 3): Continuous batching (+11%), inflight weight updates (+117%), better thread sync (+39%) — all standard distributed systems techniques applied to RL
- Fireworks: Delta compression for weight updates — network engineering for RL
- Cursor Warp Decode: GPU kernel engineering for MoE inference — memory bandwidth optimization, not algorithm improvement
- PufferLib 4.0: 20M steps/sec through infrastructure optimization
- MaxText/GPT-OSS: Compiler-driven optimization (XLA) vs manual kernel engineering — the “let the compiler handle it” approach
The Shared Pattern
Every optimization follows the same recipe:
- Identify the systems bottleneck (not the ML bottleneck)
- Apply standard distributed systems techniques
- Get 2-4x speedup
Finbarr’s KV cache reuse across weight updates is emblematic: “it works fine, handwavy epsilon-delta proof” — pragmatic systems engineering over ML perfectionism.
Hardware Divergence
The infrastructure is specializing:
- Prefill-optimized hardware (Rubin CPX) vs decode-optimized (Groq LPU)
- PrfaaS: cross-datacenter disaggregation enabled by hybrid attention
- Warp decode: output-centric parallelism for Blackwell MoE decode
- The “one GPU type fits all” era is ending
Implication
RL infrastructure is becoming a competitive moat. Teams that invest in systems engineering (async RL, continuous batching, inflight updates) can run 4x more experiments per dollar. This compounds: more experiments → better models → more users → more feedback signal.
Cursor is the archetype: they invest in both ML (RL from user feedback) and systems (warp decode, real-time RL pipeline), and the combination creates a flywheel.
Open Questions
- Will open-source RL infra (Open Instruct, MaxText) close the gap with proprietary stacks?
- Does the “pragmatic” approach (ignoring KV cache staleness, inflight updates) have hidden failure modes at larger scale?
- How much of the “frontier” is the model vs the infrastructure?
Connections
- Coding Agent Paradox — infrastructure maturation as partial resolution
- RL Infrastructure — the concept page
- Scaling & Compute — infrastructure efficiency as a scaling lever