Scaling Laws & Compute
Overview
The relationship between compute investment and AI model capability. Encompasses pretraining scaling laws (Chinchilla), post-training/RL scaling, inference-time compute, and the economics of AI infrastructure.
Key Insights from Sources
- RL scaling doesn’t require mega-clusters — delta compression of weight updates makes distributed RL practical. The “you need a single giant cluster” narrative is wrong for RL (even if true for pretraining). Source
Open Questions
- Where does the next 10x capability come from? More pretraining compute, better RL, inference-time compute, or something else?
- How do scaling laws change for post-training vs pretraining?