chinchilla-compute-optimal-training

Chinchilla: Compute-Optimal Large Language Model Training

Source: https://deepmind.com/research/publications/2022/an-empirical-analysis-of-compute-optimal-large-language-model-training Author: Hoffmann et al. (DeepMind) Date: 2022

Summary

The Chinchilla paper establishes the compute-optimal scaling law: for a given compute budget, model size and training tokens should scale equally. Models before Chinchilla were significantly undertrained. The 70B Chinchilla model matched or beat GPT-3 (175B) by training on 4x more tokens.

Key Claims

• Optimal relationship: model size ∝ training tokens for fixed compute
• Most large models (GPT-3, Gopher) were undertrained — too large for their token budget
• Chinchilla (70B) = Gopher (280B) in performance by using 4x more data
• “Chinchilla scaling law” became the dominant paradigm for model training decisions
• Implication: for a fixed compute budget, smaller models trained on more data outperform larger models on less data

Concepts

• Scaling & Compute — the foundational compute-optimal scaling result
• Synthetic Data — data quantity as the co-equal scaling axis with model size