A Recipe for Training Neural Networks
Source: https://karpathy.github.io/2019/04/25/recipe/ Author: Andrej Karpathy Date: 2019 (evergreen)
Summary
Practical guide to training neural networks, focused on debugging and systematic iteration. Karpathy’s famous “recipe” — become one with the data, overfit one batch first, regularize, tune hyperparameters. Anti-pattern: trusting default settings and hoping the network trains.
Key Claims
- Become one with the data: visualize, understand distributions, find outliers
- Overfit one batch first — verify the model can learn at all before scaling up
- Incremental complexity: start simple, add one thing at a time
- The most common failure mode: trusting that the network is doing what you think
- Explicit advice against “default settings and prayer” approach to training
Entities
Concepts
- RL Infrastructure — training debugging applies equally to RL