Deep Neural Nets: 33 Years Ago and 33 Years From Now
Source: https://karpathy.github.io/2022/03/14/lecun1989/ Author: Andrej Karpathy Date: 2022
Summary
Karpathy recreates LeCun’s 1989 handwritten digit network with modern tools (PyTorch, auto-grad), comparing what was hard vs. trivially easy 33 years later. Uses this as a lens to predict what 2056 might look like for today’s networks.
Key Claims
- LeCun 1989: trained a neural network in Lisp with manually-computed gradients, took days
- Same experiment with PyTorch 2022: minutes, 10 lines of code
- Insight: the algorithms haven’t changed much — the tooling and scale have
- Prediction: 2056 will look back at 2022’s manual prompt engineering and custom architectures as similarly primitive
- The “bitter lesson” applied: compute + simple algorithms win over clever human engineering
Entities
Concepts
- Scaling & Compute — historical perspective on compute progress