There Are No New Ideas in AI… Only New Datasets
Source: https://jxmo.io/posts/there-are-no-new-ideas-in-ai Author: Jack Morris Date: 2025-07-01
Summary
Provocative thesis: virtually all major AI advances in the past decade can be attributed to new datasets, not new algorithms. The same algorithms (transformers, attention, gradient descent) keep producing breakthroughs — because the datasets keep improving.
Key Claims
- Historical pattern: ImageNet → vision breakthrough; WebText → language breakthrough; GitHub code data → code capability; math datasets → reasoning breakthrough.
- Bitter lesson variant: where Sutton credits compute, Morris credits data. Both are saying “not human knowledge.”
- The implication: the next AI breakthrough is sitting in a dataset that hasn’t been properly processed yet.
- Evidence: ResNet (2015 architecture) trained on newer datasets outperforms many 2023 architectures on old datasets.
- Counter-argument: some genuinely algorithmic innovations (attention, LoRA) changed what was possible — not just data.
Connection to Other Sources
Complements Bitter Lesson (compute ≫ human knowledge) with a data-centric version. Both are skeptical of architectural cleverness as the primary source of progress.
Concepts
- Synthetic Data — if datasets drive progress, synthetic data is the next lever
- Scaling & Compute — data and compute are the two scaling dimensions