Pretraining is Data-Inefficient: KV Cache Proposal
Source: Tweet Date: 2026-04-01
Summary
Insight tweet arguing that pretraining is data-inefficient because KV caches are discarded after every forward-backward step. Proposes integrating KV cache compaction into pretraining as a path to dramatically better data efficiency.
Key Claims
- The inefficiency: during pretraining, the model sees each token in its training context only once (per pass). The KV cache that represents that context is then discarded.
- The proposal: if you could compress and retain KV caches across training steps, the model could leverage richer context from past tokens.
- Analogy: humans don’t forget everything they’ve read when they learn something new. The pretraining paradigm does.
- Expected benefit: potentially dramatic improvements in data efficiency, at the cost of more complex training infrastructure.
Concepts
- Scaling & Compute — data efficiency as compute efficiency
- RL Infrastructure — KV cache management in training