The Continual Learning Problem
Source: https://jessylin.com/2025/10/22/the-continual-learning-problem/ Author: jessylin.com Date: 2025-10-22
Summary
Overview of the continual learning problem in neural networks — how models catastrophically forget previously learned information when trained on new data. Relevant for agents that need to learn from experience without losing prior capabilities.
Key Claims
- Catastrophic forgetting: when a neural network is fine-tuned on new data, it overwrites weights that encoded previous knowledge — the model forgets old tasks.
- Why it matters for agents: agents that learn from user interactions need to update their behavior without forgetting their core capabilities.
- Approaches: (1) replay (store old examples and mix with new), (2) elastic weight consolidation (protect weights important for old tasks), (3) progressive networks (add capacity, don’t overwrite).
- The alignment tension: continual learning from user feedback could cause drift from alignment training — models optimize for recent interactions.
- Current state: continual learning is solved for narrow distributions; remains open for general multi-task settings.
Connection to Other Sources
Fundamental challenge underlying Factory Signals and recursive language models. The latter assumes the forgetting problem is managed.
Concepts
- Agent Memory — continual learning is the memory problem for model weights
- RL Infrastructure — RL-based continual learning approaches
- AI Alignment — forgetting alignment training is an alignment risk