Autonomous Research
Definition
The use of AI agents to conduct research loops autonomously: form hypothesis → design experiment → execute → observe result → revise. The agent closes the research loop that humans typically mediate.
Current Examples (2026)
Karpathy’s autoresearch project:
- Agent edits
train.py, runs 5-minute GPU training, checks validation loss, keeps/discards changes - One overnight run: ~20 improvements, 11% reduction in time-to-GPT-2
Scaled autoresearch (SkyPilot):
- 16 GPUs, 910 experiments, 8 hours
- Emergent strategies: factorial search, heterogeneous hardware exploitation
- 9x faster than sequential to reach same best loss (2.87% val_bpb improvement)
Algorithm optimization (AutoEvolver):
- Three hard optimization problems: matched or beat published SOTA from purpose-built evolutionary systems
- No evolutionary scaffolding — just problem + initial solution + evaluator
Key Dynamics
Aspiration prompting as unlock: Agents satisfice — they stop at local optima. A one-sentence prompt (“the SOTA is X, I believe you can beat it”) causes strategy shifts, not just longer search. 7x improvement observed after single-sentence intervention.
What’s automatable now: Narrow, quantifiable objectives with fast feedback loops (minimize loss, maximize score, pass unit tests). The agent can play directly on metrics.
What isn’t automatable yet (Nathan Lambert):
- The gap between “on-paper accurate model” and “production-ready aligned system”
- Organizational tacit knowledge; deep intuitions about what matters
- Multi-task alignment: optimizing one metric often trades off against others
Lossy vs Recursive Self-Improvement
Lambert argues we’re seeing lossy self-improvement, not recursive. The loop is real but friction at every step prevents exponential acceleration:
- Automatable research is too narrow
- Complexity grows faster than AI can handle it
- Human oversight requirements don’t disappear