Autoresearch: Autonomous LLM Training Optimization
Source: https://github.com/karpathy/autoresearch Author: Andrej Karpathy Date: 2025
Summary
Framework for AI agents to autonomously optimize LLM training overnight. Agent modifies train.py, runs 5-minute training experiments, checks validation metric (bits per byte), keeps or discards changes, and repeats. ~12 experiments per hour. Human authors program.md to guide research strategy.
Architecture
prepare.py— data prep (unchanging)train.py— the single modifiable file (model, optimizer, training loop)program.md— human-authored instructions guiding agent modifications
Key Claims
- Fixed 5-minute time budget per experiment — hardware-agnostic, reproducible
- ~12 experiments/hour with autonomous iteration
- Human sets research direction via program.md; agent handles experimentation
- Metric: validation bits per byte (hardware-agnostic)
- Division of labor: human = strategy, agent = optimization
Connection to Other Sources
Direct implementation of SkyPilot’s autonomous research and Autonomous Research concept. Karpathy’s minimal reference implementation.
Entities
Concepts
- Autonomous Research — autonomous overnight LLM optimization
- Coding Agents — agent as autonomous researcher