Scaling Karpathy’s Autoresearch: What Happens When the Agent Gets a GPU Cluster
URL: https://blog.skypilot.co/scaling-autoresearch/
Author: Alex Kim, Romil Bhardwaj (SkyPilot)
Published: 2026
Summary
The SkyPilot team gave Claude Code access to 16 GPUs and let it run Karpathy’s autoresearch project autonomously. Over 8 hours it ran ~910 experiments, improved validation loss by 2.87%, and discovered emergent research strategies impossible with single-GPU sequential search.
Key Claims
- 910 experiments in 8 hours across 16 GPUs — equivalent to ~72 hours sequential (9x speedup to reach same best loss).
- Sequential bottleneck → factorial search: With 1 GPU, agent does greedy hill climbing (one hypothesis per 5 minutes). With 16, it runs factorial grids of 10-13 experiments per wave, catching parameter interaction effects.
- Emergent hardware strategy: Agent discovered it had both H100s and H200s, and self-organized to screen ideas on cheaper H100s and validate winners on H200s. Nobody told it to do this.
- Architecture discovery phase was most valuable: Agent found that scaling model width mattered more than any single hyperparameter — insight requiring parallel search over multiple widths in one wave.
- Phases: Hyperparameter sweeps → architecture discovery → fine-tuning → optimizer tuning → diminishing returns. Each phase built on prior findings.
- Autoresearch setup: Three files —
prepare.py(read-only data),train.py(agent modifies),program.md(instructions). Agent minimizes val_bpb within 5-minute training budget. - SkyPilot as infrastructure: YAML-defined experiments launched across Kubernetes; agent manages its own cluster via
sky logsfor results.
Implications
- Removes the infrastructure bottleneck from AI research automation — the agent becomes the researcher if given adequate compute.
- Interaction effects between parameters (invisible to sequential search) are only discoverable with parallel experimentation.
- Heterogeneous hardware exploitation was not prompted — emergent behavior from the agent’s own reasoning.