Measuring AI Ability to Complete Long Tasks
Source: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ Author: METR (Model Evaluation and Threat Research) Date: 2026-01-23
Summary
METR’s empirical study on AI models’ ability to complete long-horizon tasks. Key finding: AI task-completion ability has been doubling roughly every 7 months, measured by maximum task duration that models can complete reliably.
Key Claims
- Metric: “task horizon” — the longest task (in human-hours) that a model can complete with >50% success rate.
- 2024 data: frontier models could reliably handle tasks taking ~2-4 hours of human effort.
- Growth rate: approximately doubling every 7 months — faster than most other AI capability metrics.
- Implication: if this trend holds, models will handle week-long tasks by 2026, month-long by 2027.
- Hard tasks: those requiring sustained context, complex dependencies, or environmental interaction are the hardest.
- Interesting decomposition: most failures come from mid-task context loss or error cascades, not initial misunderstanding.
Connection to Other Sources
Provides empirical grounding for claims in Simon Willison’s cognitive debt piece — longer task horizons = more cognitive debt per session. Supports AutoEvolver results on algorithm optimization.
Entities
- METR — AI safety evaluation organization
Concepts
- Autonomous Research — task horizon directly measures research agent capability
- Coding Agents — primary task domain measured
- Scaling & Compute — capability scaling in terms of task horizon, not perplexity