Meta-Harness: Automatic Agent Harness Optimization
Source: https://yoonholee.com/meta-harness/ Author: Yoon Ho Lee et al. Date: 2025
Summary
System for automatically optimizing agent harnesses (prompts, tool definitions, control logic) by giving the optimizer access to full source code, scores, and execution traces of every prior candidate — 10M tokens of diagnostic context per step. Ranks #2 among Claude Opus agents on TerminalBench-2 (76.4% pass rate).
Key Claims
- Novel approach: optimizer gets 10M tokens of diagnostic context (vs 26K for competing methods)
- Full access to source code, scores, execution traces of all prior candidates — not just summaries
- Text classification: 48.6% accuracy vs 40.9% for prior work, using 4x fewer context tokens
- Math reasoning: +4.7 percentage points across 5 held-out models from a single retrieved strategy
- TerminalBench-2: #2 Claude Opus (76.4%), #1 Haiku (37.6%)
- Key insight: diagnostic richness enables more targeted optimization than aggregate metrics
Connection to Other Sources
Directly relevant to Anthropic’s agent eval framework and effective harnesses. Meta-Harness automates what those articles describe as manual harness design.
Concepts
- Coding Agents — TerminalBench-2 performance
- Context Engineering — harness as context configuration artifact