Meta-Harness: End-to-End Optimization (Updated)
Updated results page for Meta-Harness (Yoonho Lee et al.). Extends original page with new benchmarks on text classification, math reasoning, and agentic coding.
Key New Results
- Text Classification: 48.6% vs ACE’s 40.9% (+7.7 points) using 4x fewer context tokens on GPT-OSS-120B
- Math Reasoning: +4.7 points average across 5 held-out models on IMO-level problems, with a single discovered retrieval harness
- TerminalBench-2: 76.4% on Opus 4.6 (#2 overall), 37.6% on Haiku 4.5 (#1 overall)
- Matches OpenEvolve/TTT-Discover accuracy with 10x fewer evaluations
What Makes It Different
- 10M tokens of diagnostic context per optimization step vs ≤26K for all prior methods
- Proposer (Claude Code) navigates a full filesystem of prior candidates’ source code, execution traces, and scores via grep/cat
- The key insight: harness failures are hard to diagnose without raw execution traces; compressing to scores/summaries loses critical information
- Filesystem-based interface means the agent reads what it needs, not a fixed prompt window
Comparison Table
| Method | History | Mtok/iter |
|---|---|---|
| Self-Refine | Last | 0.001 |
| OPRO | Window | 0.002 |
| AlphaEvolve | Window | 0.022 |
| TTT-Discover | Window | 0.026 |
| Meta-Harness | Full | 10.0 |
Takeaways
- The “give the optimizer everything” approach works dramatically better than compression
- This validates the general principle: more context > better prompts, even for optimization
- Transfer across models: a harness discovered with one model improves others
- TerminalBench-2 results show practical agent improvement from harness optimization
Connections
- Meta-Harness (Original) — original page with foundational claims
- Harness Engineering — the concept being automated
- Context Engineering — 10M token context as key differentiator
- Coding Agents — TerminalBench-2 results