Measuring the Performance of Our Models on Real-World Tasks
Source: https://openai.com/research/measuring-model-performance-on-real-world-tasks Author: OpenAI Date: 2025-09-26
Summary
OpenAI’s framework for evaluating models on real-world tasks rather than academic benchmarks. Motivated by the growing gap between benchmark performance (improving) and real-world utility (harder to measure).
Key Claims
- Benchmark saturation: standard benchmarks (MMLU, HumanEval) are saturating — frontier models score near-ceiling, making differentiation difficult.
- Real-world task eval: OpenAI developed 50 tasks drawn from actual user requests — coding projects, business analysis, research questions.
- Key finding: benchmark rankings and real-world task rankings don’t always align. Models that excel on benchmarks don’t always excel on real tasks.
- The difficulty gap: real tasks are harder than benchmarks because they’re ambiguous, multi-step, and require judgment.
- Takeaway: model selection should be based on real-world task performance, not academic benchmarks.
Entities
Concepts
- Coding Agents — coding tasks are central to their real-world eval
- Reward Hacking — benchmark saturation = models gaming narrow evals