Are LLMs Not Getting Better?
URL: https://entropicthoughts.com/no-swe-bench-improvement
Author: kqr (Entropic Thoughts)
Published: 2026
Summary
A statistical analysis of METR’s data on LLM merge rates (pull request quality, not just test passage) in software engineering. Finds that a constant function fits the data better than a linear upward slope — suggesting LLMs haven’t improved in actual software engineering ability since early 2025.
Key Claims
- Two different success criteria: METR compared LLM code that “passes all tests” vs code that “would be approved by a maintainer.” The 50% horizon drops from 50 minutes to 8 minutes under the stricter criterion.
- Merge rate = real measure: Test passage is gameable; maintainer approval is the actual bar. This is what the author focuses on.
- Statistical finding: Fitting merge rate data:
- Gentle upward slope: Brier score 0.0129
- Piecewise constant: Brier score 0.0117
- Fully constant function: Brier score 0.0100 (best fit)
- Implication: The fully constant model (no improvement) fits better than assuming any improvement since early 2025.
- Caveat: Post-script acknowledges possible step-up with newer Anthropic and Google models after the dataset ends, but notes similar claims were made throughout 2025 and didn’t hold.
- The buzz gap: “The gap between buzz and actual performance was larger than we thought” during 2025.
Implications for Alex
This is a counterweight to hype. Benchmark leaderboard position ≠ real coding improvement. If Alex is evaluating coding agents for product decisions, merge rate (or equivalent “would a real human accept this?”) is the right metric.