Agent Benchmarks Stop Ranking Before They Run Out of Tasks
Organizations: Lenovo Group · Marquette University · Cornell University
Abstract
Data-science and ML-engineering agents are evaluated on benchmarks such as MLE-bench, ALE-Bench, SWE-fficiency and AlgoTune, whose tasks yield a quantity (a Kaggle metric, contest score or speedup). When one stops separating the best models, the usual remedy is a harder version. Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, we find much of this saturation lies in the measurement, not the tasks. Most agent benchmarks reduce a run to one pass/fail bit, which fails in two ways. It is too coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than its tests' errors, and correcting them reorders neighbours no more than deleting a random sixth of each agent's passes. It is capped at the reference: ALE-Bench's top ten, separated by contest score, tie when graded against the median human, although nearly half the runs beat that human. On fixed tasks, a higher bar only moves the problem; graded credit fixes it. We derive a clipped log-ratio score with dated anchors and cost as a separate axis. In a Terminal-Bench re-run, pass/fail-tied models differ over tenfold on the quantity the tests discard.
Figures & tables
| Benchmark | Models | Tasks | Beat ref. | Official | Answer key | Capped | Objective |
|---|---|---|---|---|---|---|---|
| ALE-Bench | 145 | 40 | 47.8% | 3 (0.07) | 1 (1.00) | 1 (0.93) | 3 (0.03) |
| SWE-fficiency | 7 | 430 | 36.3% | 1 (0.95) | 4 (0.10) | 5 (0.01) | 4 (0.02) |
| AlgoTune | 18 | 154 | 70.6% | 2 (0.51) | 2 (0.63) | 2 (0.88) | 3 (0.58) |
| METR † | 20 | 210 | 3.7% | = key | 3 (0.51) | 3 (0.55) | n/a ∗ |
| MLE-bench † | 27 | 74 | 10.8% | = key | 4 (0.10) | 3 (0.52) | 3 (0.73) |
| Earlier models (46) | Later models (95) | ||||||
|---|---|---|---|---|---|---|---|
| Score | Tiers ( ) | Human | Zero | Tiers ( ) | Human | Zero | Split-half |
| Bar: median human (key) | 1 (0.94) | 0.89 | 9% | 1 (1.00) | 0.91 | 0% | 0.81 |
| Bar: contest winner | 1 (1.00) | 0.43 | 89% | 2 (0.08) | 0.61 | 73% | 0.66 |
| Bar: tuned on earlier | 2 (0.41) | 0.88 | 0% | 1 (0.85) | 0.81 | 0% | 0.78 |
| Per-task bars, calibrated | 1 (0.95) | 0.89 | 4% | 1 (0.85) | 0.75 | 0% | – |
| Graded (6 bars per task) | 2 (0.94) | 0.92 | 0% | 3 (0.15) | 0.95 | 0% | – |
Agent Benchmarks
Agent benchmarks measure how well AI systems plan, use tools and memory, and complete tasks in interactive environments.
See all 130 papers