Efficient Benchmarking of AI Agents

Franck Ndzomga

cs.AIMar 24, 2026

Efficient Benchmarking of AI Agents

Authors: Franck Ndzomga

Abstract

Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings at substantially lower cost. Unlike static language model benchmarks, agent evaluation is subject to scaffold-driven distribution shift, since performance depends on the framework wrapping the underlying model. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurations, we find that absolute score prediction degrades under this shift, while rank-order prediction remains stable. Exploiting this asymmetry, we propose a simple optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates (30-70%). This mid-range difficulty filter, motivated by Item Response Theory, reduces the number of evaluation tasks by 44-70% while maintaining high rank fidelity under scaffold and temporal shifts. It provides more reliable rankings than random sampling, which exhibits high variance across seeds, and outperforms greedy task selection under distribution shift. These results suggest that reliable leaderboard ranking does not require full-benchmark evaluation.

Figures & tables

Figure 1: The Robustness Gap. Spearman

\rho

(ranking) and

R^{2}

(score prediction) across evaluation regimes. Ranking fidelity remains robust even as absolute score prediction rapidly degrades under temporal, scaffold and random shift.

Benchmark	Agents	Tasks	Scaffolds	Avg task $\rho$
CoreBench Hard	38	45	2	0.10
GAIA	32	165	2	0.15
Mind2Web	22	300	2	0.03
SciCode	24	65	2	0.12
SWE-bench Verified	28	50	2	0.29
TAU-bench Airline	26	50	2	0.08

Table 1: Benchmark statistics. Avg task

\rho

: mean pairwise Spearman correlation between task outcome vectors across agents; higher values indicate a more homogeneous benchmark.

Benchmark	Tasks	Agents	Med. $/run	Max $/run	Med. $/task	Range $/task
SWE-bench Verified	50	33	$163	$1600	$3.26	$0.08–32.00
CoreBench Hard	45	45	$66	$510	$1.47	$0.05–11.33
GAIA	165	32	$140	$2829	$0.85	$0.05–17.14
Mind2Web	300	22	$276	$1610	$0.92	$0.02–5.37
SciCode	65	33	$67	$625	$1.03	$0.00–9.62
TAU-bench Airline	50	26	$22	$180	$0.44	$0.01–3.61

Table 2: Per-run evaluation costs on HAL benchmarks. Costs reflect API fees and vary with model and scaffold.

Protocol	Best	Avg	Best	Avg	Best	Avg	Benchmarks
	$\rho$		$\tau$		$R^{2}$
LOAO	0.95	0.91	0.85	0.79	0.90	0.74	8
Within-Scaffold LOAO	0.96	0.86	0.87	0.74	0.89	0.65	8
Temporal	0.90	0.85	0.81	0.73	0.71	0.52	8
LOSO	0.92	0.87	0.80	0.73	0.65	0.50	7
Random 20%	0.90	0.86	0.84	0.79	0.54	0.46	8

Table 3: Rank prediction (

\rho

\tau

) vs. score prediction (

R^{2}

) across evaluation protocols. ”Best” selects the top-performing task selection method per benchmark; ”Avg” averages across methods.

Figure 2: Range of Performance by Selection Strategy. Mean, best-case, and worst-case Spearman

\rho

across overlapping benchmarks and evaluation protocols. Mid-Range task selection offers the overall best ranking preservation across benchmarks.

Figure 3: Performance Stability Across Distribution Shifts. Average Spearman

\rho

by task selection strategy, disaggregated across four evaluation protocols (LOAO, LOSO, Random 20%, and Temporal shift). Mid-Range selection maintains reliable rankings (

>0.85

mean

\rho

) regardless of the evaluation regime.

Benchmark	$k$	MR	Greedy	Easiest	Stratified	$\mu\pm\sigma$	Worst / Best
						Random
TerminalBench	39	0.980	0.988	0.970	0.979	0.980 $\pm$ .003	0.962 / 0.991
GAIA	77	0.946	0.927	0.928	0.930	0.938 $\pm$ .015	0.872 / 0.987
SWE-bench Mini	20	0.920	0.881	0.922	0.902	0.903 $\pm$ .026	0.695 / 0.977
USACO	81	0.938	0.886	0.861	0.914	0.924 $\pm$ .033	0.700 / 0.995
$\tau$ -bench Airline	27	0.944	0.834	0.920	0.874	0.860 $\pm$ .038	0.708 / 0.957

Table 4: Per-benchmark mean

\rho

across all evaluation protocols. Random reports mean

\pm

std over 100 seeds per protocol; Worst/Best denote extreme seed outcomes across all protocols.

				MR $\rho$			Cost Savings
Benchmark	$N$	$k_{\mathrm{MR}}$	Reduction	LOAO	LOSO	Temp.	Median	Range
TerminalBench	89	39	56%	0.986	0.984	0.975	—	—
GAIA	165	82	50%	0.956	0.926	0.974	$70.3	$40.1–$ 237.7
SWE-bench Mini	50	21	58%	0.924	0.896	0.968	$94.5	$37.9–$ 234.0
USACO	307	93	70%	0.940	—	0.950	$39.3	$19.6–$ 55.8
$\tau$ -bench Airline	50	28	44%	0.967	0.956	0.945	$9.8	$5.2–$ 22.4

Table 5: Benchmark reduction from mid-range filtering. Task reduction is the fraction eliminated; cost savings (per agent, per benchmark) assume linear scaling with task count, using median agent cost from HAL leaderboards. TerminalBench lacks HAL-style cost data. USACO LOSO is unavailable because all but one agent use a single scaffold. SciCode excluded due to insufficient mid-range tasks (

k=4

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Post-hoc sensitivity analysis of mid-range bands. The average Spearman rank correlation

\rho

(blue, left axis) and average task reduction percentage (red, right axis) across seven benchmarks for increasingly narrow mid-range bands. Shaded regions denote

\pm 1

standard deviation across the benchmarks. The 30–70 band, chosen a priori , strikes a highly favorable balance.

Figure 6: Task Selection Strategy $\times$ Benchmark (Mean $\rho$ Across Protocols). Each cell shows the mean Spearman $\rho$ across evaluation protocols; worst-case $\rho$ is shown below in gray. Asterisks mark the best-performing strategy per benchmark.

Figure 7: Why Easiest -

k

\approx

MR : Task Difficulty Distributions. Each panel shows the task pass-rate density for one benchmark, with Easiest -

k

(green) and MR (orange) task sets overlaid. Dashed lines mark the 30–70% mid-range band. Panels are ordered by decreasing task overlap between the two strategies.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 502 papers

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
Sep 30, 2026Michael Hardy, Ruhana Azam, Anka Reuel +2AI Agent Evaluation Agent Evaluation
Stanford University · University of Illinois Urbana-Champaign · Senior Author
EvalResearchBench: Can AI Agents Design Their Own Evaluations?
Oct 3, 2026Yaolun Zhang, Tianyi Xu, Yujie Zhao +3Automated Evaluation Benchmark Construction
Oregon State University · AG2 AI · University of California, San Diego +1
A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
May 27, 2026Tomer Keren, Nitay Calderon, Asaf Yehudai +3Computer-Use Agent Benchmarks Tool-Use Evaluation
Faculty of Data and Decision Science Technion · IBM Research

Sep 30, 2026J/K move · Enter open · S save

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer+1

Stanford University · University of Illinois Urbana-Champaign · Senior Author

AI Agent Evaluation Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

Abstract

AI Agent Evaluation

Explore similar work

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

EvalResearchBench: Can AI Agents Design Their Own Evaluations?

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks