Agents Are Systems, Not Models: Rethinking Agentic Evaluation

Luis Wiedmann; Leander Girrbach; Cordelia Schmid; Zeynep Akata

cs.AIOct 1, 2026

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

Authors: Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata

Organizations: Technical University of Munich, MCML · Helmholtz Munich · Inria, École normale supérieure, CNRS, PSL Research University

Abstract

Agent evaluations increasingly go beyond a single success rate, reporting metrics such as cost, consistency, and robustness. Yet they typically treat the agent itself as fixed. In practice, an agent is a configurable system: users decide what to tell it, how long to let it run, and which model to use, and each of these choices can change how well and how consistently it performs. We study these choices on a new benchmark of four scientific tasks, where a coding agent must find and correctly operate a published specialist model. We investigate five parts of the agent's configuration: task information, reasoning, self-verification, time budget, and backbone model. We find substantial run-to-run variability, with approximately 54% of the outcome variance coming from repeating the same configuration rather than changing it. Across configurations, the information provided to the agent has the largest effect, exceeding both time budget and model size, while also reducing cost and improving calibration. Configuration choices also interact: additional time helps only when the agent has sufficient information or a capable enough model to use it. Finally, a trajectory-based taxonomy of agent behavior reveals that prompting an agent to verify its answer has little effect on its verification behavior, whereas providing a dedicated verification tool changes that behavior substantially. These results suggest that agents should be evaluated as configurable systems themselves, and that some desired behaviors are more effectively implemented in the system than requested through prompting. We release the benchmark and more than 18,000 agent trajectories.

Figures & tables

Figure 1: Architecture of our agentic AI4S benchmark. The agent (center) has access to different specialist models and is evaluated against their published performance (right). By varying the axes of the agent’s configuration within the same task (left), we can systematically study its performance, reliability, and behavior under controlled, repeatable conditions (bottom).

	redshift- estimation	mmlu- astronomy	promoter- prediction	rna-folding
Domain	Astrophysics	Astrophysics	Genomics	Genomics
Size	20 galaxy images	152 questions	613 DNA sequences	300 RNA sequences
Metric	$R^{2}$	Accuracy	MCC	$F_{1}$
Specialist	AstroCLIP	AstroSage-8B	DNABERT-2	RiNALMo
Challenge	Pre-processing	Declining	Fine-tuning	Post-processing
Regime	gap_positive	gap_negative	gap_positive	gap_positive

Table 1: The four tasks of the benchmark, together with one example instance per task. Each task belongs to one of two domains and one of two regimes, and focuses on a different aspect of the specialist models’ operation.

Runs & Completion
	redshift-	mmlu-	promoter-	rna-
	estimation	astronomy	prediction	folding
# Runs	2,160	2,160	2,160	2,160
Completion Rate	76.3% $\pm$ 1.8%	95.3% $\pm$ 0.9%	89.3% $\pm$ 1.3%	61.7% $\pm$ 2.0%
% Cells that (always / mixed / never) complete	54.6 / 40.3 / 5.1	81.9 / 18.1 / 0.0	63.7 / 36.1 / 0.2	21.3 / 72.0 / 6.7
Anchors

Table 2: Results per task for the main experiments (432 cells

\times

5 runs). Completion rate is computed over all runs; all other outcome rows are computed over completed runs.

C_{\text{out}}

is computed over cells with at least 3 completed runs. Full breakdown in Table 8 . † Raw scale, not comparable across tasks.

Figure 2: Outcome variance across cells. Left : mean

\mathcal{G}

vs. SD per cell (

\geq

3 completed runs), with Spearman correlation. Color: all/some/no runs clear the hurdle (green/orange/red). Right : variance between vs. within cells, for all completed runs and hurdle-clearing runs only.

Axis	Rank	Effect	Rank	Effect	Rank	Effect	Mean Rank
	promoter-prediction		redshift-estimation		rna-folding
Information	1	1.925	1	1.452	1	1.496	1.00
Model	2	0.704	4	0.170	2	1.253	2.67
Reasoning	3	0.274	3	0.172	3	0.302	3.00
Budget	4	0.142	2	0.201	4	0.255	3.33
Verification	5	0.131	5	0.130	5	0.077	5.00

Table 3: Effect of each axis on

\mathcal{G}

per task (range of mean

\mathcal{G}

across the axis’s levels, divided by the within-cell standard deviation). Axes are ranked within each task (1 = largest effect).

Panel of Figure 3 — Figure 3: Left : $\mathcal{G}$ and number of completed runs per Model and Budget ; error bars show 95% CI of the mean. Right : distribution of a run’s cost per Information level, colored by that level’s mean $\mathcal{G}$ . Additional plots are in Fig. 6 .

Figure 4: Distribution of verification behavior categories per level of the Verification axis, without (Baseline) and with the oracle tool, pooled across Qwen-122B, Step-3.7-Flash, and all tasks.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Grid	Cells per task	Runs per task	Trajectories
Main experiments (3 Qwen3.5 models)	432	2,160	8,640
Step-3.7-Flash (no act-only )	96	480	1,920
Claude Sonnet 5	144	720	2,880
Oracle, Qwen-122B	144	720	2,880
Oracle, Step-3.7-Flash	96	480	1,920
Total			18,240

Appendix

Table 4: Composition of the released trajectories. Each cell is repeated 5 times and the grid covers all four tasks.

	run_bash		read_file		write_file		web_fetch		finish		Other/malformed		Total
Task	Calls	Err.%	Calls	Err.%	Calls	Err.%	Calls	Err.%	Calls	Err.%	Calls	Err.%	Calls	Err.%
redshift-estimation	28.51	11.4	4.24	5.7	3.56	3.9	0.79	0.0	0.76	0.1	0.07	100.0	37.93	9.7
mmlu-astronomy	17.84	12.7	2.92	1.3	2.36	1.4	0.07	3.8	0.95	0.0	0.05	100.0	24.20	9.9
promoter-prediction	16.31	16.2	3.09	0.7	3.73	3.7	0.04	4.3	0.89	0.1	0.04	100.0	24.10	11.8
rna-folding	33.21	14.2	3.73	4.0	6.14	3.4	1.62	0.7	0.62	0.3	0.18	100.0	45.49	11.6

Appendix

Table 5: Mean tool calls per run and error rate per tool, main experiments.

Task	Fetches/run	Ref. correct %	Ref. decoy %	Off-list %	HTTP OK %	Top off-list hosts
redshift-estimation	0.79	88.9	0.1	11.0	84.4	colab.research.google.com , lite.duckduckgo.com
mmlu-astronomy	0.07	0.0	73.0	27.0	87.4	lite.duckduckgo.com , en.wikipedia.org
promoter-prediction	0.04	84.8	0.0	15.2	85.9	huggingface.co , raw.githubusercontent.com
rna-folding	1.62	95.9	0.0	4.1	86.7	api.github.com , lite.duckduckgo.com

Appendix

Table 6: web_fetch destinations, main experiments. HTTP OK % is the share of fetches that return a 2xx status. A fetch that returns 404 still counts as successful in Table 5 , because the harness executed it without an error.

Task	Specialist	mentioned	read_docs	executed	researched_online
redshift-estimation	astroclip	100.0%	99.9%	99.3%	28.3%
	astrosage	27.3%	27.1%	0.2%	0.1%
	dnabert-2	0.0%	0.0%	0.0%	0.0%
	rinalmo	0.0%	0.0%	0.0%	0.0%
mmlu-astronomy	astroclip	30.5%	30.2%	0.3%	0.0%
	astrosage	100.0%	98.6%	66.1%	5.0%

Appendix

Table 7: Engagement with each specialist model (% of runs), main experiments. The correct specialist model per task is in bold, mmlu-astronomy has none. redshift-estimation and mmlu-astronomy were run before the genomics models were available to the agent, however, the genomics tasks, whose sandbox contained all four models, show that the additional models do not keep the agent from finding the right one.

Task	Hurdle failure rate (95% CI)	$C_{\text{out}}$	ICC, all runs	ICC, clears hurdle	Mean per-cell SD	95% CI (5 runs)
redshift-estimation	53.8% [51.4, 56.2]	0.727	33.3%	36.2%	85.1%	91.9%
mmlu-astronomy	4.7% [3.9, 5.7]	0.905	29.4%	6.9%	17.6%	20.8%
promoter-prediction	0.5% [0.2, 0.9]	0.986	47.9%	49.7%	10.7%	12.9%
rna-folding	8.7% [7.3, 10.3]	0.825	45.7%	35.2%	12.8%	13.0%
Pooled (3 gap_positive tasks, task means removed)	20.6% [19.5, 21.7]	0.858	39.4%	46.1%	—	—

Appendix

Table 8: Hurdle decomposition and run-to-run variance per task. Mean per-cell SD and the 95% CI half-width of a cell’s mean score (at 5 runs) are expressed as % of

|\mathcal{R}-\mathcal{T}|

, so they are comparable across tasks. The pooled CI half-width in units of

\mathcal{G}

is shown in Fig. 5 (right).

Figure 5: Completion and variance details, pooled across the three gap_positive tasks. Left : cells by their number of completed runs (0–5); color shows whether all, some, or none of a cell’s completed runs clear the hurdle (green/orange/red), and gray marks cells without any completed run. Right : mean 95% CI half-width of a cell’s mean

\mathcal{G}

at 3–5 runs, for all completed runs (light blue) and for hurdle-clearing runs only (dark blue).

Panel of Figure 6 — Figure 6: Left : $\mathcal{G}$ and number of completed runs per Information and Budget level; error bars show 95% CI of the mean. Right : distribution of a run’s wall-clock time per Information level, colored by that level’s mean $\mathcal{G}$ .

Info/Budget	Short	Medium	Long
None	$-0.429$	$-0.441$	$-0.713$
None	89.1%	92.0%	92.2%
Identity	$-0.514$	$-0.519$	$-0.516$
Identity	83.8%	84.4%	84.4%
Interface	$-0.605$	$-0.560$	$-0.380$
Interface	76.1%	71.9%	65.9%

Appendix

Table 9: Mean

R^{2}

\mathcal{H}

failure rate, Budget

\times

Information, redshift-estimation .

Panel of Figure 7 — Figure 7: Calibration error per level of Reasoning , Model , and Information , pooled across the three gap_positive tasks. Positive values mean the agent overestimates its score. Each box is colored by that level’s mean $\mathcal{G}$ .

	redshift-	mmlu-	promoter-	rna-
	estimation	astronomy	prediction	folding
Mean Runtime (s)	344.2	188.3	278.0	476.0
Mean Cost (USD)	0.277	0.107	0.115	0.424
Mean Steps / Tool Calls / Errors	33 / 38 / 4	21 / 24 / 2	20 / 24 / 3	41 / 45 / 5

Appendix

Table 10: Mean cost and effort per run in the main experiments (432 cells

\times

5 runs).

Task	Tokens	Cost	Tokens	Cost	Tokens	Cost	Tokens	Calculated	Real	Savings
	Qwen-35B-A3B		Qwen-122B-A10B		Qwen-397B-A17B		Sonnet 5
redshift	659.5 / 10.3	$186	397.2 / 7.7	$183	340.1 / 7.0	$229	147.8 / 4.1	$337	$124	63.3%
mmlu	222.3 / 3.7	$63	142.0 / 3.2	$67	150.7 / 3.1	$102	65.4 / 2.8	$159	$66	58.6%
promoter	306.0 / 7.1	$91	143.0 / 4.9	$73	117.4 / 4.1	$85	88.5 / 3.9	$216	$84	60.9%
rna	978.5 / 15.3	$275	579.3 / 13.0	$273	530.3 / 13.5	$367	140.8 / 4.6	$328	$110	66.4%

Appendix

Table 11: Cost of one full 720-run task grid, per model. Tokens shown as prompt / completion, in millions.

Model	Task(s)	Baseline	Oracle	Baseline	Oracle	Baseline	Oracle	Baseline	Oracle	Baseline	Oracle
		Outcome		Cal. Err.		Cost (USD)		Wall-clock (s)		Steps / TC / Err.
Qwen-122B	mmlu-astronomy †	0.707	0.725	0.152	$-$ 0.002	0.093	0.125	165	220	19 / 22 / 2	21 / 24 / 2
Qwen-122B	Pooled gap_positive	0.822	0.923	0.093	0.002	0.245	0.358	346	453	28 / 32 / 4	34 / 39 / 5
Step-3.7-Flash	mmlu-astronomy †	0.677	0.721	0.285	0.065	0.043	0.091	164	309	18 / 20 / 2	26 / 28 / 3
Step-3.7-Flash	Pooled gap_positive	0.865	0.889	0.073	0.026	0.126	0.177	369	455	29 / 31 / 4	35 / 37 / 5

Appendix

Table 12: Oracle ablation: outcome, calibration error, cost/effort with vs. without the oracle_check tool.

Model	Axis	Baseline	Oracle	Baseline	Oracle	Baseline	Oracle	Baseline	Oracle
		promoter		redshift		rna		Mean Rank
Qwen-122B	Information	1	1	1	1	1	1	1.00	1.00
	Reasoning	2	2	3	2	2	2	2.33	2.00
	Verification	3	4	4	4	3	3	3.33	3.67
	Budget	4	3	2	3	4	4	3.33	3.33
Step-3.7-Flash	Information	1	2	1	1	1	1	1.00	1.33

Appendix

Table 13: Axis-effect ranks on

\mathcal{G}

per task, without (Baseline) and with the oracle tool, for Qwen-122B-A10B and Step-3.7-Flash (1 = largest effect). Unlike Table 14 , the Qwen-122B baseline includes the act-only level, so its ranks differ from that table.

Model	Axis	Rank	Effect	Rank	Effect	Rank	Effect	Mean Rank
		promoter		redshift		rna
Qwen-122B	Information	1	2.172	1	2.027	1	1.410	1.00
	Verification	2	0.256	3	0.195	3	0.193	2.67
	Budget	4	0.086	2	0.313	2	0.262	2.67
	Reasoning	3	0.116	4	0.028	4	0.041	3.67
Step-3.7-Flash	Information	1	0.848	1	0.951	1	1.412	1.00

Appendix

Table 14: Axis effects on

\mathcal{G}

for two open-weight models of similar size: Qwen-122B-A10B and Step-3.7-Flash (198B-A11B). Step-3.7-Flash does not support the act-only level of Reasoning , so both blocks exclude this level; Reasoning ’s rank and effect are computed over its remaining two levels.

Model	Task(s)	Outcome	Cal. Err.	Cost (USD)	Wall-clock (s)	Steps / TC / Err.
Qwen-122B	mmlu-astronomy †	0.702	0.153	0.089	164	17.8 / 21.6 / 2.1
Qwen-122B	Pooled gap_positive	0.802	0.105	0.228	347	25.7 / 31.9 / 3.7
Step-3.7-Flash	mmlu-astronomy †	0.677	0.285	0.043	164	17.8 / 19.9 / 2.1
Step-3.7-Flash	Pooled gap_positive	0.865	0.073	0.126	369	28.9 / 30.8 / 4.1

Appendix

Table 15: Qwen-122B-A10B vs. Step-3.7-Flash: outcome, calibration error, cost, and effort, excluding act-only as in Table 14 .

Model	Axis	Rank	Effect	Rank	Effect	Rank	Effect	Mean Rank
		promoter		redshift		rna
Qwen-397B	Information	1	1.884	1	1.126	1	1.051	1.00
	Reasoning	3	0.467	2	0.238	2	0.233	2.33
	Budget	2	0.497	4	0.106	4	0.070	3.33
	Verification	4	0.235	3	0.207	3	0.178	3.33
Claude Sonnet 5	Budget	1	1.589	1	1.878	1	1.796	1.00

Appendix

Table 16: Axis effects on

\mathcal{G}

for an open-weight and a closed model: Qwen-397B-A17B vs. Claude Sonnet 5.

Model	Task(s)	Outcome	Cal. Err.	Cost (USD)	Wall-clock (s)	Steps / TC / Err.
Qwen-397B	mmlu-astronomy †	0.715	0.131	0.141	159	19.9 / 25.0 / 2.8
Qwen-397B	Pooled gap_positive	0.870	0.092	0.315	342	24.7 / 31.4 / 2.6
Claude Sonnet 5	mmlu-astronomy †	0.679	$-$ 0.043	0.220	210	11.5 / 11.6 / 0.3
Claude Sonnet 5	Pooled gap_positive	0.828	$-$ 0.016	0.407	244	13.5 / 14.2 / 0.5

Appendix

Table 17: Qwen-397B-A17B vs. Claude Sonnet 5: outcome, calibration error, cost, and effort.

	promoter-prediction		redshift-estimation		rna-folding
Axis	P(1)	$\eta^{2}$	P(1)	$\eta^{2}$	P(1)	$\eta^{2}$
Information	1.00	0.413 [0.378, 0.445]	1.00	0.296 [0.281, 0.323]	1.00	0.260 [0.231, 0.289]
Model	0.00	0.080 [0.061, 0.100]	0.00	0.007 [0.002, 0.016]	0.00	0.145 [0.117, 0.177]
Reasoning	0.00	0.011 [0.004, 0.019]	0.00	0.008 [0.002, 0.017]	0.00	0.008 [0.003, 0.018]
Budget	0.00	0.009 [0.004, 0.017]	0.00	0.000 [0.000, 0.004]	0.00	0.004 [0.000, 0.012]
Verification	0.00	0.002 [0.000, 0.008]	0.00	0.003 [0.001, 0.012]	0.00	0.000 [0.000, 0.006]

Appendix

Table 18: Rank stability and partial

\eta^{2}

of each axis per task. P(1): share of bootstrap draws in which the axis has the largest effect size.

\eta^{2}

: partial

\eta^{2}

in a main-effects model, with 95% bootstrap interval.

	All completed runs		Matched configurations ( $k=1$ )
Level	Mean	Median	Mean	Median
Model = Qwen-35B	$-0.067$ [ $-0.093$ , $-0.039$ ]	$-0.137$ [ $-0.185$ , $-0.073$ ]	$-0.045$ [ $-0.074$ , $-0.017$ ]	$-0.107$ [ $-0.157$ , $-0.027$ ]
Model = Qwen-122B	$-0.030$ [ $-0.053$ , $-0.010$ ]	$-0.018$ [ $-0.033$ , $0.000$ ]	$-0.021$ [ $-0.042$ , $0.001$ ]	$-0.004$ [ $-0.029$ , $0.005$ ]
Model = Qwen-397B	$0.025$ [ $0.000$ , $0.047$ ]	$0.015$ [ $-0.002$ , $0.027$ ]	$0.036$ [ $0.014$ , $0.059$ ]	$0.016$ [ $0.004$ , $0.028$ ]
Information = none	$-0.002$ [ $-0.036$ , $0.031$ ]	$0.057$ [ $0.028$ , $0.060$ ]	$0.004$ [ $-0.027$ , $0.042$ ]	$0.057$ [ $0.027$ , $0.060$ ]
Information = identity	$0.004$ [ $-0.035$ , $0.040$ ]	$0.009$ [ $0.000$ , $0.078$ ]	$0.020$ [ $-0.017$ , $0.053$ ]	$0.012$ [ $0.004$ , $0.086$ ]

Appendix

Table 19: Change in

\mathcal{G}

from the short to the long budget (long minus short), pooled over the three gap_positive tasks, with 95% bootstrap intervals. Matched keeps only configurations with at least one completed run under every budget.

Model	LLM s / step	Steps / min	Tokens / s	CV	LLM share	Tool share
Qwen-35B	2.4	7.2	127	0.26	30%	70%
Qwen-122B	4.1	6.0	83	0.16	41%	59%
Qwen-397B	4.5	6.1	70	0.24	45%	55%
Step-3.7-Flash	4.3	6.0	94	0.18	46%	54%
Claude Sonnet 5	4.6	3.6	84	0.16	27%	73%

Appendix

Table 20: Generation speed and use of time per model, medians over all runs of the respective grid. Steps are LLM calls; LLM s / step is the generation time per step. CV is the coefficient of variation of the generation speed across runs.

Model	Task	0	1	2	3–4	$\geq$ 5	Mean calls	Flagged
Qwen-122B	redshift-estimation	19.7%	38.9%	8.3%	11.7%	21.4%	4.6	9.7%
	mmlu-astronomy	8.1%	68.3%	8.3%	7.5%	7.8%	1.7	0.1%
	promoter-prediction	3.8%	44.9%	11.0%	18.9%	21.5%	3.0	2.5%
	rna-folding	27.8%	44.6%	9.4%	9.2%	9.0%	1.7	2.8%
Step-3.7-Flash	redshift-estimation	44.6%	25.8%	4.8%	5.6%	19.2%	4.0	14.6%
	mmlu-astronomy	34.6%	25.2%	12.3%	14.6%	13.3%	2.0	1.5%

Appendix

Table 21: Number of oracle_check calls per run (share of runs) and share of runs flagged as likely hill-climbing.

Axis	Level	Completion	Mean accuracy
Information	none	92.8%	0.699
	identity	93.9%	0.712
	interface	98.7%	0.710
	protocol	95.7%	0.617
Model	Qwen-35B	89.6%	0.626
	Qwen-122B	97.5%	0.707

Appendix

Table 22: mmlu-astronomy : completion rate and mean accuracy per level of Information and Model . Backbone alone: 0.967; AstroSage reference: 0.671; chance: 0.25.

	Hurdle failure rate				ICC, hurdle-clearing
Task	$0\mu$	$0.5\mu$	$1\mu$	$2\mu$	$0\mu$	$0.5\mu$	$1\mu$	$2\mu$
redshift-estimation	48.7%	53.4%	53.8%	54.1%	35.8%	36.2%	36.2%	53.3%
mmlu-astronomy	4.2%	4.3%	4.7%	5.5%	6.9%	7.1%	6.9%	7.2%
promoter-prediction	0.2%	0.3%	0.5%	0.6%	48.7%	48.7%	49.7%	50.0%
rna-folding	0.2%	7.5%	8.7%	9.5%	45.8%	35.7%	35.2%	34.2%
Pooled	16.4%	20.0%	20.6%	20.9%	47.4%	44.8%	46.1%	46.4%

Appendix

Table 23: Hurdle failure rate and ICC among hurdle-clearing runs for different multiples of the default margin

\mu

. Pooled: three gap_positive tasks, task means removed.

	Error Category		Execution Quality		Model Usage		Planning & Exploration		Result		Verification Behavior
Axis	Rank	V	Rank	V	Rank	V	Rank	V	Rank	V	Rank	V
Information	1	0.233	1	0.264	1	0.234	1	0.271	1	0.191	1	0.112
Reasoning	5	0.041	4	0.044	4	0.050	4	0.038	4	0.053	4	0.030
Verification	4	0.048	5	0.031	5	0.020	5	0.012	5	0.031	5	0.023
Budget	3	0.120	3	0.070	2	0.108	3	0.039	2	0.152	2	0.108
Model	2	0.157	2	0.138	3	0.097	2	0.084	3	0.095	3	0.067

Appendix

Table 24: Association between each axis and the category distribution of each taxonomy dimension in the main experiments, measured by Cramér’s V (0 = no association, 1 = perfect association). Axes are ranked within each dimension.

Figure 8: Distribution of categories for each taxonomy dimension in the main experiments.

Explore similar work

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard
Sep 30, 2026Michael Hardy, Ruhana Azam, Anka Reuel +2Agentic Evaluations Scaffolds
Stanford University · University of Illinois Urbana-Champaign · Senior Author
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +9Agent Harness Agentic Benchmarks
1Peking University · 2Qiyuan Tech
AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
May 19, 2026Parsa Mazaheri, Kasra MazaheriLarge Language Model Agents Leaderboard
University of California, Santa Cruz · Massachusetts Institute of Technology

Sep 30, 2026, cs.AIJ/K move · Enter open · S save

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer+1

Stanford University · University of Illinois Urbana-Champaign · Senior Author

Agentic Evaluations Scaffolds Agentic

Open paper AskPDF arXiv

Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

Abstract

Explore similar work

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

AgentAtlas: Beyond Outcome Leaderboards for LLM Agents