Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Michael Hardy; Ruhana Azam; Anka Reuel; Mykel Kochenderfer; Sanmi Koyejo

cs.AISep 30, 2026

Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Authors: Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer, Sanmi Koyejo

Organizations: Stanford University · University of Illinois Urbana-Champaign · Senior Author

Abstract

Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models. We ask which conclusions current agent evaluations reliably support and what additional evaluation would improve them. We develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change rankings. We find: (1) Reliability depends on the measurement goal. Fixed model-scaffold systems are ranked reliably (0.935-0.994), while underlying-model reliability is substantially lower (0.148-0.841). (2) Scaffold choice can change conclusions. Inter-scaffold reliability measures whether scaffolds preserve model rankings, showing that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability of a benchmark by at most 0.097 when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from 0.44 to 0.75 at the same task budget and can reduce projected cost by up to 83%. Evaluation design should follow the intended claim: identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.

Figures & tables

Estimand	Object of inference	$\bm{\sigma_{o}^{2}}$	$\bm{\sigma_{\delta}^{2}(\mathcal{D})}$	Eq.
$E\rho^{2}_{M(b)}$ , $\text{SNR}_{M(b)}$	Does benchmark $b$ preserve model order?	$\sigma_{M}^{2}$	$\dfrac{\sigma_{IM}^{2}}{n_{i}}+\dfrac{\sigma_{MA}^{2}}{n_{a}}+\dfrac{\sigma_{IMA,e}^{2}}{n_{i}n_{a}}$	( 3 )
$E\rho^{2}_{MA(b)}$	Does benchmark $b$ preserve system order?	$\sigma_{M}^{2}+\sigma_{A}^{2}+\sigma_{MA}^{2}$	$\dfrac{\sigma_{IM}^{2}+\sigma_{IA}^{2}+\sigma_{IMA,e}^{2}}{n_{i}}$	( 4 )
$\rho^{(b)}_{AA^{\prime}}$	Do scaffolds induce the same model order?	$\sigma_{M}^{2}+\dfrac{\sigma_{IM}^{2}}{n_{i}}$	$\sigma_{MA}^{2}+\dfrac{\sigma_{IMA,e}^{2}}{n_{i}}$	( 5 )
$E\rho^{2}_{M}$ , $\text{SNR}_{M}$	Does the leaderboard preserve model order?	$\sigma_{M}^{2}$	$\dfrac{\sigma_{BM}^{2}}{n_{b}}+\dfrac{\sigma_{MA}^{2}}{n_{a}}+\dfrac{\sigma_{BMA}^{2}}{n_{b}n_{a}}+\dfrac{\sigma_{IM[B]}^{2}}{n_{b}n_{i}}+\dfrac{\sigma_{BIMA,e}^{2}}{n_{b}n_{i}n_{a}}$	( 7 )

Table 1: Reliability estimands. Each row applies

E\rho_{o}^{2}=\sigma_{o}^{2}/(\sigma_{o}^{2}+\sigma_{\delta}^{2})

\text{SNR}_{o}=\sigma_{o}^{2}/\sigma_{\delta}^{2}

, but changes the object whose ordering should generalize. The

n_{f}

terms denote equal allocation over facet

f

Benchmark	Models	Tasks	Agent Scaffolds	Model–Scaffold Pairs
AssistantBench	18	33	2	30
CORE-Bench Hard	34	45	3	56
GAIA	21	165	2	34
Online-Mind2Web	13	300	2	23
SciCode	17	65	3	37
ScienceAgentBench	19	102	2	25

Table 2: HAL dataset design summary. Counts denote unique levels of each facet.

Figure 1: Reliability depends on the object being ranked & task replication cannot remove scaffold-dependent model advantages. D-study reliability as tasks are added for base models and fixed model–scaffold systems. Dashed lines are task-only asymptotes for model rankings. Estimates are posterior medians with

68\%

standard error HDIs. See also Table 16 .

Figure 2: Scaffolds are rank-relevant and result in distinct task-level consequences. Left : posterior inter-scaffold reliability

\rho_{AA^{\prime}}^{(b)}

. Middle and Right : model-minus-scaffold system rank-relevant variance contrasts (over denominator from Eq. 4 ) at the benchmark and task levels, respectively; positive values favor the model contribution. Task-interaction contrasts measure task-specific sensitivity. Points and bars denote posterior means and medians, respectively, with

68\%

standard error HDIs.

Panel of Figure 3 — Figure 3: Broader measurement conditions reduce error that an increase in the number of test items alone cannot. Pooled D-studies compare concentration within one benchmark against allocation across multiple benchmarks. Estimates are posterior medians; panel (a) includes $68\%$ standard error HDIs. HAL and Harbor are fitted separately. The reference bands $[2,3]$ represent minimal detection limits used in laboratory measurement ( CDER, 2024 ; Sheehan and Yost, 2026 ; Taleuzzaman, 2018 ) , and are descriptive rather than benchmark calibrated thresholds.

External Benchmark	Mean score	$\widehat{\theta}$	Difference	LLM Observations
BFCL v4	$0.429\,[-0.282,\,0.836]$	$\mathbf{0.714}\,[0.147,\,0.928]$	$+0.285$	7
Terminal-Bench 2.0	$0.524\,[-0.165,\,0.869]$	$\mathbf{0.810}\,[0.361,\,0.954]$	$+0.286$	7
SWE-bench Verified	$0.565\,[0.306,\,0.746]$	$\mathbf{0.765}\,[0.595,\,0.870]$	$+0.200$	20
$\tau^{2}$ -bench Core	$0.333\,[-0.515,\,0.852]$	$\mathbf{0.867}\,[0.383,\,0.977]$	$+0.534$	6
Mean $\tau$	$0.463$	$\mathbf{0.789}$	$+0.326$	—

Table 3: Agreement with external agent benchmarks. Kendall’s

\tau

for the unweighted HAL mean and reliability-adjusted latent (

\widehat{\theta}

) model effect (median posterior) with

[95\%]

CIs.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Models	Tasks	Scaffold	Model-Scaffold Pair	Mean Score
algotune	9	5	4	18	0.078
arcagi2	9	5	4	18	0.067
bigcodebench	9	1	4	18	0.222
bixbench	9	5	4	18	0.122
codepde	9	1	4	18	0.000
cybergym	9	2	4	18	0.1 39

Appendix

Table 4: Descriptive statistics from the complementary Harbor Index dataset. Bolded datasets have at least three tasks and were used in the analysis of benchmark and item effects on agent system signal.

Benchmark	$n$	$\tau_{b}(\bar{x},Z)$	$\tau_{b}(\hat{\theta},Z)$	$\Delta_{b}$	Bootstrap interval	LOO range
BFCL	7	0.429	0.714	$-0.286$	$[-1.111,\phantom{-}0.556]$	$[-0.533,\phantom{-}0.000]$
SWE-bench	20	0.533	0.755	$-0.222$	$[-0.503,\phantom{-}0.045]$	$[-0.258,-0.129]$
$\tau^{2}$ -core	6	0.333	0.867	$-0.533$	$[-1.231,\phantom{-}0.000]$	$[-0.800,-0.400]$
Terminal-Bench 2.0	7	0.524	0.810	$-0.286$	$[-0.824,\phantom{-}0.000]$	$[-0.400,-0.133]$

Appendix

Table 5: Within-benchmark agreement with the aggregated external reference. Negative differences favor

\hat{\theta}

. Bootstrap intervals are exploratory, marginal 95% percentile intervals and are not adjusted for multiple comparisons.

External Benchmark	Mean score	$\widehat{\theta}$	Difference	LLM Observations
Multilevel $\tau_{\text{partial}}$	$0.430\,[0.266,\,0.570]$	$\mathbf{0.632}\,[0.507,\,0.732]$	$+0.202$	52

Appendix

Table 6: Hierarchical concordance with external agent benchmarks. Kendall’s

\tau

for the unweighted HAL mean and reliability-adjusted latent (

\widehat{\theta}

) model effect (median posterior) with

[95\%]

CIs.

Benchmark	Agent Names
AssistantBench	hal_generalist, browser-use
CORE-Bench Hard	hal_generalist, coreagent, claude_code
GAIA	hal_generalist, hf_open_deep_research
OnlineMind2Web	browser-use, seeact
SciCode	hal_generalist, tool_calling_agent, scicode_zero
ScienceAgentBench	hal_generalist, sab_selfdebug

Appendix

Table 7: Agent harness/scaffold by benchmark.

Figure 4: Number of benchmarks tested (left) and agents tested (right) per model.

Figure 5: Matrix indicating all the variations of benchmarks, scaffold and models (with reasoning-effort) that were included in our dataset.

Effect	Levels	Estimate	SD	95% CrI	$\widehat{R}$	Bulk ESS	Tail ESS
Group-level standard deviations
Scaffold, $\sigma_{a}$	13	0.79	0.44	$[0.06,\ 1.74]$	1.00	3454	3360
Benchmark, $\sigma_{b}$	9	2.45	0.73	$[1.39,\ 4.21]$	1.00	3743	3783
Benchmark $\times$ scaffold, $\sigma_{ba}$	21	0.69	0.36	$[0.06,\ 1.43]$	1.00	3580	3615
Benchmark $\times$ model, $\sigma_{bm}$	182	0.52	0.17	$[0.12,\ 0.80]$	1.00	1771	2110
Benchmark $\times$ model $\times$ scaffold, $\sigma_{bma}$	287	0.55	0.14	$[0.24,\ 0.79]$	1.00	1559	2062

Appendix

Table 8: Full posterior summary. Bayesian fit estimates for the full leaderboard-level model (Eq. 6 ). Group-level entries are standard deviations of random intercepts on the latent log-odds scale. Intervals are equal-tailed 95% posterior credible intervals.

Estimand	Object of inference	$\bm{\sigma_{o}^{2}}$	$\bm{\sigma_{\delta}^{2}(\mathcal{D})}$	Eq.
$E\rho^{2}_{M(b)}$ , $\text{SNR}_{M(b)}$	Does benchmark $b$ preserve model order?	$\sigma_{M}^{2}$	$\dfrac{\sigma_{IM}^{2}}{n_{i}}+\dfrac{\sigma_{MA}^{2}}{n_{a}}+\dfrac{\sigma_{IMA,e}^{2}}{n_{i}n_{a}}$	( 3 )
$E\rho^{2}_{MA(b)}$	Does benchmark $b$ preserve system order?	$\sigma_{M}^{2}+\sigma_{A}^{2}+\sigma_{MA}^{2}$	$\dfrac{\sigma_{IM}^{2}+\sigma_{IA}^{2}+\sigma_{IMA,e}^{2}}{n_{i}}$	( 4 )
$\rho^{(b)}_{AA^{\prime}}$	Do scaffolds induce the same model order?	$\sigma_{M}^{2}+\dfrac{\sigma_{IM}^{2}}{n_{i}}$	$\sigma_{MA}^{2}+\dfrac{\sigma_{IMA,e}^{2}}{n_{i}}$	( 5 )
$E\rho^{2}_{M}$ , $\text{SNR}_{M}$	Does the leaderboard preserve model order?	$\sigma_{M}^{2}$	$\dfrac{\sigma_{BM}^{2}}{n_{b}}+\dfrac{\sigma_{MA}^{2}}{n_{a}}+\dfrac{\sigma_{BMA}^{2}}{n_{b}n_{a}}+\dfrac{\sigma_{IM[B]}^{2}}{n_{b}n_{i}}+\dfrac{\sigma_{BIMA,e}^{2}}{n_{b}n_{i}n_{a}}$	( 7 )
$\displaystyle\lim_{n_{i}\rightarrow\infty}E\rho_{M(\cdot)}^{2}$	Does having unlimited tasks preserve model order?	$\sigma_{M}^{2}$	$\dfrac{\sigma_{MA}^{2}}{n_{a}}$ or $\frac{\sigma_{BM}^{2}}{n_{b}}+\frac{\sigma_{MA}^{2}}{n_{a}}+\frac{\sigma_{BMA}^{2}}{n_{b}n_{a}}$	( 8 )
${\Delta\rho^{2}_{(M-A)(b)}}$	Do models have more benchmark- relevant signal than scaffolds?	$\sigma_{M}^{2}-\sigma_{A}^{2}$	$\sigma_{MA}^{2}+\frac{\sigma_{IM}^{2}+\sigma_{IA}^{2}+\sigma_{IMA,e}^{2}}{n_{i}}$	(11)

Appendix

Table 9: Reliability estimands. Each row applies

E\rho_{o}^{2}=\sigma_{o}^{2}/(\sigma_{o}^{2}+\sigma_{\delta}^{2})

\text{SNR}_{o}=\sigma_{o}^{2}/\sigma_{\delta}^{2}

, but changes the object whose ordering should generalize. The

n_{f}

terms denote equal allocation over facet

f

Estimator	Metric	AssistantBench	CORE-Hard	GAIA	Mind2Web	SciCode	ScienceAgentBench	SWE-mini	$\tau$ -bench	USACO
Full	dCor ${}^{2}_{n}$	.404	.899	.780	.872	.703	.651	.833	.828	.430
Per-benchmark	dCor ${}^{2}_{n}$	-.005	.107	.046	.039	.013	.007	.046	.051	.032
Full	Kendall $\tau$	.478	.859	.771	.744	.731	.582	.764	.787	.590
Per-benchmark	Kendall $\tau$	.081	.248	.196	.151	.128	.101	.167	.180	.142
Full	Spearman $\rho$	.619	.961	.900	.896	.867	.795	.916	.930	.731
Per-benchmark	Spearman $\rho$	.111	.340	.271	.211	.171	.137	.236	.255	.199

Appendix

Table 10: Association between observed rankings and latent-effect rankings. “Full” uses the ecosystem model (Eq. 6 ); “Per-benchmark” uses an independently fitted model for each benchmark (Eq. 2 )

Quantity	Estimate	Standard error
$\operatorname{elpd}_{\mathrm{loo}}$	$-11{,}217.1$	$94.2$
$p_{\mathrm{loo}}$	$3{,}032.0$	$32.0$
$\operatorname{LOOIC}$	$22{,}434.2$	$188.5$
Pareto diagnostic	Count	Percentage
$\widehat{k}\leq 0.7$	$29{,}902$	$>99.9\%$
$0.7<\widehat{k}\leq 1$	$21$	$<0.1\%$

Appendix

Table 11: PSIS-LOO diagnostics for the full Bayesian logistic mixed model.

Figure 6: Posterior distributions for benchmark-level LOO ablations for the paper’s quantities of interest

Facet	Mean	Median	Min.	Max.	SD	SE	CV
Residual	.237	.235	.226	.267	.008	.002	.035
Scaffold ( $A$ )	.020	.019	.008	.037	.007	.002	.338
Benchmark ( $B$ )	.258	.259	.229	.298	.015	.003	.060
Benchmark–scaffold ( $BA$ )	.026	.028	.000	.035	.009	.002	.345
Benchmark–model ( $BM$ )	.020	.020	.009	.031	.004	.001	.208
Benchmark–model–scaffold ( $BMA$ )	.014	.013	.010	.019	.003	.001	.180

Appendix

Table 12: Leave-one-benchmark–scaffold-out proportions of variance. SE is the standard error across refits, and CV is the coefficient of variation.

Facet	Mean	Median	Min.	Max.	SD	SE	CV
Residual	0.236	0.236	0.231	0.241	0.002	0	0.008
Scaffold ( $A$ )	0.020	0.020	0.016	0.029	0.002	0	0.098
Benchmark ( $B$ )	0.259	0.259	0.252	0.268	0.002	0	0.009
Benchmark–scaffold ( $BA$ )	0.026	0.026	0.022	0.030	0.001	0	0.056
Benchmark–model ( $BM$ )	0.020	0.020	0.015	0.022	0.001	0	0.065
Benchmark–model–scaffold ( $BMA$ )	0.014	0.014	0.008	0.016	0.001	0	0.092

Appendix

Table 13: Leave-one-LLM-out proportions of variance. SD is the standard deviation across refits, SE is the standard error across refits, and CV is the coefficient of variation.

Corr.	AssistantBench	CORE-Bench	GAIA	Mind2web	SciCode	Sci.AgentBench	SWEbench	$\tau$ -bench	USACO
Spearman	0.442	0.824	0.79	0.621	0.662	0.611	0.808	0.683	0.692
dCor ${}^{2}_{n}$	0.141	0.64	0.576	0.33	0.36	0.323	0.633	0.408	0.395

Appendix

Table 14: Median correlation of reported and estimated ranks across posterior draws

Separation	Required comparison	Support in the observed design
$M$ versus $BM$	The same model appears on multiple benchmarks.	Nine models appear on all nine benchmarks; additional models provide partial cross-benchmark replication.
$A$ versus $BA$	The same scaffold appears on multiple benchmarks.	One scaffold spans eight benchmarks; an additional shared scaffold connects AssistantBench.
$MA$ versus $BMA$	The same model–scaffold pair recurs across benchmarks.	Cross-benchmark models evaluated through shared scaffolds create repeated model–scaffold pairs.
$M$ versus $MA$	Models are observed under multiple scaffolds, with overlap across models.	The cross-benchmark models span at least nine of the 13 scaffolds, and shared scaffolds provide common comparison conditions.
$BM$ versus $BMA$	Within a benchmark, models are observed under more than one scaffold.	Each benchmark contains shared or locally replicated scaffold conditions in addition to benchmark-specific scaffolds.
$I[B]$ versus $IM[B]$	Each item is attempted by multiple models.	Benchmark items are repeatedly scored across leaderboard models.

Appendix

Table 15: Observed comparisons supporting the pooled variance decomposition. “Separation” identifies the principal pair of components distinguished by each overlap pattern.

Estimand	Assistant	CORE-Hard	GAIA	Mind2Web	SciCode	ScienceAgent	SWE-mini	$\tau$ -Airline	USACO
$\lim_{n_{i}\rightarrow\infty}E\rho_{M}^{2}$ ( 8 )	0.210	0.892	0.325	0.153	0.840	0.683	0.552	0.669	0.377
$E\rho_{M}^{2}$ ( 3 )	0.182	0.841	0.318	0.148	0.811	0.628	0.543	0.572	0.375
$E\rho_{MA}^{2}$ ( 4 )	0.935	0.972	0.990	0.974	0.975	0.987	0.993	0.944	0.994

Appendix

Table 16: Reliability depends on the object being ranked. Reliability estimates for each benchmark under model and model–scaffold systems as measurement systems, including the task limit for models as the object of measurement.

Figure 7: Benchmarks differ sharply in ranking signal. Posterior model-ranking signal-to-noise ratio as tasks are added. Shaded bands give external detection-limit reference values. Points and intervals are posterior medians and

68\%

standard error HDIs.

Benchmark	$E\rho^{2}_{M(i)}$	$E\rho^{2}_{A(i)}$	$\rho^{(b)}_{AA^{\prime}}$	$E\rho^{2}_{M(b)}$	$E\rho^{2}_{A(b)}$	$E\rho^{2}_{MA(b)}$	$E\rho^{2}_{M(b)}(\infty)$
scicode	0.602	0.534	0.746	0.736	0.516	0.971	0.765
gaia	0.263	0.705	0.333	0.328	0.579	0.990	0.334
taubench_airline	0.549	0.539	0.545	0.533	0.759	0.938	0.619
swebench_verified_mini	0.599	0.857	0.504	0.499	0.639	0.992	0.506
usaco	0.238	0.253	0.392	0.429	0.478	0.993	0.432
assistantbench	0.362	0.445	0.287	0.260	0.521	0.911	0.305

Appendix

Table 17: Posterior-mean reliability and generalizability coefficients per benchmark.

E\rho^{2}_{M(i)}

and

E\rho^{2}_{A(i)}

are task reliabilities (

\sigma^{2}_{o}/(\sigma^{2}_{o}+\sigma^{2}_{io})

) for model and scaffold rankings;

\rho^{(b)}_{AA^{\prime}}

is inter-scaffold reliability (Eq. 5 );

E\rho^{2}_{M(b)}

E\rho^{2}_{A(b)}

, and

E\rho^{2}_{MA(b)}

are G-coefficients by object of measurement (Eqs. 3 – 4 ); the last column is the task-saturated model ceiling

\lim_{n_{i}\to\infty}E\rho^{2}_{M(b)}

(Eq. 8 ).

Figure 8: Reliability by Object of Measurement across all tasks in a benchmark. Estimated using variance components of Equation 2 , and posteriors of Equations 3 and 4 . Intervals are pseudo-standard error at 0.68 HDI, bar estimates are medians and points are means of posterior draw distributions.

	Assistant	CORE-Hard	GAIA	Mind2Web	SciCode	ScienceAgent	SWE-mini	$\tau$ -Airline	USACO	Mean Score	$\widehat{\theta}$
assistantbench	1.0	0.300	0.485	-0.061	0.103	0.215	-0.404	0.180	0.215	0.205	0.261
corebench_hard	0.300	1.0	0.487	0.400	0.492	0.458	0.148	0.581	0.167	0.687	0.835
gaia	0.485	0.487	1.0	0.028	0.260	0.479	0.215	0.363	0.182	0.453	0.606
onlinemind2web	-0.061	0.400	0.028	1.0	0.171	0.056	0.424	0.315	-0.141	0.581	0.348
scicode	0.103	0.492	0.260	0.171	1.0	0.184	0.117	0.477	0.225	0.202	0.472
scienceagentbench	0.215	0.458	0.479	0.056	0.184	1.0	0.129	0.587	-0.085	0.392	0.499

Appendix

Table 18: Internal Score Transportability. Kendall’s tau correlation matrix of benchmarks and summary statistics against each benchmark’s mean accuracy, the unweighted HAL mean score, and reliability-adjusted latent (

\widehat{\theta}

) model effect (median posterior)

k	mean ρ	median ρ	ρ min	ρ max	mean τ	median τ	τ min	τ max	top@1 agreement	mean abs rank change	cost savings
5	0.790	0.801	0.584	0.917	0.634	0.640	0.458	0.766	0.640	7.221	0.94
10	0.891	0.896	0.804	0.952	0.742	0.745	0.646	0.829	0.756	5.110	0.88
15	0.932	0.936	0.870	0.966	0.799	0.802	0.720	0.858	0.834	4.052	0.82
20	0.954	0.957	0.915	0.978	0.837	0.839	0.778	0.887	0.912	3.323	0.76
25	0.967	0.969	0.940	0.982	0.863	0.865	0.811	0.901	0.960	2.822	0.69
30	0.976	0.977	0.957	0.988	0.886	0.889	0.842	0.923	0.990	2.385	0.63

Appendix

Table 19: Task subsampling shows rank information preservation. Bootstrapped correlations of subsampled tasks and leaderboard Ranks

Figure 9: Proportion of Variance Explained in the posterior distributions from Eq. 2 . “Item” from benchmarking literature represents “task” commonly found in agentic evaluation literature. “Agent” represents “scaffold”.

Figure 10: Posterior density of Model-Agent difference of rank-relevant variance across the leaderboard (left), across a sampled benchmark (middle), and across a sampled task (right). Positive numbers suggest LLM variance contribution is greater than that of agent scaffold.

Figure 11: Published score versus posterior model percentile ranks using model-wise mean published score (x-axis) and draw-wise posterior median rankings with 95% CIs. Vertical intervals are

95\%

posterior rank intervals; jitter separates horizontal ties.

Figure 12: Rank order changes by benchmark based on comparison with median posterior rank from full variance decomposition model of Eq. 6 .

Figure 13: Cumulative posterior dominance rankings across individuals. The point estimate for each individual

m

represents their global dominance score, defined as the expected probability that their latent parameter

\theta_{m}=u_{m}^{(M)}+u_{mb}^{(}MB)

is strictly greater than that of a randomly selected peer

m^{\prime}

from the population, given the observed data

y

\text{Global Dominance}(m)=\frac{1}{M-1}\sum_{m^{\prime}\neq m}\Pr(\theta_{m}>\theta_{m^{\prime}}\mid y)

where

M

is the total number of individuals, and the pairwise probabilities are empirically derived via Bayesian draws as:

\Pr(\theta_{m}>\theta_{m^{\prime}}\mid y)\approx\frac{1}{D}\sum_{d=1}^{D}\mathbb{I}\left(\theta_{m}^{(d)}>\theta_{m^{\prime}}^{(d)}\right)

with

\mathbb{I}(\cdot)

denoting the indicator function and

D

representing the total number of posterior draws. Individuals are arranged along the vertical axis in ascending order of their global dominance scores. The vertical dashed line at

\Pr=0.5

denotes the theoretical baseline of a perfectly average individual who exhibits structural parity relative to the rest of the cohort. Deviations toward

1.0

signify deterministic stochastic dominance over the population, whereas values approaching

0.0

indicate systematic underperformance relative to the group.

Figure 14: Pairwise Posterior Dominance, where color is Pr(Row > Column $\mid y$ ) across posterior draws. The point estimate for each individual $m$ represents $\Pr(\theta_{m}>\theta_{m^{\prime}}\mid y)$ , defined by the expected probability of latent parameter $\theta_{m}=u_{m}^{(M)}+u_{mb}^{(}MB)$ , for each benchmark $b$ .

Figure 15: Rank order reliability, G-coefficient, per benchmark, for both latent and observed estimations with LLM as object of measurement. Bars are medians, points are means, and intervals are pseudo-standard error 0.68 HDI across the posterior using Eq. 3

Figure 16: Full variance decomposition of Eq. 6 as percentages of total variance. Variance decompositions for individual benchmarks are in Appendix F . Black represents residual variance.

Figure 17: Standardized Absolute Inter-scaffold Reliability. Standardized inter-scaffold reliability looks at the reliability for a single fixed task, rather than the composite reliability across all tasks in a benchmark.

Figure 18: Absolute vs Relative Reliability on Observed Scale

Benchmark	Facet	LMM	Bayes LMM	GLMM	Bayes GLMM	DISCO
scicode	item	0.235	0.232	0.766	0.718	0.209
scicode	agent	0.001	0.041	0.004	0.054	0.001
scicode	model	0.010	0.011	0.053	0.057	0.012
scicode	item_agent	0.037	0.036	0.012	0.016	0.252
scicode	item_model	0.130	0.123	0.012	0.035	0.509
scicode	model_agent	0.002	0.004	0.003	0.015	0.017

Appendix

Table 20: Proportion of variation explained per benchmark. For comparability across methods, Bayesian estimates represent posterior means (rather than medians found in most of the study) to preserve proportional relationships.

Facet	LMM	Bayes LMM	GLMM	Bayes GLMM	DISCO
agent	0.016	0.039	0.020	0.040	0.014
benchmark	0.075	0.108	0.259	0.299	0.023
benchmark_agent	0.028	0.036	0.026	0.031	0.030
benchmark_item	0.252	0.234	0.328	0.297	0.117
benchmark_item_agent	0.070	0.065	0.052	0.055	0.140
benchmark_item_model	0.047	0.043	0.004	0.040	0.240

Appendix

Table 21: Proportion of variation explained full leaderboard. For comparability across methods, Bayesian estimates represent posterior means (rather than medians found in most of the study) to preserve proportional relationships.

Explore similar work

What Does an LLM-Agent Leaderboard Rank Actually Compare?
Sep 7, 2026Wei-Jung HuangLeaderboard Comparison
Independent Researcher United States
Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
Jun 18, 2026Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin +58Agentic Evaluations Agentic Benchmarks
Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations
Aug 11, 2026Vasundra SrinivasanLeaderboard Bias-Variance Decomposition
Independent Researcher, Stanford School of Engineering

Sep 7, 2026, cs.AIJ/K move · Enter open · S save

What Does an LLM-Agent Leaderboard Rank Actually Compare?

Wei-Jung Huang

Independent Researcher United States

Leaderboard Comparison

Open paper AskPDF arXiv

An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions. Our estimand-aware pairwise procedure states the comparison target and measurement source, checks common support, and evaluates the supported difference using a stated uncertainty rule and practical margin. Controlled checks evaluate the decision labels under known finite-sample conditions and show why uncertainty must be included when judging sensitivity to target reweighting. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are often unresolved; proxy labels and utility rules can also change which system is selected. DataAgentBench and Open Agent show what remains estimable from coarser public records. A leaderboard score summarizes a released evaluation, whereas a fine-grained superiority claim additionally depends on the estimand and uncertainty rule used to interpret the difference.

Abstract

Explore similar work

What Does an LLM-Agent Leaderboard Rank Actually Compare?

Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations