Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

Atul Anand

cs.AIAug 17, 2026

Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

Authors: Atul Anand

Abstract

Benchmarks scored by an LLM judge routinely adjudicate differences of a tenth of a point, but the resolution of those benchmarks has never been measured. Existing sample-complexity work covers accuracy benchmarks and leaves the judged case open. Treating the system as the object of measurement, we decompose 373,019 judgments into system, item, judge and interaction components using generalizability theory. The central result is structural: under a single judge, generalizability asymptotes to sigma2_s/(sigma2_s+sigma2_sj) regardless of item count, because the system-by-judge term carries no n_i. Items saturate; judges do not. The item cost of a target diverges as the target nears that ceiling. The ceiling is a property of pointwise rubric scoring, not of LLM judging. Run as a pairwise preference in both presentation orders, sigma2_sj falls two orders of magnitude below sigma2_s and the ceiling rises to 0.986 (bootstrap [0.934, 1.000] on 11 systems), so one judge suffices. Pairwise buys a different problem: a system presented first wins 8.6 percentage points more often than the same system presented second, a bias 1.23x the median improvement claimed in the 53 published win-rate comparisons we recovered. Protocol design dominates panel size. Measured floors are 0.41-1.24 points on a 0-5 scale at native item counts, against a median reported improvement of 0.28 points; on the one benchmark recurring often enough for an exactly matched comparison, all 17 recovered MT-Bench improvements fall below MT-Bench's own floor, and 70% of the win-rate claims fall below the pairwise floor. An audit of 628 arXiv papers, double-coded by two independent models and validated against blind human coding (kappa=0.73), finds fewer than one paper in four states whether its evaluation was run more than once, and only 46-67% report uncertainty of any kind.

Figures & tables

Benchmark	$n_{s}$	$n_{i}$	MDD	95% CI	$E\rho^{2}$	ceiling	95% CI	$P(\geq\!0.8)$	judges
Screened (ours)	59	30	1.24	[1.17, 1.31]	0.695	0.746	[0.65, 0.81]	6.2%	2
MT-Bench	42	42	0.63	[0.60, 0.66]	0.463	0.600	[0.46, 0.72]	0.0%	25
Arena-Hard	42	17	0.71	[0.65, 0.76]	0.737	0.792	[0.69, 0.86]	41.0%	2
AlpacaEval 2	42	32	0.58	[0.54, 0.61]	0.542	0.706	[0.55, 0.80]	2.0%	7

Table 1: Detectability and the single-judge ceiling. MDD and

E\rho^{2}

at a common

n_{i}=30

, one judge, one run, so benchmarks are compared at an equal item budget. Intervals are parametric bootstrap (600–800 resamples);

P(\geq\!0.8)

is the bootstrap probability that the ceiling reaches the conventional threshold. The final column is the number of judges needed to reach

E\rho^{2}=0.8

at 30 items and 3 replicates.

Figure 1: Generalizability against item count under a single judge. Dashed lines are the asymptotes of Eq. ( 4 ); the solid horizontal line is the conventional

E\rho^{2}=0.8

threshold. All four flatten below it under the nine-judge universe ; under the six current-generation judges three of the four asymptote above it (§ 5.5 ).

Protocol	design	$\sigma^{2}_{s}$	$E\rho^{2}$ @30	ceiling	MDD ${}_{0\text{-}5}$	judges	items
as collected
Generic 0–5 rubric	42 $\times$ 42 $\times$ 9	4.5%	0.463	0.600	0.63	25	never
MT-Bench native	38 $\times$ 38 $\times$ 8	9.1%	0.700	0.804	0.56	3	860
matched: common 38 systems, 33 items, 8 judges
Generic 0–5 rubric	38 $\times$ 33 $\times$ 8	4.8%	0.483	0.623	0.61	18	never
MT-Bench native	38 $\times$ 33 $\times$ 8	8.4%	0.685	0.798	0.56	3	never

Table 2: Protocol design on identical MT-Bench prompts: our generic holistic 0–5 rubric versus MT-Bench’s own FastChat template, 1–10 scale, and GPT-4 reference answers for math, reasoning and coding. The two arms drew from different item pools (43 and 50 candidates) and did not survive collection equally—under the native protocol’s longer prompts the oai-cheap judge returned empty completions often enough to be dropped, along with 12 of its 50 items—so the upper block reports each arm as collected and the lower block refits both on the 38 systems, 33 items and 8 judges common to the two, which is the contrast the text relies on. MDD is on a common 0–5 basis. Judges is the count needed for

E\rho^{2}=0.8

at 30 items and 3 replicates; items the count at one judge and a single run, which diverges as the target approaches the ceiling.

Arm	design	$\sigma^{2}_{s}$	$\sigma^{2}_{sj}$	share	ceiling	$E\rho^{2}$ @24	[email protected]
System first (AB)	11 $\times$ 20 $\times$ 4	0.0874	0.00034	33.7%	0.996	0.937	1
Baseline first (BA)	12 $\times$ 19 $\times$ 4	0.0705	0.00075	27.6%	0.989	0.914	1
Balanced	11 $\times$ 19 $\times$ 4	0.0801	0.00110	35.7%	0.986	0.934	1
AB, frontier-only	9 $\times$ 20 $\times$ 4	0.0432	0.00207	19.8%	0.954	0.858	1
BA, frontier-only	9 $\times$ 19 $\times$ 4	0.0442	0.00119	17.5%	0.974	0.864	1
position bias: slot-A advantage $8.6$ points, sd across systems $7.3$ , max $25.0$

Table 3: The pairwise arm: 12 systems against a fixed gpt-4o-mini baseline on 24 shared items, 4 judges, 2 replicates, run in both presentation orders. The response is a win indicator, so

\sigma^{2}_{s}

and MDD are in win-rate units and are not comparable to the 0–5 rows elsewhere; ceiling and

E\rho^{2}

are unitless and are. “Balanced” is the mean of the two orders, which is the design a careful pairwise evaluation actually runs. “Frontier-only” drops the 2023 models, matching the restriction of Appendix A . Under a binary preference

\sigma^{2}_{sj}

falls two orders of magnitude below

\sigma^{2}_{s}

and the ceiling all but vanishes—at the cost of a position facet that a pointwise design does not have.

Benchmark	ceiling	95% CI	judges	ceiling	95% CI	judges
	primary: current-generation 6			sensitivity: all 9
Screened (ours)	0.929	[0.89, 0.95]	1	0.746	[0.66, 0.81]	2
MT-Bench	0.798	[0.68, 0.87]	5	0.600	[0.44, 0.71]	25
Arena-Hard	0.918	[0.86, 0.95]	1	0.792	[0.69, 0.86]	2
AlpacaEval 2	0.829	[0.71, 0.90]	3	0.706	[0.55, 0.80]	7

Table 4: Primary judge universe: the six current-generation judges, with the nine-judge panel beside it as sensitivity. The six drop the deliberately weak judge, the legacy model and the route replica, and are the universe a practitioner assembling a panel today would actually draw from. Quantities are as in Table 1 : MDD and

E\rho^{2}

at a common

n_{i}=30

, one judge, one run; ceiling is

\sigma^{2}_{s}/(\sigma^{2}_{s}+\sigma^{2}_{sj})

with a parametric bootstrap interval; judges is the count needed for

E\rho^{2}=0.8

at 30 items and 3 replicates. Restricting to current-generation judges raises every ceiling and lowers every judge requirement, and three of the four item sets then clear

0.8

—but the ordering of item sets, the finite ceiling, and the divergence of the item cost near that ceiling are unchanged.

Disclosure	Rate
Names the judge model	91.2%
States the number of evaluation items	70.0%
Reports any uncertainty (CI / SD / test)	45.8%
Pins a judge version	38.8%
Controls presentation order	15.9%
States whether the evaluation was repeated	19.8%

Table 5: Disclosure among 92 papers that report a judged system comparison.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Iso-cost allocation. At a fixed budget of

n_{i}n_{j}n_{r}

calls, MDD against how the budget is split; the marked point is the optimum. It is interior at every budget, and it moves toward more judges as the budget grows rather than saturating.

Figure 3: Probability that two adjacent systems in the true top ten are returned in the wrong order, against item count, at one and three judges. The curves flatten well short of the 0.5 chance line: items buy rank stability slowly, and a third judge buys more than doubling the items.

Figure 4: Sizing nomogram. Contours of

E\rho^{2}

over judges and items; read off the panel a target requires. Each contour flattens to a horizontal asymptote, and below that judge count no item budget reaches the target, the ceiling, drawn.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 529 papers

May 25, 2026

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Validating an LLM judge against human annotations usually means reporting several agreement statistics: accuracy, precision, recall,

F_1

, Cohen's

κ

, and one or more rank correlations. A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling, and those choices rarely stated. For binary criteria -- the common case in rubric-based evaluation, where each criterion is graded MET or UNMET -- most of the reported numbers are redundant: Pearson's

r

, Spearman's

ρ

, Kendall's

τ_b

, the phi coefficient

φ

, and the Matthews Correlation Coefficient all reduce to a single number on non-degenerate binary data, so reporting several of them only creates an illusion of corroborating evidence. Cohen's

κ

is the one agreement coefficient that adds information: it shares

φ

's numerator but normalizes differently, and the gap between them measures how far the judge's positive-label rate has drifted from the human's. We then trace what changes when a judge may abstain with a CANNOT_ASSESS verdict: the three common ways of handling abstentions are not interchangeable preprocessing choices but answer different questions, and they break the binary equivalences. The same equivalences reappear, up to a negligible finite-sample correction, for multi-judge ensembles scored with Fleiss'

κ

or Krippendorff's

α

. We close with a reporting checklist that names the judgment scale, the abstention and tie handling mode, coverage, the confusion matrix, and the aggregation level alongside any scalar agreement coefficient.

Delip Rao, Chris Callison-Burch

University of Pennsylvania

Inter-Rater Reliability LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation

Ask PDF

Mar 12, 2026

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.

Eddie Landesberg

LLM Evaluation LLM-as-a-Judge Best-of-N Sampling LLM Reranking

Ask PDF

Mar 2, 2026

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation

LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be averaged out by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. We introduce a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges to scenarios, is demonstrated to be the optimal strategy for a fixed judge panel and judge-call budget: the score recovers the panel mean exactly while matching the cost of single-judge evaluation. Empirical results on MT-Bench and MindEval validate the effectiveness of CyclicJudge as predicted, across both general-purpose and domain-specific evaluation settings.

Ziyi Zhu, Olivier Tieleman, Alexey Bukhtiyarov +1

Slingshot AI · Department of Engineering, University of Cambridge

LLM Evaluation Bias in LLM-as-a-Judge LLM-as-a-Judge

Ask PDF

Abstract

LLM-as-a-Judge

Explore similar work

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation