Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking

Solomon Messing

cs.CLApr 13, 2026

Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking

Authors: Solomon Messing

Organizations: ML Commons

Abstract

LLM evaluations drive which models get deployed, what safety standards get adopted, which research conclusions get published, and how projections of AI's labor-market impact get made. Yet standard confidence intervals ignore variability from judge model choice, model temperature, and prompt phrasing, producing under-coverage that worsens with more data. The omitted variance can shift results enough to reverse conclusions \citep{baumann2025llmhacking, huang2026dropping}; pipelines that fail to average over it leave the surface that ``benchmark hacking'' exploits \citep{singh2025leaderboard}. This paper decomposes LLM pipeline uncertainty into its sources, distinguishes variance that shrinks with more data from sensitivity to researcher design choices, and uses design-study projections to reduce total evaluation error (TEE). Across the demonstrations, naive standard errors are 40 - 60% smaller than the TEE-corrected SE. Using Chatbot Arena data, we show naive 95% CI coverage drops as $n$ grows while TEE-corrected coverage holds at 95%, and TEE-guided pipelines restrict the benchmark gaming surface from 56 to 32 Elo ( $K=27$ ), below the human-leaderboard baseline. We show further that a small pilot recovers honest CIs and projects which design changes most improve precision. Acting on those projections halves MMLU estimation error against the answer key at equivalent cost, and raises per-match agreement with human votes by 7.9 percentage points on Chatbot Arena.

Figures & tables

Figure 1: Naive confidence intervals fail because pipeline variance from prompt, judge, and temperature choice do not shrink with items (

N

). (a) 95% CI coverage versus

N

in 5 scenarios (A-E). (b) Variance components accounted for in each scenario. Scenario A corresponds to the naive estimate

\Phi^{-1}(0.975)\times s/\sqrt{N}

; scenarios B-E represent adding more variance parameters estimated according to the framework introduced below.

Panel of Figure 2 — Figure 2: TEE separates aggregation-reducible noise from design-choice sensitivity at every pipeline stage. For content-annotation tasks (e.g., labeling social media posts) the SUT layer is omitted.

Panel of Figure 3 — Figure 3: Safety decomposition (a) and cost-efficiency frontier (b) for AILuminate. Judge-model variance dominates the decomposition in (a). A TEE-guided design that aggregates over diverse judges captures most of the available SE reduction at modest cost (b).

Figure 4: TEE-guided estimation on MMLU halves the naive RMSE at the same API budget and maintains nominal 95% CI coverage, while naive under-covers at 91 - 93% because its CI ignores prompt sensitivity. (a) RMSE against ground-truth accuracy by API call budget. (b) 95% CI coverage versus API budget. 1,000 Monte Carlo replicates.

Figure 5: Naive coverage on Chatbot Arena drops as the sample grows, while TEE maintains nominal coverage. Left: coverage versus match sample size

n_{m}

for naive and TEE. Right: half-width of CI versus

n_{m}

. TEE bottoms out at the appropriate floor (

\sigma_{V}^{2}/V+\sigma_{J}^{2}/J

) while naive keeps shrinking. Ground truth

p^{*}=0.389

is the mean across all 3 judges and 5 prompt variants in the Likert factorial.

Figure 6: Best-of-

K{=}27

gaming surface on Chatbot Arena across pipeline configurations. “Gaming surface” is the expected Elo boost a developer earns by submitting

K{=}27

model variants and reporting only the highest-scoring one. Variance surface computed by bootstrapping 4,676 Arena matches.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

			SUT layer			Judge layer
	Any CI/SE	Item sampling	SUT prompt	SUT temperature	SUT replications	Judge prompt	Judge model	Judge temperature	Interactions	Contamination
MMLU [ Hendrycks et al., 2021 ]	-	-	-	-	-	n/a	n/a	n/a	-	High
Open LLM Leaderboard	-	-	-	-	-	n/a	n/a	n/a	-	Mixed
AlpacaEval [ Dubois et al., 2024 ]	$\star$	$\star$	-	-	-	-	-	-	-	High
HELM [ Liang et al., 2023 ]	$\star$	$\star$	-	-	-	n/a	n/a	n/a	-	High
Chatbot Arena [ Chiang et al., 2024 ]	$\star$	$\star$	-	-	-	n/a	n/a	n/a	-	Medium

Appendix

Table 1: Uncertainty reporting in prominent LLM evaluations. SUT-layer columns capture variance in the system under test. Judge-layer columns capture variance in the scoring pipeline.

Factor	Type	Levels	Examples
Item $i$	Random	$N$ (large)	A specific social media post, a safety test prompt
Category $c$	Random grouping	$C$	Hate speech type, hazard category
Prompt variant $v$	Random	$V$ (3 - 5)	“Is this post hateful?” vs. “Does this contain hate speech?”
Temperature $h$	Fixed	$H$ (e.g., 3)	$T=0.0,0.7,1.0$
Model $m$	Fixed	Small set	GPT-4o, Claude Haiku 4.5, Gemini 2.0 Flash
Scoring method	Fixed (Tier 2)	Typically 1	Exact match, Likert, LLM-as-judge

Appendix

Table 2: Factors in the TEE data-generating process (DGP), classified as fixed or random. Random factors (item, prompt variant, replication) represent exchangeable draws from larger populations. Their variance components shrink with aggregation within a chosen design (Tier 1). Fixed factors (temperature, model, scoring method) represent specific researcher choices. When no aggregation occurs, their sensitivity indices quantify researcher degrees of freedom (Tier 2). When the design does cross multiple levels of a fixed factor (e.g., averaging over a panel of judges), the corresponding sensitivity contribution also shrinks as

1/L

. This across-design aggregation is what the empirical sections rely on for judge-model averaging.

Variance component	Empirical share	How it inflates scores	Mitigation
Prompt sensitivity $\sigma^{2}_{\phi}$	1.2%	Passive: benchmark’s single prompt carries unquantified phrasing noise. Active: developer tunes output to the known prompt style	Average $\geq$ 3 diverse prompt variants
Judge model $\sigma^{2}_{\lambda}$ (fixed)	8.8%	Passive: a single judge’s systematic bias shifts all scores. Active: developer optimizes for the specific judge’s preferences	Multi-judge average or report all judges
Item $\times$ prompt $\sigma^{2}_{\alpha\phi}$	2.5%	Passive: item-specific phrasing noise inflates CIs. Active: developer targets responses the chosen prompt phrasing scores leniently	Multi-prompt averaging
Item $\times$ judge $\sigma^{2}_{\alpha\lambda}$	18.0%	Passive: a single judge’s item-level idiosyncrasies inflate CIs. Active: developer fine-tunes on items the chosen judge scores leniently	Multi-judge averaging eliminates this interaction
Prompt $\times$ judge $\sigma^{2}_{\phi\lambda}$	0.7%	Specific (judge, prompt) cell biases compound when both are fixed	Multi-prompt $\times$ multi-judge averaging

Appendix

Table 3: Exploitable variance components in single-configuration benchmarks (Chatbot Arena Likert decomposition, pooled across the four categories). Empirical shares from the full-factorial decomposition (4,676 matches, 5 prompt variants, 3 judges). Reported as shares of per-observation variance

\sigma^{2}_{\text{total}}

because gaming exploits single-configuration noise directly, before averaging shrinks any component. “How it inflates scores” describes two channels: (1) passive inflation, where the benchmark operator’s single-configuration design leaves noise that widens CIs for all submissions, and (2) active exploitation, where a developer who can probe the benchmark’s specific configuration optimizes against it.

Feature	TEE	Camuffo	Haase	Song	Wang	NIST
		(2026)	(2026)	(2025)	(2025)	(2026)
G-theory grounded	$\star$	$\star$	-	$\star$	-	-
Interaction terms	$\star$	-	-	$\star$	-	-
D-study projections	$\star$	-	-	$\star$	-	-
Two-tier separation	$\star$	-	-	-	-	-
Scoring method comparison	$\star$	-	-	-	-	-

Appendix

Table 4: Feature comparison of TEE with concurrent frameworks for LLM evaluation uncertainty. Stars indicate features present. Dashes indicate absent or not addressed. Cell entries verified against each paper’s text and tables.

G-Theory	TEE
Object of measurement	Item $i$
Facet (rater)	Model/judge $m$
Facet (occasion)	Prompt variant $v$
Fixed facet	Temperature $h$
Universe score $\mu_{p}$	$\theta_{hm}=\mu+\tau_{h}+\lambda_{m}$
G-study	Factorial data collection

Appendix

Table 5: Mapping between classical generalizability theory (G-theory) and TEE terminology. TEE applies G-theory’s variance decomposition and decision-study framework to LLM evaluation pipelines, treating items as the object of measurement and prompt variants, judges, and replications as measurement facets.

Source	df	EMS
Category ( $\kappa$ )	$C-1$	$\sigma^{2}_{\rho}+R\sigma^{2}_{\epsilon}+R\sigma^{2}_{\alpha\phi}+VR(\sigma^{2}_{\delta}+\sigma^{2}_{\alpha\tau})+nVR\sigma^{2}_{\kappa}$
Item w/in cat. ( $\delta$ )	$C(n-1)$	$\sigma^{2}_{\rho}+R\sigma^{2}_{\epsilon}+R\sigma^{2}_{\alpha\phi}+VR(\sigma^{2}_{\delta}+\sigma^{2}_{\alpha\tau})$
Prompt ( $\phi$ )	$V-1$	$\sigma^{2}_{\rho}+R\sigma^{2}_{\epsilon}+R\sigma^{2}_{\alpha\phi}+NR(\sigma^{2}_{\phi}+\sigma^{2}_{\phi\tau})$
Item $\times$ Prompt ( $\alpha\phi$ )	$(N-1)(V-1)$	$\sigma^{2}_{\rho}+R\sigma^{2}_{\epsilon}+R\sigma^{2}_{\alpha\phi}$
Residual ( $\rho$ )	$NV(R-1)$	$\sigma^{2}_{\rho}$

Appendix

Table 6: Expected mean squares for the balanced mixed design with

C

categories,

n=N/C

items per category,

V

prompt variants, and

R

replications, conditioning on model

m

and temperature

h

. Temperature interaction terms (

\sigma^{2}_{\alpha\tau}

\sigma^{2}_{\phi\tau}

) are confounded with item and prompt main effects at a single temperature level. A multi-temperature design adds rows for Temperature (

H-1

df), Item

\times

Temperature, and Prompt

\times

Temperature.

Figure 7: REML estimator convergence. (a) Bias falls below 5% for all components once total observations exceed 1,000. (b) RMSE by component across sample sizes, showing monotonic decline. Interaction components (

\sigma^{2}_{\alpha\phi}

\sigma^{2}_{\alpha\lambda}

) require larger samples for precise estimation than main effects. 1,000 simulations per configuration.

Figure 8: Small-

V

prompt sensitivity estimation. (a) Relative bias of

\hat{\sigma}^{2}_{\phi}

versus number of prompt variants

V

. At

V\!=\!2

, bias is 4 - 5%; at

V\geq 3

, bias drops below 4%, supporting the minimum design recommendation of

V\geq 3

. (b) Directional accuracy of the D-study recommendation (add prompts vs. add replications) near the knife-edge boundary where both interventions yield equal gains. Accuracy exceeds 90% when one source clearly dominates but drops to near chance at the boundary. 1,000 replicates per condition. Axis label uses

K

for prompt variants (simulation convention).

K=V

in main text notation.

Scenario	Key parameter	Rel. bias (%)
1. Correct specification	-	$-0.9$
2. Correlated RE	$\alpha=2$	$-2.0$
3. Non-exchangeable prompts	ratio $=8$	$-1.6$
4. Non-normal scores	df $=5$	$-7.2$
5. Heterogeneous category $\times$ variant	0.25/4 $\times$	$-2.3$

Appendix

Table 7: D-study projection bias under five scenarios: one correctly specified baseline and four misspecification conditions (correlated random effects, non-exchangeable prompts, heavy-tailed scores, heterogeneous category interactions). All scenarios produce

|\text{bias}|\leq 8\%

, confirming that D-study projections are stable under the assumption violations tested.

N_{\text{sim}}=1,000

Figure 9: D-study projection validation. (a) Projected % change in variance for each intervention (1,000 datasets). Red diamonds: true change. (b) Projection accuracy under assumption violations, with bias

\leq 8\%

. Details in Section E.2 .

Figure 10: BT pairwise scoring is stable under Likert pathologies. Left: at low prevalence, binary classification collapses while BT remains stable. Upper right: central tendency compression (

\kappa

) degrades Likert ranking recovery, while BT is unaffected. Lower right: coarser Likert scales (fewer points) reduce recovery, while BT is invariant to discretization. The full simulation sweeps five pathologies (

\kappa

\sigma_{\text{anchor}}

\sigma_{\text{slope}}

, nonlinear

\gamma

, discretization). This figure shows the three with the largest effects.

N_{\text{sim}}=200

Component	Value	Notes
$\sigma^{2}_{\phi}$ (prompt)	0.015	rescaled
$\tau$ (temperature levels)	$\{-0.15,0,0.15\}$	$\mathrm{pop\_var}(\tau)=0.015$
$\lambda$ (judge levels)	$\{-0.15,0,0.15\}$	$\mathrm{pop\_var}(\lambda)=0.015$
$\sigma^{2}_{\alpha}$ (item)	0.04
$\sigma^{2}_{\kappa}$ (category)	0.015
$\sigma^{2}_{\epsilon}$ (residual)	0.03

Appendix

Table 8: DGP parameters for the variance underestimation simulation. Design-side components (

\sigma^{2}_{\phi}

\sigma^{2}_{\tau}

\sigma^{2}_{\lambda}

) are rescaled to similar magnitudes so each scenario’s coverage drop is comparably visible. Item- and residual-side components are scaled proportionally so total variance is comparable to the ideology Likert fit.

Scenario	Sampled	Adds	Captures	Misses
A (naive)	$V{=}1,M{=}1,H{=}1,R{=}1$	$s/\sqrt{N}$	-	all design-side
B	$V{=}3,M{=}1,H{=}1,R{=}5$	(1\|variant)	$\sigma^{2}_{\phi}$	$\tau,\lambda$ , interactions
C	$V{=}3,M{=}3,H{=}1,R{=}5$	+ (1\|judge)	+ $\sigma^{2}_{\lambda}$	$\tau$ , interactions
D	$V{=}3,M{=}3,H{=}3,R{=}5$	+ (1\|temp)	+ $\sigma^{2}_{\tau}$	interactions
E	same data as D	+ (1\|variant:judge), (1\|variant:temp)	+ design $\times$ design	-

Appendix

Table 9: Scenarios A - E progressively enrich the fitted random-effects model. Each row indicates the factorial dimensions the scenario samples, the variance component the new random effect captures, and what remains uncaptured.

Figure 11: Naive 95% CIs lose coverage as sample size grows because the SE shrinks while the bias from omitted pipeline factors stays constant. Panel (a) plots each scenario’s SE relative to the full TEE SE, with the naive Scenario A more than

10\times

smaller at large

N

. Panel (b) plots 95% CI coverage versus

N

for the same scenarios, showing how each added random effect (prompt in B, judge in C, temperature in D, then pairwise interactions in E) closes one more component of the omitted variance and lifts coverage one step toward nominal.

N\in\{20,50,100,200,500,1,000,2,000\}

;

N_{\text{sim}}=1,000

$\gamma$	Label	Coverage (%)	Max $\|\text{rel.\ bias}\|$ (%)	D-study $\rho$
0	Baseline	90.5	8.8	1.00
0.5	Mild	88.7	6.7	1.00
1	Moderate	90.3	3.4	1.00
2	Strong	91.2	6.0	1.00

Appendix

Table 10: TEE stability under latent item ambiguity.

\gamma

controls the strength of ambiguity-driven dependence between item difficulty and interaction variances, with sparse 3-way interactions for ambiguous items (

z_{i}>1

). CI coverage does not degrade, component bias stays below 9%, and D-study intervention rankings are preserved exactly across all ambiguity levels.

N_{\text{sim}}=1,000

Component	LPM	GLMM	LPM	GLMM	LPM	GLMM
	Pairwise (%)		Safety (%)		MMLU (%)
Judge / SUT model ( $\sigma^{2}_{\lambda}$ , sensitivity)	68.4	50.9	43.8	70.7	25.0	27.4
Within-category item ( $\sigma^{2}_{\delta}$ )	7.6	13.7	20.8	11.4	35.0	23.4
Prompt ( $\sigma^{2}_{\phi}$ )	0.4	0.4	0.0	0.0	14.7	35.7
Item $\times$ judge / SUT ( $\sigma^{2}_{\alpha\lambda}$ )	1.9	1.9	16.7	4.3	5.3	2.7
Prompt $\times$ judge / SUT ( $\sigma^{2}_{\phi\lambda}$ )	0.1	0.0	12.4	7.9	13.8	5.4

Appendix

Table 11: LPM vs. GLMM Var(

\hat{\theta}

) decomposition for the three binary benchmarks (pairwise ideology, safety classification, MMLU). LPM shares are on the probability scale; GLMM shares are on the logit scale, so absolute SE values are not directly comparable, but component rankings are. For the GLMM, the level-1 residual is fixed at

\pi^{2}/3

following the logistic-link convention [ Nakagawa and Schielzeth, 2010 ] . All three fits include the cell-level random effect. For MMLU the model layer is the system under test (SUT) rather than a judge. The SUT row plays the same role as “judge model” in the other two benchmarks. Spearman rank correlations between LPM and GLMM shares are

\rho=0.99

(pairwise),

0.91

(safety), and

0.95

(MMLU), confirming that the dominant components are not LPM artifacts.

Figure 12: Judge-model variance dominates both ideology pipelines, but the secondary rankings differ. After judges, Likert is dominated by item heterogeneity and pairwise by position bias, a factor with no Likert analogue. Operational design:

N{=}150

V{=}5

M{=}3

H{=}3

R{=}3

(pairwise also crosses

P{=}2

position orderings).

Component	Pilot (% Var $(\hat{\theta})$ )	Full (% Var $(\hat{\theta})$ )
Judge model ( $\sigma^{2}_{\lambda}$ , sensitivity)	42.4	54.4
Within-category item ( $\sigma^{2}_{\delta}$ )	48.6	27.0
Prompt $\times$ judge ( $\sigma^{2}_{\phi\lambda}$ )	5.4	9.3
Prompt ( $\sigma^{2}_{\phi}$ )	-	8.0
Item $\times$ judge ( $\sigma^{2}_{\alpha\lambda}$ )	2.2	0.7
Item $\times$ prompt ( $\sigma^{2}_{\alpha\phi}$ )	1.3	0.6

Appendix

Table 12: Variance component estimates for the Likert ideology demonstration (CoT protocol): a 30-item, 3-variant pilot subset (

N\!=\!30

V\!=\!3

M\!=\!3

H\!=\!3

R\!=\!3

; 2,430 calls) compared with the 150-item full run (

N\!=\!150

V\!=\!5

R\!=\!3

; 20,250 calls). The pilot correctly identifies the two dominant components (within-category item and judge-model design sensitivity) and recovers their rank ordering. Quantitative shares shift between pilot and full because the divisors differ across designs. The pilot pools

\sigma^{2}_{\rho}

and

\sigma^{2}_{\epsilon}

into a single “replicate noise” row because the small

N

does not separately identify the cell-level term. Shares of mean variance

\Var(\hat{\theta})

at the indicated pilot and full designs.

Figure 13: Judge averaging is the dominant lever in Likert ideology scoring. Bars show the projected percentage change in

\Var(\hat{\theta})

for each single-factor intervention. Committing to a single judge inflates variance by 43.8% because

\sigma^{2}_{\lambda}

(54.4% of

\Var(\hat{\theta})

M=3

) loses the

1/M

averaging benefit. Doubling items reduces variance by 31% by halving the within-category item term (

\sigma^{2}_{\delta}

, 26.9% of

\Var(\hat{\theta})

). Adding 2 prompt variants buys 11.3%. Replications contribute negligibly. Same factorial design as Figure 12 .

Figure 14: Doubling items and judge averaging are the dominant levers in pairwise ideology scoring. Bars show the projected percentage change in

\Var(\hat{\theta})

for each single-factor intervention. Doubling items gives the largest reduction (47.5%) by halving every

N

-divided term jointly. Committing to a single judge inflates variance by 40.6% because

\sigma^{2}_{\lambda}

(68.4% of

\Var(\hat{\theta})

M=3

) loses the

1/M

averaging benefit. Adding 2 prompt variants buys 2.7%. Replications contribute negligibly. The position factor (

\sigma^{2}_{\pi}

, 20.9%) is fixed at

P=2

in this design and cannot be reduced by averaging. Same factorial design as Figure 12 .

Figure 15: Per-category residual variance (

\hat{\sigma}^{2}_{\epsilon}

) across five ideology dimensions, estimated from the Likert scoring decomposition. Residual variance spans a fourfold range across categories.

N=150

items,

R=8

replications per cell.

Figure 16: Per-category residual variance (

\hat{\sigma}^{2}_{\epsilon}

) for binary safety scoring, by hazard category. Sexual content and violence produce the most variable judge classifications across repeated calls. Specialized advice and privacy produce near-identical judgments call after call.

Figure 17: Item-level pipeline variance predicts misclassification against human ground truth (

r=-0.68

). Each point is one of 50 prompts about political institutions from Waight et al. [2026] . The x-axis shows each item’s variance across 45 LLM pipeline configurations (5 prompts

\times

3 judges

\times

3 temperatures). The y-axis shows accuracy against 9-coder human majority vote.

Component	$\hat{\sigma}^{2}$	% Var $(\hat{\theta})$	Tier
Within-category item ( $\sigma^{2}_{\delta}$ )	0.063	35.0	1
SUT model ( $\sigma^{2}_{\lambda}$ , sensitivity)	0.001	25.0	2
Prompt ( $\sigma^{2}_{\phi}$ )	0.001	14.7	1
Prompt $\times$ SUT ( $\sigma^{2}_{\phi\lambda}$ )	0.002	13.8	1
Item $\times$ SUT ( $\sigma^{2}_{\alpha\lambda}$ )	0.028	5.3	1
Between-category ( $\sigma^{2}_{\kappa}$ )	0.009	5.1	1

Appendix

Table 13: Variance components for MMLU exact-match scoring (200 items, 5 prompt variants, 3 temperatures, 3 SUTs, 8 replications; 72,000 calls). Item heterogeneity dominates the analyst’s mean variance (35%); SUT-model design sensitivity (25%), prompt sensitivity (15%), and prompt

\times

SUT interaction (14%) jointly contribute most of the rest. Shares of mean variance

\Var(\hat{\theta})

at the operational design (

N{=}200

V{=}5

M{=}3

H{=}3

R{=}8

); each component

\hat{\sigma}^{2}_{k}

contributes

\hat{\sigma}^{2}_{k}/d_{k}

where

d_{k}

is its D-study divisor.

Figure 18: Within-category item heterogeneity dominates MMLU variance, with SUT-model and prompt sensitivity also substantial. Each lollipop shows one component’s contribution to

\Var(\hat{\theta})

at the operational design (

N{=}200

items,

V{=}5

prompts,

M{=}3

SUTs,

H{=}3

temperatures,

R{=}8

replications). Within-category item heterogeneity contributes 35.0%, SUT-model design sensitivity 25.0%, prompt main effect 14.7%, prompt

\times

SUT interaction 13.8%, and item

\times

SUT 5.3%. Replicate noise and cell-level idiosyncrasy each fall below 0.5%. The D-study directs budget toward broader item coverage and SUT diversity, not prompt engineering or replications.

Figure 19: Per-category residual variance (

\hat{\sigma}^{2}_{\epsilon}

) for MMLU across subject categories. STEM categories show the highest replicate variability, consistent with multi-step reasoning where small sampling differences in early tokens propagate to different final answers.

Domain	SE underest.	Width ratio	Naive cov.	TEE cov.
MMLU (multiple-choice QA)	57%	43%	93%	93%
Safety classification (AILuminate)	43%	57%	80%	93%
Likert ideology	60%	40%	42%	89%
Pairwise ideology	40%	60%	27%	56%

Appendix

Table 14: Naive vs TEE standard errors across four empirical demonstrations. SE underest. and width ratio are mean ratios across the 45 per-domain configurations. Naive cov. and TEE cov. are the 95% CI coverage rates against the target grand mean. Nominal coverage is 95%.

Tier	Item	Report
Minimum	Prompt text	Exact text of all prompt variants, including system prompt
	Model identity	Model name, version, API provider, date
	Temperature	Temperature setting, top- $p$ /top- $k$ if applicable
	Replications	Number per cell ( $R$ ), scoring/extraction logic
Standard	Prompt variants	Number ( $V\geq 2$ ), how generated, whether original included
	Sensitivity flag	Whether prompt sensitivity was assessed, qualitative result

Appendix

Table 15: Three-tier reporting checklist for LLM-based measurement studies. Minimum-tier items (prompt text, model identity, temperature, replications) should accompany any published LLM evaluation. Standard adds prompt variant assessment. Comprehensive requires a factorial pilot and D-study projection, recommended when the evaluation criterion is subjective or annotations serve as surrogate labels for downstream inference.

Component	Original	Closed frontier	Open-weight
Within-category item ( $\sigma^{2}_{\delta}$ )	76.5	21.7	79.0
Item $\times$ judge ( $\sigma^{2}_{\alpha\lambda}$ )	12.2	25.5	15.9
Judge model ( $\sigma^{2}_{\lambda}$ , sensitivity)	1.3	40.6	3.7
Between-category ( $\sigma^{2}_{\kappa}$ )	2.8	11.5	$<$ 0.1
Prompt $\times$ judge ( $\sigma^{2}_{\phi\lambda}$ )	4.1	0.4	0.4
Item $\times$ prompt ( $\sigma^{2}_{\alpha\phi}$ )	2.8	0.2	0.3

Appendix

Table 16: Variance decomposition by judge capability tier (safety domain, same 24 AILuminate items, 3,240 calls per tier). Shares of mean variance

\Var(\hat{\theta})

at each tier’s operational design (

N{=}24

V{=}3

M{=}3

H{=}3

R{=}5

). In the closed-frontier tier, judge-model design sensitivity (40.6%) and item

\times

judge (25.5%) together dominate. In the original and open-weight tiers, within-category item heterogeneity dominates because

N{=}24

is small. Frontier models disagree more about which items are safe, so judge disagreement cannot be explained as a capability artifact.

Figure 20: TEE estimation raises agreement with Arena humans by

+7.9

pp on Likert and

+3.6

pp on pairwise across 4,676 matches. Each dot is one pipeline’s agreement (AB-only, ties excluded).

Category	Pipeline	$N$	Overall	$N_{\text{AB}}$	AB-only
Creative Writing	Single-judge Likert	967	0.403	343	0.668
	Single-judge pairwise	1,062	0.445	577	0.692
	TEE Likert	1,064	0.424	278	0.763
	TEE pairwise	1,064	0.461	475	0.743
Persuasion	Single-judge Likert	1,280	0.362	327	0.618
	Single-judge pairwise	1,426	0.393	749	0.621

Appendix

Table 17: Agreement with Chatbot Arena human preferences by scoring pipeline and task category. Overall counts tied predictions as correct when the human label is also tie. AB-only restricts to battles where both sides are decisive.

Figure 21: Per-category D-study projection (dev split, x-axis) vs. observed Likert agreement lift (test split, y-axis). Spearman

\rho=0.80

across the four category points.

Figure 22: Naive (single judge

\times

variant, binomial Wald) vs. TEE (factorial, crossed random-effects Wald) 95% CI coverage and half-width for the scoring-family mean

p^{*}=0.389

, across

K=300

Monte Carlo draws per

n_{b}

. Top: naive coverage degrades as

n_{b}

grows because binomial SE shrinks past the between-cell variance floor. TEE coverage holds at or above nominal. Bottom: TEE half-width bottoms out at the

\sigma^{2}_{V}/V+\sigma^{2}_{J}/J

floor while naive keeps shrinking. Ribbon: Monte Carlo

\pm 1.96

SE.

Figure 23: Bootstrap SE for per-model BT

\log\pi_{m}

under each of the four pipelines (

N=300

reps,

55

models). Top: ratio of TEE-aware to naive bootstrap SE per model, log-scaled x-axis. Diamond is the per-pipeline median. Bottom: scatter of naive vs. TEE-aware SE. Dashed line is equality. Single-cell pipelines cluster above the diagonal (naive under-estimates). TEE pipelines cluster on the diagonal (cell variance is absorbed by aggregation).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Jun 18, 2026

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

Can we trust evaluation scores to capture an LLM's true real-world performance? Certifiable evaluation answers this question by providing guarantee for LLM evaluation. In particular, existing methods sequentially curate evaluation samples and keep updating confidence intervals (CIs) that cover the true performance with high probability (e.g., 95%) until some conditions are satisfied, e.g., the CI width reaches a target precision. However, existing methods are not generally anytime-valid: the claimed coverage (e.g., 95%) may fail when CIs are repeatedly updated and used to decide when to stop, leaving a gap between theoretical rigor and practice. This paper bridges this gap by proposing Celeus, a Certifiable framework for Efficient LLM evaluation, which leverages E-processes to build anytime-valid CIs. Concretely, we propose signals that combine two ingredients: (i) Uncertainty-guided sampling to select informative samples for evaluation, and (ii) Surrogate-assisted approximations for unevaluated samples. We prove that such signals remain unbiased for the evaluation score conditional on the past, enabling statistically-grounded and anytime-valid

e

-process CIs. More importantly, the two ingredients reduce estimation variance and help reach the target precision with fewer evaluated samples. We also prove that CIs obtained by Celeus can shrink at a near-parametric rate up to logarithmic factors and analyze the oracle variance-optimal sampling rule that motivates the empirical uncertainty-guided one. Experiments show that Celeus reaches the target precision using 54-62% fewer evaluated samples than baselines, while preserving anytime-valid coverage.

Zhijian Zhou, Zesheng Ye, Zhaorun Chen +2

The University of Melbourne · Department of Computer Science, University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign

LLM Evaluation Anytime-Valid Inference Adaptive Sampling Uncertainty Sampling

Ask PDF

Sep 20, 2026

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated confidence intervals (CIs), hypothesis tests, and judge-bias corrections, such claims are unreliable. We address these issues in several contributions. First, we find that running statistics over raw LLM judge scores leads to inflated false positives: counterintuitively, for many inter-rater agreement metrics, false positive risk peaks at "almost perfect" human-LLM agreement. To help researchers understand how to run statistics over LLM judges responsibly, we present guidance and tooling for the statistical analysis of mixed human-AI judge designs, and implement nine hypothesis tests via prediction-powered inference (PPI), including the first known PPI corrections for four rank-based tests (Wilcoxon signed-rank, Mann-Whitney U, and omnibus variants). To keep PPI++ stable with small human-labeled calibration sets, we introduce bootstrap-adaptive power tuning, which shrinks the estimated weight toward a target estimated from the labeled data, and accounts for that weight's own sampling variance. Second, through Monte Carlo simulations, we derive recommendations for what CI, p-value, and FWER correction methods to use for small-sample AI evaluations (N<100), and warn researchers against bootstrap CIs. We package these recommendations into evalstats, an open-source Python package that selects calibrated methods automatically, and demonstrate it in three scenarios, including one where a real LLM judge validated at "substantial agreement" would have led a researcher to publish a spurious finding. evalstats is publicly available at https://github.com/ianarawjo/evalstats.

Ian Arawjo

Department of Computer Science and Operations Research (DIRO) Université de Montréal Montréal, Quebec, Canada

Statistical Hypothesis Testing LLM Evaluation LLM-as-a-Judge Prediction-Powered Inference Bias in LLM-as-a-Judge

Ask PDF

Sep 8, 2026

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

Large language model (LLM) benchmarks are often treated as fixed datasets with stable scores, yet their outcomes depend on configurable evaluation pipelines. We audit eight cybersecurity benchmarks across 10 proprietary, open-weight, and cybersecurity-specialized LLMs. By modeling benchmarks as measurement pipelines, we identify 15 systematic failure modes and show that a single pipeline choice can change a model's score by more than 80 percentage points and substantially alter model rankings. At the cross-benchmark level, two semantically similar task pairs rank the same models differently because of incompatible evaluation conventions. Under an evaluation harness that standardizes pipeline choices while preserving task semantics, nine of 10 models shift by at least three ranks on at least one benchmark. These results show that cybersecurity LLM benchmark scores are pipeline-dependent and motivate pipeline-aware auditing as a core requirement for reliable model evaluation.

Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah +1

Qatar Computing Research Institute, HBKU

LLM Evaluation LLM Auditing LLM Reliability LLMs for Cybersecurity

Ask PDF

Abstract

LLM Evaluation

Explore similar work

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks