Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects

Tianxi Li; Jie Ding

stat.MLOct 6, 2026

Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects

Authors: Tianxi Li, Jie Ding

Organizations: School of Statistics, University of Minnesota

Abstract

Large language models (LLMs) are increasingly used as judges for automated AI evaluation. A common practice is to randomize prompt sequences and average the resulting scores, but its statistical validity remains unclear. We show that LLM evaluation mechanisms can be approximated by a class of Markov generalized linear mixed models (GLMMs), supported by out-of-sample predictions across three major commercial LLMs. Using a first-order Markov GLMM, we study leaderboard ranking and group comparison. For leaderboard ranking, randomize-and-average selection is consistent under a mild separation condition, and a Williams square design can improve efficiency when item qualities are close. For group comparison, naive averaging can yield inconsistent conclusions about differences in group-level quality because of the response model's nonlinearity. Empirical results further support the validity of the proposed model-based inference beyond the first-order theory, including settings with higher-order sequence memory. We illustrate the approach in an application where AI judges compare two graphical model estimation methods.

Figures & tables

Panel of Figure 1 — Figure 1 : Dependence on the immediately preceding rating in the illustrative judge (Gemini 2.5 Flash), visible in the raw data before any modeling.

Figure 2 : Cross-model serial-contrast estimates of coefficients, with the bootstrap

95\%

intervals.

Panel of Figure 3 — Figure 3 : Predictive performance check. Each panel has two blocks. Above the rule : the twelve transition terms. Below the rule : the three conditional entropies $H(Y_{t}\mid Y_{t-1},\dots,Y_{t-k})$ , $k=1,2,3$ . For every term we compare the value observed in the $100$ new batches against the $95\%$ prediction intervals. The intervals are shifted to the common center for visualization purposes.

		Naive average (%)		GLMM contrast (%)
Setting	Budget	random	Williams	random	Williams
Setting 1 (Flat)	$N{=}6$	$44.6$	$44.1$	$44.4$	$44.5$
	$N{=}12$	$47.8$	$47.7$	$47.8$	$48.2$
	$N{=}18$	$49.8$	$49.3$	$49.8$	$49.5$
Setting 2 (Uneven)	$N{=}6$	$46.7$	$47.9$	$49.6$	$49.0$
	$N{=}12$	$48.0$	$49.9$	$51.3$	$51.5$

Table 1 : Top-1 accuracy (%) for all four combinations of estimator and design,

V=6

10{,}000

replicates per cell (Monte Carlo standard error

0.71

on each gain). The item panel is

\bm{\theta}=(\theta^{\ast}+\delta,\theta^{\ast},\dots,\theta^{\ast})

with

\delta=2.8/\sqrt{N}

, the local scaling of Theorem 1 , so the selection problem holds constant difficulty as the budget grows. Setting 1 has a nearly flat position profile, so the removable design variance is nearly zero; Setting 2 has a strongly uneven profile and a substantial predicted gain, which is what the naive columns show.

Panel of Figure 4 — Figure 4 : Group comparison results ( $V=30$ ), for the naive estimator and the GLMM fit. Top row: the rate at which the sign of each estimator picks group $A$ as the winner. Bottom row: the false-positive rate and the power of the test.

	Type-I error, null (%)			Sign rate (Group A), null (%)
	Naive	GLMM	GLMM	Naive	GLMM	GLMM
$N$	average	lag-1	lag-3	average	lag-1	lag-3
$30$	$26.0$	$24.5$	$25.0$	$86.5$	$41.0$	$40.5$
$60$	$38.0$	$10.5$	$11.5$	$95.5$	$43.5$	$44.5$
$90$	$51.0$	$6.0$	$4.5$	$97.5$	$46.0$	$46.5$
$120$	$66.5$	$6.5$	$6.0$	$99.0$	$49.5$	$51.5$

Table 2 : The group-comparison results under a lag-

3

model for

V=30

. The bootstrap (

B=200

) is used for inference. The results are under the null case when

\bar{\theta}_{A}=\bar{\theta}_{B}

Figure 5 : Wald

p

-value with batch-bootstrap standard error against the number of batches for the two aggregation rules, on the

30

distinctive word pairs, fitted on the first

N

batches. Shading marks

p<0.05

. Filled markers are contrasts in the direction of the published reference answer (

\widehat{\Delta}>0

, favoring GNC-lasso); hollow markers contradict it. Values below

10^{-16}

are displayed at the axis limit.

	naive			GLMM
Judge	$\widehat{\Delta}_{\mathrm{Naive}}$	$p$	outcome	$\widehat{\Delta}_{\mathrm{GLMM}}$	$p$	outcome
OpenAI	$+0.092$	${<}10^{-12}$	correct	$+2.30$	${<}10^{-12}$	correct
Claude	$-0.007$	$0.31$	inconclusive	$+1.16$	$2\times 10^{-12}$	correct
Gemini	$-0.038$	$3\times 10^{-12}$	wrong	$-0.18$	$0.24$	inconclusive

Table 3 : Full sample (

N=160

): contrasts against the published reference answer, which is

\widehat{\Delta}>0

;

p

-values are from the Wald statistic with bootstrap standard error. “Correct” means significance with the sign matching the human evaluation in Li et al. [2020] ; “inconclusive” means not significant; “wrong” means significant with the opposite sign.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure S1 : The small-pool counterpart of the bottom row of Figure 4 : the same two scenarios and the same three procedures at $V=6$ , split $3$ versus $3$ .

		Group A named (%), by budget $N$
$V=30$ ( $15$ vs. $15$ )		$30$	$60$	$90$	$120$	$150$
Naive	random shuffle	$83.1$	$90.3$	$95.1$	$96.9$	$98.1$
Naive	Williams square	$82.7$	$91.3$	$94.6$	$97.3$	$98.5$
GLMM	random shuffle	$43.7$	$44.7$	$47.6$	$45.2$	$48.5$
GLMM	Williams square	$42.7$	$47.3$	$48.0$	$48.0$	$47.6$
$V=6$ ( $3$ vs. $3$ )		$24$	$48$	$96$	$192$	$384$

Appendix

Table S1 : The balanced design does not repair the naive estimator. Rate at which each arm names the higher-variance Group A under the equal-means null Setting 1, in percent; the groups are equal, so

50

is the calibrated value and no test is involved. Each estimator is run under a random shuffle and under a Williams square on independently seeded replicates (

2000

per cell). The naive average climbs toward

100\%

under both designs, and the GLMM is near the coin flip under both. The Setting 2 control, with a true

0.4

-logit gap and matched spreads, separates all four arms from zero error by the largest budget (Group B named in

0.0\%

of replicates at

V=30

, and

0.0\%

V=6

, for every arm).

		Random shuffle (%)			Williams design (%)
		Naive	GLMM	GLMM	Naive	GLMM	GLMM
Setting	$N$	average	lag-1	lag-3	average	lag-1	lag-3
Setting 1 (Flat)	$6$	$44.4$	$44.2$	$43.3$	$43.7$	$43.9$	$43.2$
	$12$	$49.1$	$49.1$	$48.6$	$48.3$	$48.3$	$48.1$
	$18$	$47.9$	$48.0$	$47.8$	$48.9$	$49.1$	$49.0$
Setting 2 (Uneven)	$6$	$46.9$	$49.4$	$47.1$	$48.7$	$49.8$	$48.8$

Appendix

Table S2 : Selection accuracy (%) of Section 3 under a lag-3 GLMM model.

Panel of Figure 12 — Figure S2 : The two estimated term graphs of Li et al. [2020] by glasso and GNC-lasso.

Explore similar work

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo Hagendorff
AI Safety Research Group, University of Stuttgart
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLlm-As-A-Judge Raw Judge Outputs
Indeed Inc.
Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
May 11, 2026Elad Tolochinsky, Yaniv Tenzer, Yaniv RomanoLarge Language Model Evaluation Model Selection
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology

Oct 4, 2026, cs.CLJ/K move · Enter open · S save

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Laurène Vaugrante, Thilo Hagendorff

AI Safety Research Group, University of Stuttgart

Open paper AskPDF arXiv

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Abstract

Explore similar work

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Bias and Uncertainty in LLM-as-a-Judge Estimation

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization