Large language models (LLMs) are increasingly used as judges for automated AI evaluation. A common practice is to randomize prompt sequences and average the resulting scores, but its statistical validity remains unclear. We show that LLM evaluation mechanisms can be approximated by a class of Markov generalized linear mixed models (GLMMs), supported by out-of-sample predictions across three major commercial LLMs. Using a first-order Markov GLMM, we study leaderboard ranking and group comparison. For leaderboard ranking, randomize-and-average selection is consistent under a mild separation condition, and a Williams square design can improve efficiency when item qualities are close. For group comparison, naive averaging can yield inconsistent conclusions about differences in group-level quality because of the response model's nonlinearity. Empirical results further support the validity of the proposed model-based inference beyond the first-order theory, including settings with higher-order sequence memory. We illustrate the approach in an application where AI judges compare two graphical model estimation methods.
Figures & tables
Figure 1 : Dependence on the immediately preceding rating in the illustrative judge (Gemini 2.5 Flash), visible in the raw data before any modeling.
Figure 2 : Cross-model serial-contrast estimates of coefficients, with the bootstrap 95% intervals.
Figure 3 : Predictive performance check. Each panel has two blocks. Above the rule : the twelve transition terms. Below the rule : the three conditional entropies H(Yt∣Yt−1,…,Yt−k) , k=1,2,3 . For every term we compare the value observed in the 100 new batches against the 95% prediction intervals. The intervals are shifted to the common center for visualization purposes.
Naive average (%)
GLMM contrast (%)
Setting
Budget
random
Williams
random
Williams
Setting 1 (Flat)
N=6
44.6
44.1
44.4
44.5
N=12
47.8
47.7
47.8
48.2
N=18
49.8
49.3
49.8
49.5
Setting 2 (Uneven)
N=6
46.7
47.9
49.6
49.0
N=12
48.0
49.9
51.3
51.5
Table 1 : Top-1 accuracy (%) for all four combinations of estimator and design, V=6 , 10,000 replicates per cell (Monte Carlo standard error 0.71 on each gain). The item panel is θ=(θ∗+δ,θ∗,…,θ∗) with δ=2.8/N , the local scaling of Theorem 1 , so the selection problem holds constant difficulty as the budget grows. Setting 1 has a nearly flat position profile, so the removable design variance is nearly zero; Setting 2 has a strongly uneven profile and a substantial predicted gain, which is what the naive columns show.
Figure 4 : Group comparison results ( V=30 ), for the naive estimator and the GLMM fit. Top row: the rate at which the sign of each estimator picks group A as the winner. Bottom row: the false-positive rate and the power of the test.
Type-I error, null (%)
Sign rate (Group A), null (%)
Naive
GLMM
GLMM
Naive
GLMM
GLMM
N
average
lag-1
lag-3
average
lag-1
lag-3
30
26.0
24.5
25.0
86.5
41.0
40.5
60
38.0
10.5
11.5
95.5
43.5
44.5
90
51.0
6.0
4.5
97.5
46.0
46.5
120
66.5
6.5
6.0
99.0
49.5
51.5
Table 2 : The group-comparison results under a lag- 3 model for V=30 . The bootstrap ( B=200 ) is used for inference. The results are under the null case when θˉA=θˉB .
Figure 5 : Wald p -value with batch-bootstrap standard error against the number of batches for the two aggregation rules, on the 30 distinctive word pairs, fitted on the first N batches. Shading marks p<0.05 . Filled markers are contrasts in the direction of the published reference answer ( Δ>0 , favoring GNC-lasso); hollow markers contradict it. Values below 10−16 are displayed at the axis limit.
naive
GLMM
Judge
ΔNaive
p
outcome
ΔGLMM
p
outcome
OpenAI
+0.092
<10−12
correct
+2.30
<10−12
correct
Claude
−0.007
0.31
inconclusive
+1.16
2×10−12
correct
Gemini
−0.038
3×10−12
wrong
−0.18
0.24
inconclusive
Table 3 : Full sample ( N=160 ): contrasts against the published reference answer, which is Δ>0 ; p -values are from the Wald statistic with bootstrap standard error. “Correct” means significance with the sign matching the human evaluation in Li et al. [2020] ; “inconclusive” means not significant; “wrong” means significant with the opposite sign.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : The small-pool counterpart of the bottom row of Figure 4 : the same two scenarios and the same three procedures at V=6 , split 3 versus 3 .
Group A named (%), by budget N
V=30 ( 15 vs. 15 )
30
60
90
120
150
Naive
random shuffle
83.1
90.3
95.1
96.9
98.1
Williams square
82.7
91.3
94.6
97.3
98.5
GLMM
random shuffle
43.7
44.7
47.6
45.2
48.5
Williams square
42.7
47.3
48.0
48.0
47.6
V=6 ( 3 vs. 3 )
24
48
96
192
384
Appendix
Table S1 : The balanced design does not repair the naive estimator. Rate at which each arm names the higher-variance Group A under the equal-means null Setting 1, in percent; the groups are equal, so 50 is the calibrated value and no test is involved. Each estimator is run under a random shuffle and under a Williams square on independently seeded replicates ( 2000 per cell). The naive average climbs toward 100% under both designs, and the GLMM is near the coin flip under both. The Setting 2 control, with a true 0.4 -logit gap and matched spreads, separates all four arms from zero error by the largest budget (Group B named in 0.0% of replicates at V=30 , and 0.0% at V=6 , for every arm).
Random shuffle (%)
Williams design (%)
Naive
GLMM
GLMM
Naive
GLMM
GLMM
Setting
N
average
lag-1
lag-3
average
lag-1
lag-3
Setting 1 (Flat)
6
44.4
44.2
43.3
43.7
43.9
43.2
12
49.1
49.1
48.6
48.3
48.3
48.1
18
47.9
48.0
47.8
48.9
49.1
49.0
Setting 2 (Uneven)
6
46.9
49.4
47.1
48.7
49.8
48.8
Appendix
Table S2 : Selection accuracy (%) of Section 3 under a lag-3 GLMM model.
Figure S2 : The two estimated term graphs of Li et al. [2020] by glasso and GNC-lasso.
Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.
LLM-as-a-Judge evaluation has become a standard tool for assessing base model performance. However, characterizing performance via the naive estimator, i.e., raw judge outputs, is systematically biased. Recent work has proposed estimators to correct this bias, but their reliability depends critically on judge quality and, for model comparisons, on calibration stability. Sharing calibration across compared models is practically attractive but can introduce severe bias, including cases where the comparison estimate points in the wrong direction with high apparent confidence. We study these failure modes through analytical results, simulations over judge quality (J) and cross-model calibration instability (ΔJ), and a real-data MMLU-Pro case study with sign reversal. We propose J and ΔJ as diagnostics for when corrected estimates, especially shared-calibration comparisons, are likely unreliable, and provide reporting guidance for LaaJ evaluation.
Selecting the best large language model (LLM) for a fixed benchmark is often expensive, since exhaustive evaluation requires running every model on every example. Multi-armed bandit (MAB) algorithms can reduce the number of LLM calls by sequentially selecting the next model-example pair to evaluate, thereby avoiding wasted evaluations on clearly underperforming models. Further savings can be achieved by predicting model scores from the partially observed model-example score matrix using low-rank factorization. However, such predictions are not ground truth: they can be biased and may therefore lead to incorrect identification of the best model. In this work, we propose a principled framework that combines MAB with cheap predicted scores without compromising statistical validity. Specifically, we derive doubly robust estimators of each model's performance that use the low-rank predictions to reduce variance. This enables the construction of valid finite-sample confidence intervals in our setting, where models are selected adaptively and examples are sampled without replacement. Empirical results on real-world benchmarks show that our approach reduces the number of required evaluations, yielding meaningful savings in compute and cost while accurately identifying the best-performing model.
Elad Tolochinsky, Yaniv Tenzer, Yaniv Romano
Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology