Three Ways Classical Test Theory Can Mislead About LLM Judges

Louis Yiven Zhu

cs.LGAug 31, 2026

Three Ways Classical Test Theory Can Mislead About LLM Judges

Authors: Louis Yiven Zhu

Organizations: University of Oxford

Abstract

Evaluations that use a large language model (LLM) as a judge have begun to borrow reliability statistics from classical test theory and its extensions. We examine three such statistics that need one administration and no gold labels. None of them can isolate the judge, because one judge under one prompt supplies no variance component of its own. Claude Haiku 4.5 judged 210 constructed short answers against ten-element checklists. On the 180 with parsed verdicts, the Kuder-Richardson coefficient (KR-20) came out at 0.5223 on the judge's verdicts and 0.5231 on error-free gold verdicts. In simulation, bank design alone moves KR-20 from 0.01 to 0.68 at the judge's measured 4.72% error rate. The dependability index $Φ(λ)$ , a ratio of mean squared distances from the pass mark, sits 0.22 to 0.38 below the judge's accuracy against gold and returns 0.54 to 0.68 on error-free gold verdicts. Livingston-Lewis accuracy treats the rubric elements as a sample, and at a pass mark of five elements it credits error-free gold scores with 0.78, close to the judge's 0.81. A statement about the judge therefore needs gold labels or a varied scorer facet, and a reliability ratio needs the bank's spread beside it. One of the four closest judge-evaluation papers varies the prompt and still reads a reliability below 0.7 as a sign that a model cannot serve as a judge, although that reliability moves with the spread of the samples scored. We derive a decision table and four reporting lines from these two rules.

Figures & tables

Figure 1: Why KR-20 cannot identify the judge’s reliability. A fixed LLM judge scores

n

responses against

K

binary rubric elements and fills the verdict matrix

X_{ij}\in\{0,1\}

. KR-20 partitions the matrix’s variance into components for responses, elements and a residual. One judge under one prompt phrasing supplies no variance along a scorer axis. Errors that the judge repeats within a response merge with the response component, errors that it repeats across responses on one element merge with the element component, and the rest merge with the residual. Bank construction and judge error can thus produce the same coefficient, one that characterises the measurement system as a whole.

Figure 2: KR-20 responds to bank design and to judge error, and bank design dominates at realistic error rates. (a) Full grid, 60 replicates per cell; outlined cells have near-identical coefficients (

0.451

and

0.457

) at twice the judge error. (b) At the measured

4.72\%

error rate, bank design alone moves the coefficient across most of its range (

\pm 1

SD shaded). (c) At the widest bank, judge error lowers the coefficient by

0.04

5\%

and by

0.46

30\%

Question	Quantity that answers it	Requires	Tempting substitute
How often does the judge’s pass or fail match gold?	Accuracy against gold, with cluster intervals	Gold labels	$\Phi(\lambda)$ , a ratio of squared distances, or Livingston–Lewis accuracy, which counts element sampling as error
Where does the judge err, and in which direction?	False-accept and false-reject rates per checklist line	Gold labels	A whole-rubric reliability coefficient
Do the judge’s decisions depend on the wording of its instruction?	Phrasing variance components from a person-by-phrasing study (Appendix A.5 )	Varied phrasings	Agreement across repeated samples at one prompt, which measures decode-time noise
Do the checklist elements rank responses consistently?	KR-20	Neither	KR-20 on judge verdicts read as a property of the judge
Would pass or fail survive another sample of checklist elements?	Livingston–Lewis consistency and accuracy	Neither	Accuracy against gold, which holds the elements fixed

Table 1: Questions that a practitioner asks, and the quantities that answer them. The first three questions concern the judge and require gold labels or varied phrasings. The last two concern the bank and the checklist, and the label-free statistics answer them.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Artefact	File	Contents
Bank	judge_item_bank.csv	210 responses with question id, question and answer text, element text, element-level gold ( $G_{ij}$ ) and judge ( $X_{ij}$ ) verdicts and totals; judge fields blank for the 30 unparsed responses
Estimators	estimators.py	KR-20, KR-21, inter-element correlation, beta-binomial MLE and goodness of fit, Livingston–Lewis DC and CA, variance components and $\Phi(\lambda)$ , cluster bootstrap
Bank statistics	bank_stats.py	Every quantity measured on the real bank, with cluster intervals and the independence check of Section 3
Simulations	sweeps.py	Two-way sweep, measured-rate run, one-directional sweep, $\Phi(\lambda)$ control, Livingston–Lewis estimand control, phrasing identification check
Published grids	sweep_grid.json	Grid values as reported here, including the 60-replicate run at the measured $4.72\%$ error rate
Figures	make_figures.py	Regenerates Figures 2 , A.1 and A.2 from the released data; Figure 1 is a schematic

Appendix

Table A.1: Contents of the released bundle. Running python bank_stats.py , python sweeps.py and python make_figures.py regenerates every quantity and data figure with numpy , scipy , pandas and matplotlib .

Question. How does a refrigerator move heat out of its interior?

Answer as presented to the judge. A refrigerant fluid circulates through a closed loop of coils. Evaporation releases heat into the interior, which is then vented outside separately. Expansion causes the refrigerant to cool to a temperature below the interior. The cycle repeats continuously, net moving heat from cold interior to warm exterior against the natural gradient. Heat release at the condenser causes the refrigerant to condense into a liquid. The cold refrigerant absorbs heat from the interior at the evaporator coils.

Appendix

Table A.2: One response from the bank ( item_id 30, question 3), with gold verdicts

G_{ij}

and judge verdicts

X_{ij}

for all ten elements, quoted verbatim from the released file. An element with

G_{ij}=0

is either absent or represented by its distractor. Here the second sentence of the answer is the distractor for element 9. It inverts the direction of heat flow, and the judge rejects element 9. The judge errs once, accepting element 6, whose slot is empty.

Element position	1	2	3	4	5	6	7	8	9	10
Gold rate	0.506	0.583	0.561	0.533	0.550	0.567	0.561	0.528	0.544	0.578
Judge error	0.144	0.044	0.072	0.061	0.044	0.033	0.033	0.000	0.022	0.017

Appendix

Table A.3: Gold presence rates and judge error rates by element position over the 180 scored responses. Each question has its own checklist, and each position pools 13 different lines. Presence rates stay close to balance by construction, whereas judge error varies by an order of magnitude across positions.

Figure A.1: Bank and judge diagnostics over the 180 scored responses. (a) Gold and judge total-score distributions, with 6 gold totals and 9 judge totals at the ceiling of 10. (b) Judge error rate by element position, pooled over the 13 scored questions, with the aggregate

4.72\%

dashed. (c) Judge against gold totals with the identity line, where the displacement above the line is the judge’s leniency.

	Judge per-element error
Bank spread	0%	4.72%	5%	10%	20%	30%
0.0	-0.000	0.007	-0.014	-0.011	0.015	0.003
0.2	0.111	0.098	0.092	0.062	0.068	-0.009
0.4	0.366	0.317	0.328	0.253	0.174	0.045
0.6	0.575	0.522	0.519	0.451	0.325	0.168
0.8	0.722	0.679	0.682	0.608	0.457	0.266

Appendix

Table A.4: Mean KR-20 over 60 replicates per cell. Reading down any column shows bank design moving the coefficient, and reading across any row shows judge error moving it. The bold column comes from a separate 60-replicate run at the measured

4.72\%

error rate and is the column Section 3 quotes. The

5\%

column belongs to the two-way grid and comes from its own random stream, and the two columns differ by up to

0.021

Bank spread	0%	4.72%	5%	10%	20%	30%
0.0	0.087	0.085	0.111	0.106	0.091	0.112
0.2	0.097	0.083	0.097	0.078	0.093	0.100
0.4	0.061	0.068	0.063	0.085	0.088	0.111
0.6	0.044	0.047	0.048	0.041	0.065	0.084
0.8	0.021	0.030	0.028	0.043	0.059	0.078

Appendix

Table A.5: Per-cell standard deviations for Table A.4 . Precision is lowest at low bank spread, where the coefficient’s true value lies near zero and its sampling variance is largest. We leave the apparent non-monotonicity in the two lowest-spread rows uninterpreted for that reason.

Figure A.2: Two quantities that resemble accuracies without being them. (a)

\Phi(\lambda)

against the judge’s classification accuracy against gold at four cut scores on the 180 scored responses, with arrows marking gaps of

0.22

0.38

. (b) In a simulation without element sampling, where the judge’s true score departs from gold, accuracy against the judge’s own true score exceeds accuracy against gold at every cut. On the real bank, element sampling reverses the sign of this difference (Table A.6 ).

	Accuracy	$95\%$	LL CA	LL CA	Accuracy	$1-$ LL CA	LL DC
Cut	vs gold	interval	judge	gold	$-$ LL CA	(gold)	judge
4	0.961	0.922–0.994	0.882	0.846	0.079	0.154	0.836
5	0.928	0.893–0.962	0.812	0.781	0.116	0.219	0.755
6	0.900	0.851–0.945	0.759	0.750	0.141	0.250	0.697
7	0.917	0.856–0.973	0.750	0.769	0.167	0.231	0.689

Appendix

Table A.6: Livingston–Lewis (LL) estimates on the 180 scored responses. Applied to the error-free gold totals, the estimator falls short of perfect accuracy by more than the judge’s own difference at every cut. The bank’s element sampling dominates the difference column, and the judge’s gate errors offset part of it.

Scenario	$\sigma^{2}(\text{responses})$	$\sigma^{2}(\text{phrasing})$	$\sigma^{2}(\text{resid})$	$\Phi$
Phrasing irrelevant	0.994	0.000	0.090	0.916
Mild phrasing effect	0.994	0.039	0.090	0.886
Large phrasing effect	0.994	0.365	0.090	0.721
Deterministic given phrasing	0.996	0.177	0.000	0.865

Appendix

Table A.7: Identification check over 200 simulations,

N=210

R=3

, by two-way random-effects analysis of variance.

\Phi

refers to a single phrasing and averages the per-simulation values.

Line	Worked wording
Bank	Reliability computed over $K=10$ binary checklist elements on 180 responses from 13 question clusters. SD of gold totals $2.16$ on a $0$ – $10$ scale. Mean inter-element correlation $0.098$ .
Gold comparison	KR-20 is $0.5223$ on judge verdicts and $0.5231$ on gold verdicts. Livingston–Lewis accuracy at a pass mark of five elements is $0.812$ on judge totals and $0.781$ on gold totals. Because error-free scoring lands in the same region, neither value describes the judge.
$\Phi(\lambda)$	Dependability coefficient $\Phi(\lambda=0.5)=0.600$ on judge verdicts and $0.543$ on gold verdicts. This is a ratio of squared distances from the pass mark and is not comparable to the classification accuracy of $0.928$ at the same pass mark.
Judge error	Against gold, the judge accepts $10.4\%$ of absent elements ( $95\%$ cluster interval $5.5\%$ – $15.5\%$ ) and rejects $0.1\%$ of present ones ( $0.0\%$ – $0.3\%$ ). Five of the 130 checklist lines hold 25 of the 85 errors. Gate accuracy at a pass mark of five elements is $0.928$ ( $0.893$ – $0.962$ ) on the 180 scored responses and $0.795$ on all 210 when unparsed outputs count as errors.

Appendix

Table A.8: Worked wording for the four reporting lines of Section 5 , with this study’s values.

Term	Meaning
Response	One constructed short answer, the unit the judge scores; the released file calls it an item
Bank	The set of responses a judge scores, here 210 constructed short answers
Element	One binary checklist criterion, with $K=10$ per question and a separate checklist for each question
Element position	The index $j$ of an element within its question’s checklist
Gold verdict $G_{ij}$	Whether element $j$ is present in response $i$ , fixed by construction
Judge verdict $X_{ij}$	The judge’s YES ( $1$ ) or NO ( $0$ ) for element $j$ of response $i$

Appendix

Table A.9: Terms and notation.

Explore similar work

Jun 17, 2026cs.CL

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.

Justin D. Norman, Michael U. Rivera, D. Alex Hughes

UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708

Llm-As-A-Judge Judges Agreement Validation Discriminative

Ask PDF

Oct 4, 2026cs.CL

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Laurène Vaugrante, Thilo Hagendorff

AI Safety Research Group, University of Stuttgart

Topic pending

Ask PDF

Apr 23, 2026cs.CL

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized. We study repeated identical evaluations on 29 tasks spanning 10 categories using two OpenAI judge models (GPT-4o-mini and GPT-4.1-mini), with 50 pairwise trials and 50 pointwise trials per question, supplemented by temperature and prompt-sensitivity ablations. Across judges, pairwise preferences flip on average 13.6% of the time, with 28% of questions exceeding a 20% flip rate and one question reaching 56%. GPT-4o-mini also exhibits a significant first-position bias (72% A-majority, p = 0.024). At the same time, mean pointwise score gaps are small (0.19--0.36 on a 10-point scale) and not statistically significant in aggregate, producing a pairwise--pointwise gap: judges frequently choose a winner even when their own scalar scores provide little evidence of a meaningful quality difference. Beyond within-judge instability, cross-judge agreement is only 76% (

κ= 0.51

), semantically equivalent prompt templates change majority outcomes in 25% of tested cases, and deterministic decoding reduces but does not eliminate inconsistency. A reliability curve analysis shows that, in our dataset, 11 repeated trials are needed for a majority vote to recover the 50-trial reference verdict with 95% probability on average, rising to 15 for high-variance questions. These findings suggest that single-trial LLM judging is often too noisy for high-stakes evaluation, and that multi-trial aggregation, position randomization, and explicit uncertainty reporting should be standard practice. Because both judges are from a single provider, cross-provider replication remains an important next step.

Abel Yagubyan

Independent Researcher

Llm-As-A-Judge Large Language Model Judges Flips Position Bias

Ask PDF