A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

Jianlin Chen; Wenhui Chen; Ziyao Lin; Chi Man Vong

cs.AIAug 25, 2026

A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

Authors: Jianlin Chen, Wenhui Chen, Ziyao Lin, Chi Man Vong

Organizations: South China University of Technology · University of Macau

Abstract

LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile: invariance S, the probability that a verdict is unchanged under construct-preserving edits, and construct sensitivity R, the probability that it changes under minimal construct-changing edits. We show that S and R are independent and that no scalar summary preserves all relevant comparisons. We measure the profile across 7 judges and 4 domains using 7 construct-changing intervention types and 5 register-only controls, with intervention direction determined by human annotators and generation, verification, and judging assigned to disjoint model families. At matched invariance S >= 0.90, judges average S = 0.945 but R = 0.319. Sensitivity also differs between scope and strength edits: R_scope = 0.383 versus R_strength = 0.262, a +0.121 gap with the same sign for all 7 judges. We further audit five public label sets and find that surface-only predictors reproduce 55%-67% of labels in paired mode, including 67.4% of MT-Bench human votes. These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated, motivating joint reporting of invariance and sensitivity and auditing the validation set itself.

Figures & tables

Figure 1: The paper’s claim structure, in two tiers. The left group is the claim that stands on its own: no external label set enters C1, C2 or C3, and direction is assigned by humans under Protocol 1 . The right group is support, and each of its stages carries the consequence of its own null result in the dashed box beneath it. The independence is the figure’s shape rather than an assertion in this caption, and it is why the premise cannot evaporate on one experiment (§ 1.1 ).

Figure 2: The two conditions of Definition 2 on one claim. Left: the property the judge literature measures: an edit that changes register, restatement, clause order, length or formatting must not move the verdict. Hedging is deliberately absent from that list, for the reason in Appendix B . Right: the property we could not find measured: an edit that changes what the claim covers or how hard it commits must move it. The right column’s frame is drawn open because the correct verdict there is set by human annotation (§ 4.2 ), not read off the sentence.

Figure 3: The

(S,R)

plane. The two degenerate instruments of Proposition 2 are the marked corners: a judge that never moves sits at

(1,0)

, and one that flips on anything sits at

(0,1)

. The dashed line joining them is the level set on which a symmetric scalar summary scores them identically (Corollary 1 ), which is why this paper reports the pair.

\mathcal{F}(J)

traces one judge as

\tau

varies, and its default

\tau

J_{b}

, is the only point usually reported.

J_{a}

sits at the same invariance as

J_{b}

and in a different quadrant, with the projection to the axis drawn: what invariance testing alone resolves is that one coordinate, which does not separate valid from insensitive . The corners are glossed in Table 1 .

	$R$ high	$R$ low
$S$ high	valid	insensitive : reliable, and measuring something else
$S$ low	unstable: may be flipping indiscriminately	invalid

Table 1: The four corners of the profile, and what each is. The two cells in the top row are the ones that matter: invariance testing alone cannot separate them, because it does not measure the coordinate on which they differ.

work	establishes	assumes, and we test
Judge reliability audits [ 38 ]	judges are unreliable, and exact-match agreement overstates discrimination	that the label sets are ground truth. Judges are measured against fixed sets, never audited
Mechanistic accounts of judge bias [ 55 ]	surface manipulations move judge scores, visibly in activations	only specificity : edits that should not flip a verdict. Sensitivity is untested, and no bias type there is scope or quantifier
Noise-corrected evaluation [ 14 ]	TPR/FPR estimated on a calibration set can be corrected for	that the calibration set is sound. The guarantees inherit whatever it leaks
Judge input-shortcut probes [ 33 ]	judges follow injected metadata cues and never acknowledge them	that the shortcut is in reading the input. Perturbs inputs, not the construct, and audits no label set
Automatic factual-consistency metrics [ 57 ]	claim support can be scored by one alignment function	that the axis of interest is support. A mis-scoped claim is not unsupported, so such a metric is silent by construction, not by weakness
Surface and style confounds, in judges and in reward models [ 60 , 54 , 59 , 22 , 29 , 32 , 30 , 7 , 56 , 53 ]	verbosity, order, formatting and length shift verdicts, and preference labels are recoverable from them, well enough that benchmarks are now designed against it	that the property should not have mattered, and that the confound is model-side. Every one is an invariance test; the label-set side has no headroom metric, no cross-set audit, no reporting convention

Table 2: What the nearby results establish, and the assumption each one needs that we test.

Figure 4: The decomposition on one worked claim. From a single licensed sentence, scope moves right (the referent set grows while commitment is untouched) and strength moves up: the referent set is fixed and commitment rises. The seven

T_{\mathrm{C}}

edit types sit beside the axis each moves, and in each cell the changed span is coloured with the axis that moved it; every cell is one edit from the origin. The diagonal is the case the verifier rejects: an item that moved both axes cannot separate them, and admitting it would contaminate the very contrast § 5.2 rests on.

slot	axis	$n$	agree	usable	note
domain_extension	scope	25	0.973	25
population_widening	scope	42	0.857	34
quantifier_strengthening	scope	32	0.958	31
tense_modality_shift	scope	45	0.852	26	low yield
condition_removal	strength	35	0.867	33
hedging_removal	strength	58	0.851	48

Table 3: Human direction annotation, per slot. agree is mean pairwise agreement over three annotators; usable counts items whose majority label matches the intervention’s intended direction. The two columns answer different questions and a slot can fail either one alone: tense_modality_shift has ordinary agreement and low yield — readers agreeing that the edit changed nothing — while the two length-manipulating invariance slots have ordinary yield and agreement below the pre-set bar of 0.75.

intervention	axis	$n$	human dir.	agreement	flip rate
$T_{\mathrm{C}}$ — should flip: scope
domain extension	scope	25	100%	0.973	0.445
population widening	scope	34	81%	0.857	0.299
quantifier strengthening	scope	31	97%	0.958	0.445
tense modality shift	scope	26	58%	0.852	0.358
$T_{\mathrm{C}}$ — should flip: strength

Table 4: Validity profiles.

S

from the

T_{\mathrm{I}}

control arm,

R

from

T_{\mathrm{C}}

split by axis. Per-slot

n

and human confirmation are part of the result, not appendix material: § 4.2 explains why.

n

counts items whose majority human label matched the intervention’s intent and whose slot met the agreement bar: the items that actually enter the arm. human dir. is that count as a fraction of items annotated, so a low value marks a slot the generator often failed to move rather than one humans could not read: tense modality shift is the clearest case, at 57.8% with ordinary agreement. † below the pre-set bar of 0.75 and therefore excluded (§ 4.2 ). Pipeline discard before annotation was 24.2% overall, dominated by the length tolerance; it is reported as an aggregate because build_pairs.py keys its rejections by reason rather than by slot, and a per-slot figure we did not record is not one we will estimate.

Panel of Figure 5 — Figure 5: Validity profiles for 7 judges. Frontiers are drawn for the judges that expose a decision threshold, which under Remark 1 is all of them that grade; any judge that will not grade is a single labelled point, per Proposition 3 . The dashed diagonal is $S+R=1$ , where a judge is trading one property for the other rather than possessing both. Curves are the monotone hull over our threshold sweep: at each invariance we keep the best sensitivity achieved at that invariance or stricter, so a curve is a frontier rather than a trace of every threshold. Every vertex is an attained threshold, so a long straight segment is interpolation between two of them and not a measured path; the ringed vertex is the matched-invariance operating point Table 5 reads off.

judge	$\tau$	$S$	$R$	$R^{\textsc{scope}}$	$R^{\textsc{strength}}$	pts	regrade
anthropic/claude-haiku-4.5	2.75	0.917	0.561	0.629	0.500	9	0.41
nvidia/nemotron-3-nano-30b-a3b	0.25	0.906	0.447	0.474	0.423	7	0.83
google/gemini-2.5-flash-lite	0.25	0.968	0.408	0.474	0.349	6	1.00
mistralai/mistral-small-3.2-24b-instruct	2.25	0.958	0.309	0.405	0.223	4	0.96
deepseek/deepseek-v4-flash	3.25	0.906	0.245	0.362	0.140	9	0.68
z-ai/glm-4.7-flash	0.25	0.979	0.168	0.239	0.107	2	0.98

Table 5: Per-judge profiles.

R

is reported per axis and never without

S

(Corollary 1 ); the final column records whether the row was read off a frontier or a default threshold, because the two are not comparable. † predicted, not measured

intervention	axis	$n$	$R$	median $\|\Delta\|$ length
quantifier_strengthening	scope	31	0.445	3.2%
domain_extension	scope	25	0.445	8.8%
tense_modality_shift	scope	26	0.358	4.8%
population_widening	scope	34	0.299	7.0%
condition_removal	strength	33	0.379	13.0%
intensifier_addition	strength	49	0.235	4.3%

Table 6: Sensitivity per intervention, at each judge’s matched-invariance threshold, averaged over the 7 judges. The magnitude column is the median

|\Delta|

length the operation mechanically carries: it is a property of the edit type, not a free variable, which is why magnitude cannot be stratified independently of slot.

Panel of Figure 6 — Figure 6: Sensitivity per intervention axis, per judge, at matched $S$ , with the control-arm flip rate on the same scale. The control bar is the reference that makes the other two interpretable: a construct-edit bar near the control bar is a judge responding to a construct change no more than to an edit that should not move it at all. Judge labels are shortened vendor names; per-judge thresholds and instrument quality are in Table 5 .

		per-item mode		paired mode
label set	label origin	$n$	$\mathrm{a}_{\mathrm{imp}}$	pairs	$\kappa$	accuracy
RewardBench	pair role	5,970	0.303	2,985	0.249	62.4%
RewardBench 2	pair role	8,977	0.200	5,926	0.248	62.4%
RM-Bench	pair role	7,962	0.221	3,981	0.095	54.7%
HelpSteer2	human rating	8,008	0.068	—	—	—
MT-Bench (human)	human vote	6,710	0.300	2,575	0.348	67.4%

Table 7: Construct-blind recovery of the label sets the literature validates judges against, at the revisions pinned in results/labelsets/PROVENANCE.json , in both input modes of Definition 6 . Per-item

\mathrm{a}_{\mathrm{imp}}

is chance-corrected agreement; paired accuracy has chance at

50\%

with side order randomised. Group-aware folds throughout; every paired row clears its label-permutation null, which sits at or below 0.026. HelpSteer2 has no paired column because it ships per-response ratings rather than comparisons. The final row is a provenance-inherited set audited under the identical protocol.

Figure 7: Why a provenance-inherited label set is separable without reasoning about the construct, and what the blinded design withholds. Left: labels attach to whole records by their role in the pair, so one threshold on response length reproduces them, at the agreement reported by Chen et al. [6] . Right: the unit is one sentence with its evidence context, every field that identifies its side withheld, sides interleaved so pair membership is unrecoverable, and the label the majority of three independent annotators with no arbitration pass. Both panels read in two bands: above the rule, what the annotator sees and how its label is formed; below it, the surface route the design leaves open or closes. A class chip is drawn identically wherever it appears, so the two classes a pair role can supply compare directly against the four.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The unit of analysis. ( 6 ) weights base items equally and, within an item, its interventions equally; pooling all interventions instead lets a hedge-dense sentence dominate the axis it is most relevant to, since a sentence carrying several hedges admits every strength slot and a bare one admits none. Intervals resample whole base items with replacement and never interventions, and folds split on the source document. The per-item counts drawn here are illustrative; measured per-slot yields are in Table 4 .

type	class / axis	what it does
domain extension	$T_{\mathrm{C}}$ / scope	extend the claim to an adjacent domain
population widening	$T_{\mathrm{C}}$ / scope	widen who or what the claim covers
quantifier strengthening	$T_{\mathrm{C}}$ / scope	strengthen a quantifier (some $\to$ all, often $\to$ always)
tense modality shift	$T_{\mathrm{C}}$ / scope	restate a one-time observation as a standing property
condition removal	$T_{\mathrm{C}}$ / strength	drop the circumstances under which it was observed
hedging removal	$T_{\mathrm{C}}$ / strength	remove a hedge the evidence relied on

Appendix

Table 8: Every intervention type, its class, and the axis it moves. Frozen in probes/edit_taxonomy.py before any judge was run; the assertion in that file fails if a type is added without an axis, so the table and the code cannot drift.

Figure 9: One pair’s route through Protocol 1 , and the five places it can stop. Four outcomes are decided per item. The fifth is decided per slot and per axis after aggregation, against a bar fixed in probes/edit_taxonomy.py before collection, and it excludes the slot whole rather than widening an interval. Of 478 annotated pairs, 246 construct and 96 control items survive, 2 carry no label and 2 slots were excluded; per-slot yields are in Table 4 . The basis field is a mandatory quotation of the span that exceeds the evidence, which is what makes a disagreement inspectable rather than a number, and presentation order is recorded so a systematic first-position preference is detectable rather than absorbed into the labels.

Panel of Figure 10 — Figure 10: Instrument quality per judge, and the two independent ways a graded judge fails. Left: the grade values a judge emits, against those surviving the one-percent usability cut, so the segment is resolution the judge appears to offer and does not deliver. Right: whether it reproduces its own grade on the same item at temperature $0$ . Rows are sorted by usable points and the regrade column does not follow that order, so neither quantity predicts the other. Every judge here emits at least seven distinct grades, which is why a study reporting distinct grades would have called all of them fine.

predictor	sees	does not see
length_only	response length in characters	any token identity
numeric_surface	counts of digits, punctuation, and casing patterns	any word
surface_ngram	character $n$ -grams, $n\leq 4$	sentence structure, evidence context

Appendix

Table 9: The three members. Feature counts and fitting procedure are fixed in analysis/validation_power.py ; group-aware cross-validation splits on source document so no document appears in both folds.

key	value	basis	refuted if
C2 — the validity profile
Sinv	0.88	Judges are already known to be fairly invariant to individual surface properties, and the strongest published verbosity-bias estimate is small ( reliability2026 , under 0.011 correlation across 21 judges). But our control arm is six properties at once including two with published effects, so invariance should sit below what any single-property study reports. High, not near-1.	$S<0.75$ — meaning the control arm is finding failures the single-property literature misses, which would be a finding in its own right and would make every $R$ comparison harder
Rsens	0.41	Pure judgement, and the weakest-founded number here. Anchored on the intuition that a judge does respond to some construct edits — quantifier changes are lexically salient — but not most. If it were near 0.8 the gap would have been noticed already; if near 0.1 judges would fail obviously broken cases.	$R>0.75$ at $S\approx 0.88$ — judges are construct-sensitive and the paper’s headline is wrong
nJudges	6	The scale the external review set as the floor for a claim about instruments rather than about one vendor.	— (design parameter, not a prediction)
nDomains	4	Same. See PEER_REVIEW_VWV.md §5 for the minimum-viable fallback of two.	—
C3 — the scope/strength asymmetry

Appendix

Table 10: Register of predicted values. Every † predicted, not measured value in this paper, with the reasoning that produced it and the measurement that would refute it. A dash in the final column marks a design parameter or a value with no single refuting observation.

quantity	why not measured here
Per-row commentary on each public set’s recoverability	needs construction details the sets do not publish
Which audited set each checklist item would have flagged	needs construction details the sets do not publish
Quantisation and API-version instrument columns	most providers do not disclose them
Self-consistency and ensemble remedy rows	two further judges in the same harness; not yet run
Sensitivity under a finer verdict space	needs a second elicitation with its own validation
The profile per language	probe items are English only

Appendix

Table 11: What this paper specifies and does not measure, so that a deliberate omission is distinguishable from an oversight. Two omissions are argued in place instead, where the omission bears on how a nearby number should be read: the accuracy-prompting arm (§ 5.3 ) and the blinded replacement set (§ 5.5 ).

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Jun 17, 2026

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.

Justin D. Norman, Michael U. Rivera, D. Alex Hughes

UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708

Inter-Rater Reliability LLM Evaluation LLM-as-a-Judge LLM Reliability Language Model Bias Evaluation Benchmark Validity+1

Ask PDF

Jul 9, 2026

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.

Zongyou Yang, Yinghan Hou, Xiaokun Yang

Dyson School of Design Engineering, Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering, Imperial College London London, United Kingdom

LLM-as-a-Judge LLM Auditing Language Model Bias Evaluation Benchmark Validity

Ask PDF

Apr 16, 2026

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval:

\textbf{(1)}

a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (

\barρ = 0.8

4.1\%

), with

33

67\%

of documents exhibiting at least one directed 3-cycle; and

\textbf{(2)}

split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed

\geq(1{-}α)

coverage, with set width serving as a per-instance reliability indicator (

r_s = {+}0.576

N{=}1{,}918

p < 10^{-100}

, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (

\bar{r} = 0.32

0.38

), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size

\approx 3.0

) and coherence moderately so (avg. set size

\approx 3.9

), while fluency and consistency remain unreliable (avg. set size

\approx 4.9

). We release all code, prompts, and cached results.

Manan Gupta, Dhruv Kumar

1BITS Pilani, Pilani Campus, India

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation LLM Reliability Summarization Evaluation Automated Evaluation

Ask PDF

Abstract

LLM-as-a-Judge

Explore similar work

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations