LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora

Veerendra Kumar Sunkavalli

cs.CLAug 30, 2026

LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora

Authors: Veerendra Kumar Sunkavalli

Organizations: Independent Researcher

Abstract

Large language models (LLMs) are increasingly used as essay graders in learning analytics, evaluated almost exclusively with agreement statistics. Educational measurement warns that raters also differ in severity, show halo, and drift as instruments. We treat LLM judges as raters and run a pre-registered rater-effects battery (many-facet Rasch severity, residual halo, generalizability/decision studies, cross-version shifts, differential functioning) on public corpora in two languages (ENEM/Essay-BR; ASAP): 2,377 essays, 12 judges, 4 providers, 5 version contrasts, replicated cells, released as a score tensor. Judge severity spans 219 points on ENEM's 0-1000 scale; on ASAP the panel spread is 15-33% of the score range against a between-trained-human gap near 1%. Judge-human correlations sit in an undiscriminating .47-.56 band. All five version contrasts shift severity beyond a family-wise permutation null (up to 133 points), and one judge was deprecated mid-study, caught by identity canaries. Two pre-registered tests returned honest nulls: severity-adjusted leaderboard reversals did not survive a permutation null, and "silent drift" was refuted: agreement moved with severity in four of five contrasts. Replication yields self-consistency (phi>=.80 at k<=2) but not human-level accuracy, and a same-instrument check overturned our own halo comparison: matched on instrument and calibration, we find no credible evidence that judge halo exceeds the trained-human range.

Figures & tables

Prior work	Contribution	Delta of this study
Judge datasheet Usami et al. (2026)	Psychometric reporting protocol for LLM judges	A crossed rater-effects experiment (severity/fit, halo, G/D-studies, versions), not a protocol
IRT on LLM short-answer graders Cong et al. (2026)	Grader ability/difficulty via IRT (ASAG)	Rater facets on essay scoring: severity CIs, halo, replications, version pairs, two languages
Agreement-metric guidance Rao and Callison-Burch (2026)	What agreement statistics to report	Quantifies what those statistics cannot see (219-pt severity inside a .09-wide $r$ band)
G-theory for LLM autoscoring Song et al. (2025)	Variance components for LLM writing scores	Multi-provider crossed panel + severity/halo/version battery + trained-human benchmarks
Variance components of LLM nondeterminism Żatuchin (2026)	D-study allocation for brand answers	Education measurement; judges as raters; validity (H5b) separated from stability (H5a)
MFRM correction of human labels Casabianca and Beiting-Parrish (2026)	Humans as raters in AI pipelines	The mirror design: LLMs as raters, human-anchored origin, cross-version instability

Table 1: Positioning against the nearest prior work: what each contributes, and what this study adds.

Judge	Provider	Tier	Version-pair role
sonnet-4	Anthropic	mid	sonnet steps; legacy-gated mid-study
sonnet-4.5	Anthropic	mid	sonnet steps
sonnet-4.6	Anthropic	mid	sonnet steps
opus-4.5	Anthropic	premium	opus step
opus-4.6	Anthropic	premium	opus step
haiku-4.5	Anthropic	cheap	–

Table 2: Judge roster: 12 models, 4 providers, 3 price tiers, 5 version contrasts: 4 within-family steps + 1 same-provider cross-generation step (llama). Full platform identifiers, per-call response metadata, and canary logs are in the artifact.

ID	Status at freeze (v1.1)	Outcome
H1 severity	confirmatory	Supported (§ 4.1 )
H2 leaderboard (orig.)	demoted to descriptive at v1.1	reported via H2b
H2b reversal test	confirmatory	Null (§ 4.2 )
H3 halo	confirmatory	knife-edge cross-instrument pass; reversed by same-instrument check (§ 4.3 )
H4 cross-corpus DJF	exploratory at v1.1	reported (§ 4.8 )
H5a dependability	confirmatory	Supported , temp-conditioned (§ 4.4 )

Table 3: Complete map of pre-registered hypotheses to outcomes. H2 (descriptive) and H4 (exploratory) were re-scoped in the pre-registration amendment itself (before any confirmatory statistic and before English collection), not post hoc.

Figure 1: Each point is one judge scoring the same 977 ENEM essays (

K{=}3

, rep-mean). Horizontal axis: severity (mean displacement from the human reference, points on the 0–1000 scale). Vertical axis: Pearson correlation with the human total. The largest rater effect in the study (219-point severity spread) is orthogonal to the statistic by which judges are commonly selected (all

r\in[.47,.56]

). Colors group model lines (sonnet, opus, haiku, nova, llama, qwen).

Judge	Severity	99% CI	Raw pts	$r$	QWK	Halo	RMSE k=3	$k^{*}$
llama3.3-70b	-0.38	[-0.54, -0.18]	+30	0.50	0.38	0.38	183	1
nova-2-lite	+0.28	[0.11, 0.42]	-23	0.50	0.33	0.49	165	2
llama4-mav	+0.49	[0.30, 0.63]	-40	0.53	0.38	0.16	165	1
qwen3-32b	+0.68	[0.47, 0.82]	-55	0.49	0.24	0.38	166	1
sonnet-4.5	+0.71	[0.53, 0.84]	-56	0.51	0.36	0.15	170	1
nova-pro	+0.81	[0.62, 0.93]	-63	0.51	0.31	0.38	167	2

Table 4: Judge-level rater effects on ENEM (977 essays,

K{=}3

). Severity point estimates and 99% essay-cluster bootstrap CIs are both on the JMLE logit scale (human origin), so intervals and points are directly comparable (bootstrap distributions are left-skewed for the most severe judges, hence the asymmetric CIs); raw points on the 0–1000 scale;

r

= Pearson correlation and QWK = quadratically weighted kappa vs. the human reference (QWK per competency on six categories, averaged); halo = mean off-diagonal residual inter-competency correlation; RMSE at

k{=}3

(points/1000);

k^{*}

= replications to reach

\phi\geq.80

at temperature 0.7.

Judge	Set 1	Set 2	Set 7	Set 8	Mean
haiku-4.5	-42.3	-26.7	-31.9	-24.5	-31.4
nova-lite	-26.2	-21.2	-30.6	-9.5	-21.9
sonnet-4.5	-23.8	-13.4	-35.8	-14.0	-21.7
llama4-mav	-25.2	-12.2	-38.9	-8.2	-21.1
llama3.3-70b	-26.5	-18.0	-27.5	-4.3	-19.1
sonnet-4.6	-23.6	-15.9	-27.9	-6.4	-18.4

Table 5: Per-judge severity on ASAP, as percent of each set’s score range (judge

k{=}3

rep-mean minus the two trained raters’ mean; negative = more severe than humans). Bottom row: the severity difference between the two trained human raters themselves — the human yardstick. English severity orderings are prompt-conditional (paraphrase

\tau=.56

); the spread magnitude is more stable (set-1 spread 26% of range under the paraphrased rubric vs. 33%).

Component	$\hat{\sigma}^{2}$	% of total
Prompt	836	3.5%
Essay (within prompt)	13,716	56.7%
Judge (severity)	4,048	16.7%
Judge $\times$ prompt	245	1.0%
Judge $\times$ essay	3,471	14.3%
Replication (within cell)	1,877	7.8%

Table 6: Variance components, ENEM total score (ANOVA-EMS estimators; (essay:prompt)

\times

judge design with replications nested in cells; 977 essays

\times

12 judges

\times

3 replications, complete).

Calibration	held-out RMSE	FF / FP (nova-lite)
none (raw)	190.7	.56 / .01
mean offset	160.4	.10 / .16
linear equating	154.2	.10 / .16
equipercentile	177.5	.14 / .14

Table 7: Treatment arm (post-registration, exploratory): panel-mean held-out RMSE (ENEM points; 200 splits, 400 eval essays, anchors

n{=}100

; 95% split intervals in the artifact) and false-fail/false-pass shares (of all essays) for the most severe cheap judge (nova-lite) at the 600-point cutscore. Reference = pooled platform score (scale reference, not error-free ground truth).

Figure 2: (a) Severity shifts across the five version contrasts against the family-wise sign-flip permutation 99% null band (ENEM, all pair members canary-clean). (b) RTM-robust differential severity: per-judge quality slopes (displacement vs. rater 2, essays binned by rater 1) as a fraction of each ASAP set’s range; orange diamonds are the rater-1-vs-rater-2 placebo slopes, which have the opposite sign where compression holds.

Pair	$\Delta$ sev	$>$ null	$\Delta$ QWK CI	equiv.
sonnet-4 $\to$ sonnet-4.5	+45	yes	[+0.025, +0.059]	no
sonnet-4.5 $\to$ sonnet-4.6	-133	yes	[-0.128, -0.090]	no
opus-4.5 $\to$ opus-4.6	-19	yes	[-0.021, -0.001]	no
nova-lite $\to$ nova-2-lite	+125	yes	[+0.122, +0.172]	no
llama3.3-70b $\to$ llama4-mav	-71	yes	[-0.013, +0.030]	no

Table 8: Cross-version severity shifts (ENEM).

\Delta

sev in raw points (positive = newer version more lenient); ¿null = exceeds the family-wise sign-flip permutation 99% max-statistic;

\Delta

QWK 90% bootstrap CI and TOST equivalence at

\pm.02

. The pre-registered silent-drift conjunction failed for every pair.

Explore similar work

LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias
Mar 31, 2026Filip J. Kucia, Anirban Chakraborty, Anna WróblewskaAutomated Essay Scoring Language Model Bias Evaluation
Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK
LLMs Do Not Grade Essays Like Humans
Mar 24, 2026Jerin George Mathew, Sumayya Taher, Anindita Kundu +1Automated Essay Scoring LLM-as-a-Judge
University of Alberta, 116 St & 85 Ave, Edmonton, AB, T6G 2R3, Canada
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart

Mar 31, 2026J/K move · Enter open · S save

LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

Filip J. Kucia, Anirban Chakraborty, Anna Wróblewska

Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK

Automated Essay Scoring Language Model Bias Evaluation Rubric-Based Evaluation Prompt Sensitivity

Open paper AskPDF arXiv

Despite growing interest in using Large Language Models (LLMs) for educational assessment, it remains unclear how closely they align with human scoring. We present a systematic evaluation of instruction-tuned LLMs across three open essay-scoring datasets (ASAP 2.0, ELLIPSE, and DREsS) that cover both holistic and analytic scoring. We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates. Our results show that strong open-weight models achieve moderate to high agreement with humans on holistic scoring (Quadratic Weighted Kappa about 0.6), but this does not transfer uniformly to analytic scoring. In particular, we observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions, meaning that models often score these traits more harshly than human raters. We also find that concise keyword-based prompts generally outperform longer rubric-style prompts in multi-trait analytic scoring. To quantify the amount of data needed to detect these systematic deviations, we compute the minimum sample size at which a 95% bootstrap confidence interval for the mean bias excludes zero. This analysis shows that LOC bias is often detectable with very small validation sets, whereas Higher-Order Concern (HOC) traits typically require much larger samples. These findings support a bias-correction-first deployment strategy: instead of relying on raw zero-shot scores, systematic score offsets can be estimated and corrected using small human-labeled bias-estimation sets, without requiring large-scale fine-tuning.

Abstract

Explore similar work

LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

LLMs Do Not Grade Essays Like Humans

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models