Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

Longwei Cong; Sonja Hahn; Sebastian Gombert; Leon Camus; Fabian Zehner; Hendrik Drachsler; Ulf Kroehne

cs.CLOct 2, 2026

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

Authors: Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne

Organizations: DIPF | Leibniz Institute for Research and Information in Education · Centre for International Student Assessment (ZIB) · Faculty of Computer Science, Goethe University Frankfurt · Chemnitz University of Technology

Abstract

Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases difficult. In this paper, we study this problem in summarization evaluation from a psychometric perspective. We fit Many-Facet Rasch Models separately to human and LLM ratings to decompose scores into latent summary quality, rater severity, dimension severity, and rating-scale thresholds. Building on this decomposition, we define residual hardness as a model-adjusted measure of judging difficulty and compare whether human and LLM judges share the same hardness structure. Across 17 open-weight LLM judges on SummEval, we find that moderate alignment in latent summary quality does not imply alignment in residual hardness. Human and LLM judges differ in which summary--dimension units remain difficult, and this mismatch is strongly dimension-dependent. Consistency shows a pronounced LLM-hard shift, whereas coherence shows a human-hard shift. We further show that human-easy but LLM-hard cases are partially predictable from observable source--summary properties. These findings suggest that aggregate human alignment reflects only part of LLM-as-a-judge reliability, while psychometric residual diagnostics support more informative judge evaluation and more targeted human--LLM collaboration.

Figures & tables

Calib.	MAE	RMSE	Res. M	Res. SD	Item rel.
Human	0.712	0.892	0.005	1.015	0.859
LLM	0.642	0.831	0.012	1.031	0.948

Table 1: MFRM fit diagnostics. MAE and RMSE are descriptive summaries of rating-level prediction error. Res. M and Res. SD denote the mean and standard deviation of standardized residuals. Item rel. denotes item separation reliability.

Figure 1: Human–LLM alignment in latent summary quality (left), summary-level residual hardness (middle), and summary–dimension residual hardness (right).

Dimension	Human	LLM	Gap
Consistency	50.8	74.7	+23.9
Coherence	57.3	34.8	-22.5
Relevance	43.0	41.6	-1.4
Fluency	48.9	48.9	-0.1

Table 2: Dimension-level residual hardness gaps. Values are percentages of summary–dimension units classified as hard within each judge group. Gap is LLM minus human hard-rate.

System type	Human	LLM	Gap
Extractive	38.9	51.8	+12.9
Abstractive	53.7	49.4	-4.3

Table 3: System-level residual hardness gaps. Values are percentages of summary–dimension units classified as hard. Gap is LLM minus human hard-rate.

Figure 2: Human–LLM residual hardness by evaluation dimension. Dashed lines indicate the overall median residual hardness within each judge group, partitioning the space into human-easy/LLM-easy, human-easy/LLM-hard, human-hard/LLM-easy, and human-hard/LLM-hard regions.

Task	$N$	Pos.	AUROC	AUPRC
All dimensions	6,400	1,494	0.631	0.341
Relevance	1,600	356	0.717	0.442
Consistency	1,600	571	0.633	0.465
Fluency	1,600	374	0.631	0.353
Coherence	1,600	193	0.621	0.197

Table 4: Prediction of human-easy/LLM-hard cases from all summary-dimension units. The positive class is the human-easy/LLM-hard quadrant; all other quadrants are negative.

Figure 3: SHAP beeswarm plots for human-easy/LLM-hard classification by evaluation dimension. Positive SHAP values indicate features that push predictions toward the human-easy/LLM-hard quadrant, while negative values indicate the opposite. Color denotes feature value from low to high.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model
Qwen/Qwen3.5-0.8B
Qwen/Qwen3.5-2B
Qwen/Qwen3.5-4B
Qwen/Qwen3.5-9B
microsoft/Phi-4-mini-instruct
microsoft/Phi-3-mini-4k-instruct

Appendix

Table 5: 17 open-weight LLM judges used for SummEval annotation in our analysis.

Group	Dimension	Exact $\kappa$	Weighted $\kappa$
Human	Coherence	-0.547	-0.854
Human	Consistency	-0.481	-0.865
Human	Fluency	-0.434	-0.855
Human	Relevance	-0.645	-0.828
LLM	Coherence	-0.401	-0.416
LLM	Consistency	-0.374	-0.615

Appendix

Table 6: Spearman correlations between residual hardness and summary–dimension-level agreement contribution. Negative correlations indicate that units with higher residual hardness contribute less to rater agreement.

Panel comparison	Quality $\rho$	Residual $\rho$
All human vs. all LLM	0.371	0.089
All human vs. $\geq 7$ B LLM	0.408	0.039
Experts vs. $\geq 7$ B LLM	0.688	0.092

Appendix

Table 7: Sensitivity of human–LLM alignment to LLM judge capability and human-panel composition. Residual-hardness correlations are computed at the summary–dimension level.

Outcome	Stat.	All raters	Exclude flagged
Quality	$r$	0.372	0.372
	$\rho$	0.371	0.371
Summary hardness	$r$	0.283	0.222
	$\rho$	0.176	0.157
Dim. hardness	$r$	0.145	0.108
	$\rho$	0.089	0.073

Appendix

Table 8: RQ1 sensitivity after excluding raters with infit or outfit mean-square outside the descriptive 0.5–1.5 range.

Dimension	All raters gap	Exclude flagged gap
Coherence	-22.5	-19.8
Consistency	23.9	18.0
Fluency	-0.1	7.3
Relevance	-1.4	-5.5

Appendix

Table 9: RQ2 dimension hard-rate gap sensitivity after excluding flagged raters. Values are percentage-point gaps, computed as LLM hard-rate minus human hard-rate.

Method	Stat.	Summary	Dim.
Main	$r$	0.283	0.145
	$\rho$	0.176	0.089
$z$ -score	$r$	0.283	0.145
	$\rho$	0.176	0.089
Rank	$r$	0.176	0.089
	$\rho$	0.176	0.089

Appendix

Table 10: Residual-hardness alignment under alternative calibration and normalization choices.

Feature group	Features
Summary text	TF–IDF features from the generated summary, using unigram and bigram features with English stop-word removal.
Source text	TF–IDF features from the source document, using unigram and bigram features with English stop-word removal.
Surface features	Summary word count, source word count, compression ratio, summary sentence count, summary digit count, and summary negation count.
Coarse metadata	System type, coded as extractive or abstractive.

Appendix

Table 11: Input features used for RQ3 prediction of human-easy/LLM-hard cases. TF–IDF features use

\texttt{min\_df}=2

, unigram/bigram ranges, and up to 6,000 features each for the summary and source fields.

Feature set	AUROC	AUPRC
Surface only	0.573 [0.555, 0.590]	0.293 [0.273, 0.314]
TF–IDF only	0.620 [0.605, 0.637]	0.326 [0.305, 0.351]
Metadata only	0.542 [0.524, 0.560]	0.260 [0.245, 0.277]
Full model	0.631 [0.614, 0.648]	0.341 [0.319, 0.365]

Appendix

Table 12: RQ3 feature ablation for predicting human-easy/LLM-hard summary–dimension units. Brackets denote 95% bootstrap confidence intervals from out-of-fold predictions. Metadata includes coarse system type only; specific system IDs are excluded.

Split strategy	AUROC	AUPRC
Stratified 5-fold	0.631 [0.614, 0.648]	0.341 [0.319, 0.365]
Summary	0.618 [0.602, 0.634]	0.326 [0.306, 0.348]
Document	0.569 [0.553, 0.586]	0.289 [0.269, 0.310]
System	0.618 [0.602, 0.634]	0.329 [0.309, 0.353]

Appendix

Table 13: RQ3 split robustness for the full model. Brackets denote 95% bootstrap confidence intervals from out-of-fold predictions. Grouped splits hold out all dimensions of the same summary, all summaries from the same source document, or all summaries from the same generation system, respectively.

Task	$N$	Pos.	AUROC	AUPRC
All dimensions	6,400	349	0.676	0.114
Relevance	1,600	102	0.703	0.206
Consistency	1,600	87	0.672	0.133
Fluency	1,600	79	0.743	0.138
Coherence	1,600	70	0.578	0.067

Appendix

Table 14: RQ3 quartile-threshold sensitivity for predicting human-easy/LLM-hard cases from all summary–dimension units.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Jun 2, 2026

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

LLM judges now score most open-ended NLP output, and their mutual agreement is routinely read as evidence that the scores can be trusted. That reading is unsafe: judges may agree because they capture quality, or because they share the same blind spots, and agreement statistics alone cannot tell these apart. We develop a geometric test that can. Treating each judge's scores as a vector, we measure spread, effective rank, the angle to human scores, and the judge-judge, judge-human, and human-human agreement triple for 42 judges on two community-built Indic benchmarks covering four domains and eight languages. Every comparison is reference-matched: a judge and a held-out rater are scored against the same two-rater mean, since an averaged reference otherwise flatters judges by several degrees. On subjective rubrics, judges agree with one another as much as humans do yet reach only 58-66% of human agreement and often concentrate on an axis humans do not weight. On the one rubric with a verifiable answer, most of that gap closes. Ensembles converge on the judges' shared axis rather than the human one, and training widens scores without rotating them. Inter-judge agreement is evidence of human alignment only after this check passes.

Sourabrata Mukherjee, Hamna Hamna, Kalika Bali +1

Microsoft Research

Inter-Rater Reliability Multilingual Language Model Evaluation LLM Evaluation LLM-as-a-Judge LLM Reliability

Ask PDF

Apr 16, 2026

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval:

\textbf{(1)}

a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (

\barρ = 0.8

4.1\%

), with

33

67\%

of documents exhibiting at least one directed 3-cycle; and

\textbf{(2)}

split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed

\geq(1{-}α)

coverage, with set width serving as a per-instance reliability indicator (

r_s = {+}0.576

N{=}1{,}918

p < 10^{-100}

, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (

\bar{r} = 0.32

0.38

), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size

\approx 3.0

) and coherence moderately so (avg. set size

\approx 3.9

), while fluency and consistency remain unreliable (avg. set size

\approx 4.9

). We release all code, prompts, and cached results.

Manan Gupta, Dhruv Kumar

1BITS Pilani, Pilani Campus, India

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation LLM Reliability Summarization Evaluation Automated Evaluation

Ask PDF

Sep 12, 2026

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings

Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faster and cheaper than human evaluation. However, a judge may produce consistent scores without necessarily agreeing with human evaluators. In this work, we study this issue using two local open-weight LLM judges, LLaMA-3-8B and Qwen2.5-7B. We evaluate 300 responses generated by an instruction-tuned GPT-2 (124M) model for 100 questions covering five categories: factual knowledge, instruction following, mathematics, reasoning, and writing. Each response is scored by nine human annotators and is evaluated three times by each LLM judge using the same rubric. We compare the judge scores with the average human scores using Pearson correlation, Spearman correlation, mean absolute error (MAE), signed bias, and self-consistency. LLaMA-3-8B shows a Pearson correlation of 0.275 with human scores, while Qwen2.5-7B achieves 0.340. Their MAEs are 27.71 and 18.64, respectively. Despite this limited agreement, both judges show high self-consistency, with exact consistency rates of 97.3% for LLaMA-3-8B and 92.3% for Qwen2.5-7B. These results show that high self-consistency does not necessarily indicate high agreement with human judgments. Our findings highlight the need to evaluate both consistency and human alignment when using local LLMs as automatic judges.

Aakash Kumar Tiwari

Department of Mathematics Indian Institute of Technology Kharagpur

LLM-as-a-Judge Bias in LLM-as-a-Judge Self-Consistency in Language Models

Ask PDF

Abstract

LLM-as-a-Judge

Explore similar work

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings