Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Authors: Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus, Fabian Zehner, Hendrik Drachsler, Ulf Kroehne
Organizations: DIPF | Leibniz Institute for Research and Information in Education · Centre for International Student Assessment (ZIB) · Faculty of Computer Science, Goethe University Frankfurt · Chemnitz University of Technology
Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases difficult. In this paper, we study this problem in summarization evaluation from a psychometric perspective. We fit Many-Facet Rasch Models separately to human and LLM ratings to decompose scores into latent summary quality, rater severity, dimension severity, and rating-scale thresholds. Building on this decomposition, we define residual hardness as a model-adjusted measure of judging difficulty and compare whether human and LLM judges share the same hardness structure. Across 17 open-weight LLM judges on SummEval, we find that moderate alignment in latent summary quality does not imply alignment in residual hardness. Human and LLM judges differ in which summary--dimension units remain difficult, and this mismatch is strongly dimension-dependent. Consistency shows a pronounced LLM-hard shift, whereas coherence shows a human-hard shift. We further show that human-easy but LLM-hard cases are partially predictable from observable source--summary properties. These findings suggest that aggregate human alignment reflects only part of LLM-as-a-judge reliability, while psychometric residual diagnostics support more informative judge evaluation and more targeted human--LLM collaboration.
Figures & tables
Calib.
MAE
RMSE
Res. M
Res. SD
Item rel.
Human
0.712
0.892
0.005
1.015
0.859
LLM
0.642
0.831
0.012
1.031
0.948
Table 1: MFRM fit diagnostics. MAE and RMSE are descriptive summaries of rating-level prediction error. Res. M and Res. SD denote the mean and standard deviation of standardized residuals. Item rel. denotes item separation reliability.
Figure 1: Human–LLM alignment in latent summary quality (left), summary-level residual hardness (middle), and summary–dimension residual hardness (right).
Dimension
Human
LLM
Gap
Consistency
50.8
74.7
+23.9
Coherence
57.3
34.8
-22.5
Relevance
43.0
41.6
-1.4
Fluency
48.9
48.9
-0.1
Table 2: Dimension-level residual hardness gaps. Values are percentages of summary–dimension units classified as hard within each judge group. Gap is LLM minus human hard-rate.
System type
Human
LLM
Gap
Extractive
38.9
51.8
+12.9
Abstractive
53.7
49.4
-4.3
Table 3: System-level residual hardness gaps. Values are percentages of summary–dimension units classified as hard. Gap is LLM minus human hard-rate.
Figure 2: Human–LLM residual hardness by evaluation dimension. Dashed lines indicate the overall median residual hardness within each judge group, partitioning the space into human-easy/LLM-easy, human-easy/LLM-hard, human-hard/LLM-easy, and human-hard/LLM-hard regions.
Task
N
Pos.
AUROC
AUPRC
All dimensions
6,400
1,494
0.631
0.341
Relevance
1,600
356
0.717
0.442
Consistency
1,600
571
0.633
0.465
Fluency
1,600
374
0.631
0.353
Coherence
1,600
193
0.621
0.197
Table 4: Prediction of human-easy/LLM-hard cases from all summary-dimension units. The positive class is the human-easy/LLM-hard quadrant; all other quadrants are negative.
Figure 3: SHAP beeswarm plots for human-easy/LLM-hard classification by evaluation dimension. Positive SHAP values indicate features that push predictions toward the human-easy/LLM-hard quadrant, while negative values indicate the opposite. Color denotes feature value from low to high.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Qwen/Qwen3.5-0.8B
Qwen/Qwen3.5-2B
Qwen/Qwen3.5-4B
Qwen/Qwen3.5-9B
microsoft/Phi-4-mini-instruct
microsoft/Phi-3-mini-4k-instruct
Appendix
Table 5: 17 open-weight LLM judges used for SummEval annotation in our analysis.
Group
Dimension
Exact κ
Weighted κ
Human
Coherence
-0.547
-0.854
Human
Consistency
-0.481
-0.865
Human
Fluency
-0.434
-0.855
Human
Relevance
-0.645
-0.828
LLM
Coherence
-0.401
-0.416
LLM
Consistency
-0.374
-0.615
Appendix
Table 6: Spearman correlations between residual hardness and summary–dimension-level agreement contribution. Negative correlations indicate that units with higher residual hardness contribute less to rater agreement.
Panel comparison
Quality ρ
Residual ρ
All human vs. all LLM
0.371
0.089
All human vs. ≥7 B LLM
0.408
0.039
Experts vs. ≥7 B LLM
0.688
0.092
Appendix
Table 7: Sensitivity of human–LLM alignment to LLM judge capability and human-panel composition. Residual-hardness correlations are computed at the summary–dimension level.
Outcome
Stat.
All raters
Exclude flagged
Quality
r
0.372
0.372
ρ
0.371
0.371
Summary hardness
r
0.283
0.222
ρ
0.176
0.157
Dim. hardness
r
0.145
0.108
ρ
0.089
0.073
Appendix
Table 8: RQ1 sensitivity after excluding raters with infit or outfit mean-square outside the descriptive 0.5–1.5 range.
Dimension
All raters gap
Exclude flagged gap
Coherence
-22.5
-19.8
Consistency
23.9
18.0
Fluency
-0.1
7.3
Relevance
-1.4
-5.5
Appendix
Table 9: RQ2 dimension hard-rate gap sensitivity after excluding flagged raters. Values are percentage-point gaps, computed as LLM hard-rate minus human hard-rate.
Method
Stat.
Summary
Dim.
Main
r
0.283
0.145
ρ
0.176
0.089
z -score
r
0.283
0.145
ρ
0.176
0.089
Rank
r
0.176
0.089
ρ
0.176
0.089
Appendix
Table 10: Residual-hardness alignment under alternative calibration and normalization choices.
Feature group
Features
Summary text
TF–IDF features from the generated summary, using unigram and bigram features with English stop-word removal.
Source text
TF–IDF features from the source document, using unigram and bigram features with English stop-word removal.
Surface features
Summary word count, source word count, compression ratio, summary sentence count, summary digit count, and summary negation count.
Coarse metadata
System type, coded as extractive or abstractive.
Appendix
Table 11: Input features used for RQ3 prediction of human-easy/LLM-hard cases. TF–IDF features use min_df=2 , unigram/bigram ranges, and up to 6,000 features each for the summary and source fields.
Feature set
AUROC
AUPRC
Surface only
0.573 [0.555, 0.590]
0.293 [0.273, 0.314]
TF–IDF only
0.620 [0.605, 0.637]
0.326 [0.305, 0.351]
Metadata only
0.542 [0.524, 0.560]
0.260 [0.245, 0.277]
Full model
0.631 [0.614, 0.648]
0.341 [0.319, 0.365]
Appendix
Table 12: RQ3 feature ablation for predicting human-easy/LLM-hard summary–dimension units. Brackets denote 95% bootstrap confidence intervals from out-of-fold predictions. Metadata includes coarse system type only; specific system IDs are excluded.
Split strategy
AUROC
AUPRC
Stratified 5-fold
0.631 [0.614, 0.648]
0.341 [0.319, 0.365]
Summary
0.618 [0.602, 0.634]
0.326 [0.306, 0.348]
Document
0.569 [0.553, 0.586]
0.289 [0.269, 0.310]
System
0.618 [0.602, 0.634]
0.329 [0.309, 0.353]
Appendix
Table 13: RQ3 split robustness for the full model. Brackets denote 95% bootstrap confidence intervals from out-of-fold predictions. Grouped splits hold out all dimensions of the same summary, all summaries from the same source document, or all summaries from the same generation system, respectively.
Task
N
Pos.
AUROC
AUPRC
All dimensions
6,400
349
0.676
0.114
Relevance
1,600
102
0.703
0.206
Consistency
1,600
87
0.672
0.133
Fluency
1,600
79
0.743
0.138
Coherence
1,600
70
0.578
0.067
Appendix
Table 14: RQ3 quartile-threshold sensitivity for predicting human-easy/LLM-hard cases from all summary–dimension units.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.