The paradigm of LLM-as-a-judge relies on a critical assumption, namely that high inter-evaluator agreement indicates reliable and objective evaluation. We present two complementary findings that challenge this assumption. First, we demonstrate that this consensus is frequently illusory. We identify and formalize Evaluation Illusion, a phenomenon where LLM judges generate sophisticated critiques yet anchor scores on shared surface heuristics rather than substantive quality. Through a large-scale study of 105,600 evaluation instances (32 LLMs × 3 frontier judges × 100 tasks × 11 temperatures), we show that model-level agreement (Spearman ρ=0.99) masks fragile sample-level agreement (Pearson rˉ=0.72; absolute agreement ICC =0.67), that merely sharing rubric structure restores 62% of total agreement, and that high-quality outputs paradoxically receive the least consistent evaluations. Second, we demonstrate that dynamically generating evaluation rubrics grounded in domain knowledge produces more meaningful assessment. We introduce MERG (Metacognitive Enhanced Rubric Generation), a knowledge-driven rubric generation framework whose domain-selective effects confirm this. Agreement increases in codified domains (Education +22%, Academic +27%) where knowledge anchors evaluators on shared standards, while it decreases in subjective domains where genuine evaluative pluralism emerges. These findings suggest that evaluation rubrics should be dynamically enriched with expert knowledge rather than relying on generic criteria, with implications for reward modeling in RLAIF.
Figures & tables
Figure 1: Illustration of Evaluation Illusion. Without knowledge grounding, evaluators form a “Shared Illusion,” unanimously rewarding the professional formatting of a fundamentally flawed business pitch. MERG forces knowledge activation, revealing that Claude penalizes the regulatory violation while Gemini continues to reward surface heuristics. GPT (3.7, not shown) penalizes even more harshly. Full analysis with all three evaluators is in Appendix E.4 .
Method
Rubric
Knowledge
Grounding
Direct Score
None
Implicit
N/A
G-Eval
Static
Implicit
Template
Prometheus
Static
Implicit
User
WritingBench
Dynamic
Implicit
Query
MERG
Dynamic
Explicit
Metacognitive
Table 1: Comparison of evaluation paradigms. While standard methods rely on implicit knowledge and shared rubrics, MERG requires explicit metacognitive grounding.
Variant
Dimensions
Source
Original (4-Stage)
Independent
Each evaluator
5-Dim Per-Dim
Shared structure
Each evaluator
Shared Stages
Shared rubric
1 evaluator
Universal
Shared rubric
Averaged
Table 2: MERG ablation variants. Four controlled variants to disentangle the sources of evaluator agreement.
Figure 2: Knowledge injection systematically reduces evaluator agreement. Baseline vs. MERG agreement across 11 temperatures. The persistent gap ( ΔK<0 ) indicates that baseline agreement is heavily reliant on shared surface heuristics rather than substantive deliberation. The gap is largest at moderate temperatures ( t≈0.3 ) and narrows at the extremes as both conditions converge toward lower agreement.
Eval. Pair
Temp
Base
MERG
ΔK
Claude/Gemini
0.0
.698
.589
− .109
Claude/Gemini
0.3
.672
.399
− .273
Claude/GPT
0.0
.680
.518
− .162
Claude/GPT
0.3
.668
.463
− .205
Gemini/GPT
0.0
.720
.457
− .263
Gemini/GPT
0.3
.643
.430
− .213
Table 3: Knowledge injection systematically reduces inter-evaluator agreement. Baseline vs. MERG agreement (Pearson r ) across different evaluator pairs and temperatures. ΔK<0 in all 10/10 conditions ( p<0.001 ), indicating that baseline agreement is largely heuristic-driven rather than reflecting genuine evaluative convergence.
Rank
Model
sˉ
σ
1
DeepSeek-R1
7.77
.63
2
Qwen3-235B-Thinking
6.83
.42
3
Qwen3-32B-Thinking
6.72
.33
4
Qwen3-14B-Thinking
6.69
.32
5
Qwen3-30B-Thinking
6.62
.28
28
Qwen3-4B-Base
3.33
.07
Table 4: Quality rankings (Top-5 and Bottom-5). Higher-quality models (Thinking tier) are ranked at the top by mean score sˉ . σ denotes the standard deviation of per-evaluator grand means, reflecting cross-evaluator scoring divergence.
Rank
Model
rˉ
σ
1
Qwen3-30B-Base
.89
.03
2
Llama-4-Scout-Base
.88
.03
3
Qwen3-14B-Base
.87
.03
4
Qwen2.5-7B-Base
.87
.04
5
Qwen2.5-32B-Base
.86
.06
28
Qwen3-235B-Thinking
.67
.05
Table 5: Agreement rankings (Top-5 and Bottom-5). The strong negative correlation with quality ( ρ=−0.513 , p=0.003 ) suggests surface features suffice for judging low-quality outputs, but fail for high-quality ones. σ denotes the standard deviation across the three evaluator-pair Pearson r values.
Method
C/G
C/GPT
G/GPT
Avg
Δr
DeepSeek-R1
Original 4-Stage
0.155
0.237
0.281
0.224
N/A
5-Dim Per-Dim
0.526
0.514
0.588
0.543
+ 0.319
Shared Stages
N/A
N/A
0.606
0.606
+ 0.063
Universal ( t=0.0 )
0.678
0.668
0.420
0.589
− 0.017
Qwen3-235B
Table 6: Structural constraints drive the majority of evaluator agreement. Ablation of inter-evaluator agreement across MERG variants. When evaluators generate rubrics independently (Original), agreement is near-random. Standardizing dimensions (5-Dim) accounts for ∼ 62% of the total agreement.
Figure 3: The Rubric Commensurability Problem. Agreement across MERG ablation variants. Independent evaluation (Original) yields near-random agreement; standardizing rubric structure (5-Dim) drives the largest increase, indicating most reported agreement is structural rather than substantive.
Figure 4: Domain-selective knowledge effects. ΔK by domain at t=0.0 . Knowledge increases agreement in codified domains (Education, Academic) but decreases it in subjective ones (Literature), ruling out the noise hypothesis.
Model
Lang
Base
MERG
ΔK
DS-R1
EN
.616
.682
+ .07
DS-R1
ZH
.621
.423
− .20
Q3-235B
EN
.800
.608
− .19
Q3-235B
ZH
.543
.638
+ .10
Table 7: Knowledge activation quality is language-dependent. Language agreement ( t=0.0 ). The cross-over interaction ( p<0.05 ) suggests models ground better in their dominant training language.
WritingBench
MERG
Suspense
Atmospheric Dread Construction
Coherence
Psychological Ambiguity
Style
The Uncanny Familiar
Grammar
Restraint in Revelation
Table 8: MERG generates deep, task-specific dimensions. Rubric comparison: WritingBench vs. MERG for a Gothic horror writing task. While baseline rubrics are generic, MERG demands genuine domain expertise.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Rank
Model
Score
Agreement ( r )
1
DeepSeek-R1-0528-Thinking
7.77
.615
2
Qwen3-235B-Thinking
6.83
.673
3
Qwen3-32B-Thinking
6.72
.776
4
Qwen3-14B-Thinking
6.69
.794
5
Qwen3-30B-Thinking
6.62
.780
6
Qwen3-235B-Instruct
6.50
.764
Appendix
Table 9: Complete 32-model rankings at t=0.0 . Quality and agreement correlation: ρ=−0.513 , p=0.003 . Top-quality models (Thinking tier) consistently show the lowest evaluator agreement, confirming the Evaluation Illusion hypothesis.
Figure 5: Sample-level agreement is universally fragile. Pearson r across 32 LLMs at 11 temperatures ( t∈[0.0,1.0] ). Each subplot shows one model; lines represent evaluator pairs ( Claude/Gemini , Claude/GPT , Gemini/GPT ); background shading indicates model type ( Base , Instruct , Thinking ). The pattern is consistent: agreement is moderate ( r≈0.5 to 0.8 ) and largely temperature-invariant, confirming that the Resolution Paradox (high model-level ρ , low sample-level r ) is a universal property of LLM evaluation, not a model-specific artifact.
Figure 6: System validity: the three-tier quality hierarchy is robust. Score distributions for all 32 models across 3 evaluators at t=0.0 . Despite the disagreement at the sample level documented throughout this paper, evaluators unanimously recover the Base < Instruct < Thinking quality ordering (Cohen’s d=2.58 ). This confirms that Evaluation Illusion does not invalidate LLM evaluation for coarse-grained model comparison; it specifically affects the fine-grained, per-sample signals that RLAIF reward models require.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.