Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

Mingyang Song; Mao Zheng; Chenning Xu

cs.CLMar 11, 2026

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

Authors: Mingyang Song, Mao Zheng, Chenning Xu

Organizations: Large Language Model Department, Tencent, China

Abstract

The paradigm of LLM-as-a-judge relies on a critical assumption, namely that high inter-evaluator agreement indicates reliable and objective evaluation. We present two complementary findings that challenge this assumption. First, we demonstrate that this consensus is frequently illusory. We identify and formalize Evaluation Illusion, a phenomenon where LLM judges generate sophisticated critiques yet anchor scores on shared surface heuristics rather than substantive quality. Through a large-scale study of 105,600 evaluation instances (32 LLMs $\times$ 3 frontier judges $\times$ 100 tasks $\times$ 11 temperatures), we show that model-level agreement (Spearman $ρ= 0.99$ ) masks fragile sample-level agreement (Pearson $\bar{r} = 0.72$ ; absolute agreement ICC $= 0.67$ ), that merely sharing rubric structure restores 62% of total agreement, and that high-quality outputs paradoxically receive the least consistent evaluations. Second, we demonstrate that dynamically generating evaluation rubrics grounded in domain knowledge produces more meaningful assessment. We introduce MERG (Metacognitive Enhanced Rubric Generation), a knowledge-driven rubric generation framework whose domain-selective effects confirm this. Agreement increases in codified domains (Education +22%, Academic +27%) where knowledge anchors evaluators on shared standards, while it decreases in subjective domains where genuine evaluative pluralism emerges. These findings suggest that evaluation rubrics should be dynamically enriched with expert knowledge rather than relying on generic criteria, with implications for reward modeling in RLAIF.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of Evaluation Illusion. Without knowledge grounding, evaluators form a “Shared Illusion,” unanimously rewarding the professional formatting of a fundamentally flawed business pitch. MERG forces knowledge activation, revealing that Claude penalizes the regulatory violation while Gemini continues to reward surface heuristics. GPT (3.7, not shown) penalizes even more harshly. Full analysis with all three evaluators is in Appendix E.4 .

Method	Rubric	Knowledge	Grounding
Direct Score	None	Implicit	N/A
G-Eval	Static	Implicit	Template
Prometheus	Static	Implicit	User
WritingBench	Dynamic	Implicit	Query
MERG	Dynamic	Explicit	Metacognitive

Table 1: Comparison of evaluation paradigms. While standard methods rely on implicit knowledge and shared rubrics, MERG requires explicit metacognitive grounding.

Variant	Dimensions	Source
Original (4-Stage)	Independent	Each evaluator
5-Dim Per-Dim	Shared structure	Each evaluator
Shared Stages	Shared rubric	1 evaluator
Universal	Shared rubric	Averaged

Table 2: MERG ablation variants. Four controlled variants to disentangle the sources of evaluator agreement.

Figure 2: Knowledge injection systematically reduces evaluator agreement. Baseline vs. MERG agreement across 11 temperatures. The persistent gap (

\Delta_{K}<0

) indicates that baseline agreement is heavily reliant on shared surface heuristics rather than substantive deliberation. The gap is largest at moderate temperatures (

t\approx 0.3

) and narrows at the extremes as both conditions converge toward lower agreement.

Eval. Pair	Temp	Base	MERG	$\boldsymbol{\Delta_{K}}$
Claude/Gemini	0.0	.698	.589	$-$ .109
Claude/Gemini	0.3	.672	.399	$-$ .273
Claude/GPT	0.0	.680	.518	$-$ .162
Claude/GPT	0.3	.668	.463	$-$ .205
Gemini/GPT	0.0	.720	.457	$-$ .263
Gemini/GPT	0.3	.643	.430	$-$ .213

Table 3: Knowledge injection systematically reduces inter-evaluator agreement. Baseline vs. MERG agreement (Pearson

r

) across different evaluator pairs and temperatures.

\Delta_{K}<0

in all 10/10 conditions (

p<0.001

), indicating that baseline agreement is largely heuristic-driven rather than reflecting genuine evaluative convergence.

Rank	Model	$\bar{s}$	$\sigma$
1	DeepSeek-R1	7.77	.63
2	Qwen3-235B-Thinking	6.83	.42
3	Qwen3-32B-Thinking	6.72	.33
4	Qwen3-14B-Thinking	6.69	.32
5	Qwen3-30B-Thinking	6.62	.28
28	Qwen3-4B-Base	3.33	.07

Table 4: Quality rankings (Top-5 and Bottom-5). Higher-quality models (Thinking tier) are ranked at the top by mean score

\bar{s}

\sigma

denotes the standard deviation of per-evaluator grand means, reflecting cross-evaluator scoring divergence.

Rank	Model	$\bar{r}$	$\sigma$
1	Qwen3-30B-Base	.89	.03
2	Llama-4-Scout-Base	.88	.03
3	Qwen3-14B-Base	.87	.03
4	Qwen2.5-7B-Base	.87	.04
5	Qwen2.5-32B-Base	.86	.06
28	Qwen3-235B-Thinking	.67	.05

Table 5: Agreement rankings (Top-5 and Bottom-5). The strong negative correlation with quality (

\rho=-0.513

p=0.003

) suggests surface features suffice for judging low-quality outputs, but fail for high-quality ones.

\sigma

denotes the standard deviation across the three evaluator-pair Pearson

r

values.

Method	C/G	C/GPT	G/GPT	Avg	$\Delta r$
DeepSeek-R1
Original 4-Stage	0.155	0.237	0.281	0.224	N/A
5-Dim Per-Dim	0.526	0.514	0.588	0.543	$+$ 0.319
Shared Stages	N/A	N/A	0.606	0.606	$+$ 0.063
Universal ( $t{=}0.0$ )	0.678	0.668	0.420	0.589	$-$ 0.017
Qwen3-235B

Table 6: Structural constraints drive the majority of evaluator agreement. Ablation of inter-evaluator agreement across MERG variants. When evaluators generate rubrics independently (Original), agreement is near-random. Standardizing dimensions (5-Dim) accounts for

\sim

62% of the total agreement.

Figure 3: The Rubric Commensurability Problem. Agreement across MERG ablation variants. Independent evaluation (Original) yields near-random agreement; standardizing rubric structure (5-Dim) drives the largest increase, indicating most reported agreement is structural rather than substantive.

Figure 4: Domain-selective knowledge effects.

\Delta_{K}

by domain at

t{=}0.0

. Knowledge increases agreement in codified domains (Education, Academic) but decreases it in subjective ones (Literature), ruling out the noise hypothesis.

Model	Lang	Base	MERG	$\Delta_{K}$
DS-R1	EN	.616	.682	$+$ .07
DS-R1	ZH	.621	.423	$-$ .20
Q3-235B	EN	.800	.608	$-$ .19
Q3-235B	ZH	.543	.638	$+$ .10

Table 7: Knowledge activation quality is language-dependent. Language agreement (

t{=}0.0

). The cross-over interaction (

p<0.05

) suggests models ground better in their dominant training language.

WritingBench	MERG
Suspense	Atmospheric Dread Construction
Coherence	Psychological Ambiguity
Style	The Uncanny Familiar
Grammar	Restraint in Revelation

Table 8: MERG generates deep, task-specific dimensions. Rubric comparison: WritingBench vs. MERG for a Gothic horror writing task. While baseline rubrics are generic, MERG demands genuine domain expertise.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Rank	Model	Score	Agreement ( $r$ )
1	DeepSeek-R1-0528-Thinking	7.77	.615
2	Qwen3-235B-Thinking	6.83	.673
3	Qwen3-32B-Thinking	6.72	.776
4	Qwen3-14B-Thinking	6.69	.794
5	Qwen3-30B-Thinking	6.62	.780
6	Qwen3-235B-Instruct	6.50	.764

Appendix

Table 9: Complete 32-model rankings at

t{=}0.0

. Quality and agreement correlation:

\rho=-0.513

p=0.003

. Top-quality models (Thinking tier) consistently show the lowest evaluator agreement, confirming the Evaluation Illusion hypothesis.

Figure 5: Sample-level agreement is universally fragile. Pearson

r

across 32 LLMs at 11 temperatures (

t\in[0.0,1.0]

). Each subplot shows one model; lines represent evaluator pairs ( Claude/Gemini , Claude/GPT , Gemini/GPT ); background shading indicates model type ( Base , Instruct , Thinking ). The pattern is consistent: agreement is moderate (

r\approx 0.5

0.8

) and largely temperature-invariant, confirming that the Resolution Paradox (high model-level

\rho

, low sample-level

r

) is a universal property of LLM evaluation, not a model-specific artifact.

Figure 6: System validity: the three-tier quality hierarchy is robust. Score distributions for all 32 models across 3 evaluators at

t{=}0.0

. Despite the disagreement at the sample level documented throughout this paper, evaluators unanimously recover the Base

<

Instruct

<

Thinking quality ordering (Cohen’s

d=2.58

). This confirms that Evaluation Illusion does not invalidate LLM evaluation for coarse-grained model comparison; it specifically affects the fine-grained, per-sample signals that RLAIF reward models require.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 553 papers

Aug 31, 2026

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

LLM-as-a-Judge pipelines are increasingly used to evaluate AI-generated text, based on the assumption that judgments arise from reasoning over candidate responses with respect to a rubric. We show that this assumption warrants further scrutiny. Classifiers trained only on rubric text, without access to any evaluated response, achieve nontrivial predictive performance on judge outputs. This suggests that rubric formulations encode recoverable evaluative signals, allowing scores to be partially anticipated independently of model outputs. Finally, counterfactual perturbations reveal that judges often fail to reliably update their decisions when either the candidate response or the rubric criterion is reversed. Our findings raise concerns about the reliability of rubric-based LLM evaluation and highlight the need for further methodological study of automated evaluation via LLMs.

Anshul Bagaria, Sowmya S Sundaram, Gokul S Krishnan +1

Centre for Responsible AI (CeRAI), Wadhwani School of Data Science and AI (WSAI), IIT Madras, Chennai, India

LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based Evaluation Bias in LLM-as-a-Judge

Ask PDF

May 25, 2026

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Validating an LLM judge against human annotations usually means reporting several agreement statistics: accuracy, precision, recall,

F_1

, Cohen's

κ

, and one or more rank correlations. A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling, and those choices rarely stated. For binary criteria -- the common case in rubric-based evaluation, where each criterion is graded MET or UNMET -- most of the reported numbers are redundant: Pearson's

r

, Spearman's

ρ

, Kendall's

τ_b

, the phi coefficient

φ

, and the Matthews Correlation Coefficient all reduce to a single number on non-degenerate binary data, so reporting several of them only creates an illusion of corroborating evidence. Cohen's

κ

is the one agreement coefficient that adds information: it shares

φ

's numerator but normalizes differently, and the gap between them measures how far the judge's positive-label rate has drifted from the human's. We then trace what changes when a judge may abstain with a CANNOT_ASSESS verdict: the three common ways of handling abstentions are not interchangeable preprocessing choices but answer different questions, and they break the binary equivalences. The same equivalences reappear, up to a negligible finite-sample correction, for multi-judge ensembles scored with Fleiss'

κ

or Krippendorff's

α

. We close with a reporting checklist that names the judgment scale, the abstention and tie handling mode, coverage, the confusion matrix, and the aggregation level alongside any scalar agreement coefficient.

Delip Rao, Chris Callison-Burch

University of Pennsylvania

Inter-Rater Reliability LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation

Ask PDF

Jun 2, 2026

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment

LLM judges now score most open-ended NLP output, and their mutual agreement is routinely read as evidence that the scores can be trusted. That reading is unsafe: judges may agree because they capture quality, or because they share the same blind spots, and agreement statistics alone cannot tell these apart. We develop a geometric test that can. Treating each judge's scores as a vector, we measure spread, effective rank, the angle to human scores, and the judge-judge, judge-human, and human-human agreement triple for 42 judges on two community-built Indic benchmarks covering four domains and eight languages. Every comparison is reference-matched: a judge and a held-out rater are scored against the same two-rater mean, since an averaged reference otherwise flatters judges by several degrees. On subjective rubrics, judges agree with one another as much as humans do yet reach only 58-66% of human agreement and often concentrate on an axis humans do not weight. On the one rubric with a verifiable answer, most of that gap closes. Ensembles converge on the judges' shared axis rather than the human one, and training widens scores without rotating them. Inter-judge agreement is evidence of human alignment only after this check passes.

Sourabrata Mukherjee, Hamna Hamna, Kalika Bali +1

Microsoft Research

Inter-Rater Reliability Multilingual Language Model Evaluation LLM Evaluation LLM-as-a-Judge LLM Reliability

Ask PDF

Abstract

LLM-as-a-Judge

Explore similar work

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

The Geometry of LLM-as-Judge: Why Inter-LLM Consensus Is Not Human Alignment