Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation

HyunJoon Jung; William Na

cs.AIApr 1, 2026

Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation

Authors: HyunJoon Jung, William Na

Abstract

LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score-coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law-both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity-Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties.

Figures & tables

Comparison	Mean $\|$ diff $\|$	$d$	Interpretation
Agent–Agent	0.143	—	Internally consistent
Human–Agent	0.188	$-0.18$	Within H–H range
Human–Human	0.201	—	Natural disagreement

Table 1: Pairwise score differences across three comparison groups. Agent–human differences fall within the range of human–human disagreement; a paired TOST supports equivalence at a margin of

0.05

(90% CI

[-0.038,0.012]

p=0.011

	Proprietary			Open-source
$k$	ICC	95% CI	Interp.	ICC	95% CI	Interp.
1	.290	[.00, .70]	Poor	.243	[.00, .67]	Poor
4	.621	[.16, .86]	Mod.	.562	[.07, .83]	Mod.
8	.766	[.42, .92]	Good	.719	[.33, .90]	Good
16	.868	[.64, .96]	Good	.837	[.57, .94]	Good
32	.929	[.80, .98]	Exc.	.911	[.75, .97]	Exc.

Table 2: ICC(2,

k

) with 95% CIs for both model pairs. Both follow a logarithmic curve (

R^{2}>0.97

Figure 2: The score–coverage dissociation. Left : Scoring reliability (ICC) improves logarithmically; error bars show 95% CIs. Right : Issue discovery follows a sublinear power law ( $b=0.69$ , $R^{2}=0.999$ ; shaded band: $\theta=0.60$ – $0.70$ ). Gray diamonds show raw observation counts before semantic deduplication; the 75% gap at $N\!=\!32$ illustrates the importance of deduplication for measuring true discovery. Both dimensions exhibit diminishing returns, but scores saturate $\sim\!2\times$ faster.

Factor	Variance ( $\omega^{2}$ )	Sig.	Interpretation
Task complexity	4–8%	∗	Harder tasks produce lower, more variable scores
Task domain	4%	∗	Different domains test different failure modes
Judge expertise	${<}$ 1%	n.s.	Controlled by structured persona conditioning

Table 3: What drives score variation? Task properties dominate; judge identity contributes

<

1%. Variance explained (

\omega^{2}

) averaged across both model pairs.

{}^{*}p<.001

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

#	Background	Exp.	O	C	E	A	N	Region
1	Accountant, 52	I	.40	.70	.35	.40	.49	US
2	IT Project Mgr, 45	E	.38	.72	.32	.42	.51	Asia
3	Retired Nurse, 62	N	.42	.68	.38	.58	.54	Latam
4	HR Specialist, 38	I	.40	.70	.41	.60	.54	EU
5	UX Designer, 29	I	.70	.68	.38	.58	.54	Asia
6	Journalist, 34	I	.72	.65	.35	.56	.55	EU

Appendix

Table 4: Complete pool of 32 agent judges. O/C/E/A/N = Big Five dimensions (0–1). Expertise: E xpert, I ntermediate, N ovice.

Task	$N_{H}$	H	PSA	$d$	$p_{\text{raw}}$	$p_{\text{Bonf}}$
dev-howto-medium	6	.67	.84	$-0.80$	.020	.300
dev-info-simple	3	.60	.81	$-1.49$	.068	1.00
dev-troubleshoot-complex	5	.83	.78	$+0.29$	.610	1.00
ecom-creative-complex	4	.73	.91	$-1.16$	.000	.005*
ecom-decision-medium	4	.86	.80	$+0.48$	.394	1.00
ecom-howto-simple	6	.82	.68	$+0.89$	.039	.585

Appendix

Table 5: Per-task comparison between human raters and agent judges.

p_{\text{raw}}

: uncorrected;

p_{\text{Bonf}}

: Bonferroni-corrected (

\times 15

). After correction, only 2 of 15 tasks remain significant.

Figure 4: Left : Sublinear discovery scaling with head/torso/tail zones annotated. Marginal contribution per judge decreases from 4.7 (N=1–4) to 2.0 (N=16–32), consistent with diminishing returns. The threshold-sensitivity band spans $\theta=0.60$ – $0.70$ . Right : Hypothesized power law distribution of the finding space. Critical findings (head) have high per-session discovery probability and are found by small panels; moderate findings (torso) require more diverse perspectives; corner cases (tail) are reached only by large panels. This distribution explains the sublinear exponent $b\approx 0.69$ .

Figure 5: Power law exponent $b$ as a function of cosine similarity threshold $\theta$ . Across all seven thresholds ( $\theta=0.50$ – $0.80$ ), the exponent remains below 1.0, confirming that the sublinear conclusion is not an artifact of a particular deduplication setting. The recommended range ( $\theta=0.60$ – $0.70$ , shaded) is chosen by cluster quality inspection: at $\theta=0.65$ , semantically equivalent insights from different agents are grouped together, at the cost of occasionally over-merging related but distinct issues. Because over-merging can only reduce unique-discovery counts, the reported discovery curve is a conservative lower bound and the sublinear conclusion is unaffected.

Figure 6: Score distributions by domain. Red histograms: agent judges (N=96 per domain). Blue lines: individual human raters. KS $p$ -values shown per domain; most domains show no significant distributional difference.

Figure 7: Effect sizes (Cohen’s $d$ ) between human raters (50 recruited, 43 retained, 86 sessions) and agent judges (N=480) by domain. Green: non-significant ( $p>0.05$ ). The overall effect is small ( $d=-0.18$ , $p=0.12$ ).

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
Mar 11, 2026Mingyang Song, Mao Zheng, Chenning XuLLM-as-a-Judge Rubric-Based Evaluation
Large Language Model Department, Tencent, China
GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
Jun 22, 2026Jeffrey FlyntLLM Agent Verification AI Agent Benchmarks
Independent Researcher
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Feb 1, 2026Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3LLM Evaluation AI Agent Evaluation
Computer Science and Information, Ben-Gurion University of the Negev

Mar 11, 2026J/K move · Enter open · S save

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

Mingyang Song, Mao Zheng, Chenning Xu

Large Language Model Department, Tencent, China

LLM-as-a-Judge Rubric-Based Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

The paradigm of LLM-as-a-judge relies on a critical assumption, namely that high inter-evaluator agreement indicates reliable and objective evaluation. We present two complementary findings that challenge this assumption. First, we demonstrate that this consensus is frequently illusory. We identify and formalize Evaluation Illusion, a phenomenon where LLM judges generate sophisticated critiques yet anchor scores on shared surface heuristics rather than substantive quality. Through a large-scale study of 105,600 evaluation instances (32 LLMs

\times

3 frontier judges

\times

100 tasks

\times

11 temperatures), we show that model-level agreement (Spearman

ρ= 0.99

) masks fragile sample-level agreement (Pearson

\bar{r} = 0.72

; absolute agreement ICC

= 0.67

), that merely sharing rubric structure restores 62% of total agreement, and that high-quality outputs paradoxically receive the least consistent evaluations. Second, we demonstrate that dynamically generating evaluation rubrics grounded in domain knowledge produces more meaningful assessment. We introduce MERG (Metacognitive Enhanced Rubric Generation), a knowledge-driven rubric generation framework whose domain-selective effects confirm this. Agreement increases in codified domains (Education +22%, Academic +27%) where knowledge anchors evaluators on shared standards, while it decreases in subjective domains where genuine evaluative pluralism emerges. These findings suggest that evaluation rubrics should be dynamically enriched with expert knowledge rather than relying on generic criteria, with implications for reward modeling in RLAIF.

Abstract

LLM-as-a-Judge

Explore similar work

Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment