How Many Humans Are 32 LLM Judges Worth?

Chao Li; Yingying Yu; Yunfeng Li

cs.CLSep 18, 2026

How Many Humans Are 32 LLM Judges Worth?

Authors: Chao Li, Yingying Yu, Yunfeng Li

Organizations: Tsinghua University · University College London · Hebei University of Economics and Business

Abstract

A panel's human-equivalent size is target-specific. Matching a fixed 32-judge panel to empirical human label distributions on three ChaosNLI tasks yields two distinct effective sizes: distributional-error matching gives $ν_{\mathrm{MSE}}=2.304$ , $3.750$ , and $3.445$ , whereas spectral matching gives $ν_H=4.242$ , $6.459$ , and $6.499$ , a gap of $1.72$ -- $1.89\times$ ; a binary-error diagnostic credits the same panels with only $1.971$ -- $2.227$ effective votes. Extrapolating the distributional-error curve at fixed squared mean residual, mean member variance, and normalized mean covariance gives asymptotes of $2.392$ , $3.990$ , and $3.655$ , with 32 judges already reaching $94.0$ -- $96.3\%$ . An exact spectral identity explains the gap: error depends on member energy and on the orientation of residual variation relative to averaging, information that the participation ratio (PR) discards. A realizable hard-label construction confirms that higher spectral diversity can coexist with worse distribution recovery even under equal member energies and nonnegative correlations, and the consensus direction retains $γ_{\mathrm{co}}=43.8\%$ , $33.7\%$ , and $35.9\%$ of centered residual variance. An external check on CC-1000, a 1,000-item Civil Comments subset with a different panel, gives $ν_H=2.84$ . For panel choice, we establish an existence result and one feasible path: exhaustive enumeration at $k\in\{5,7\}$ shows that panels beating the accuracy-top- $k$ baseline on both accuracy and $ν_H$ always exist, and greedily swapping at most two members reaches $24.8$ -- $56.0\%$ higher $ν_H$ at $0.10$ -- $1.10$ percentage points higher accuracy. Our dataset and code are available at https://github.com/Chao1208/32judges-votes.

Figures & tables

Quantity	Measurement target	Answers the question
$\nu_{H}$	PR of the normalized residual Gram matrix, matched to independent draws from $h$ .	How many independent draws have comparable spectral diversity?
$\nu_{\mathrm{MSE}}$	Squared error of the panel label distribution relative to $h$ , matched to the same draws.	How many independent draws have comparable distributional error?
$\gamma_{\mathrm{co}}$	Share of centered residual variance along the equal-weight judge direction.	How much centered variation survives equal-weight averaging?
$n_{\mathrm{eff}}$	Signed-correlation summary of binary errors relative to the supplied gold label.	How many effective votes does the binary-error diagnostic report?

Table 1: Measurement targets in the panel study.

Component	Scope and treatment
Baseline	$32\times 1{,}000\times 3=96{,}000$ records; 95,994 parseable. The 6 failures (0.00625%) retain flagged placeholders: 1/0/5 by task.
Human reference	100 labels per item; $h_{i}$ is their empirical frequency vector.
Panel audit	54,139 unique member sets per task; 150,372 fixed member additions. Shared judges and items make these overlapping comparisons.
Item stability	Fixed 500/500 halves; compares stability within observed data. Failed-item sensitivity is recorded separately.
Presentation	122,000 records, including cache reuse; its own original-response snapshot. Complete-case sets contain 489/492/492 items and 31/31/29 judges.
CC-1000 check	1,000 items, 32 judges, and 176,251 human toxicity votes; score-only public release.

Table 2: Collection and analysis design.

Measurement	MNLI-m	SNLI	$\alpha$ NLI
$n_{\mathrm{eff}}$	1.97	2.23	2.00
$\nu_{H}$	4.24	6.46	6.50
$\nu_{\mathrm{MSE}}$	2.30	3.75	3.44
$\nu_{H}/\nu_{\mathrm{MSE}}$	1.84	1.72	1.89
$\mathcal{E}$	0.19691	0.09068	0.04825
$\gamma_{\mathrm{co,all}}$	43.8%	33.7%	35.9%

Table 3: Measurement values for the same 32-judge panels on ChaosNLI.

Mean squared inner product $\bar{q}$
Anchor $A$	MNLI-m	SNLI	$\alpha$ NLI
Uniform $1/L$	0.412	0.531	0.676
Mode $A_{\mathrm{mode}}$	0.425	0.284	0.287
Distribution $h$	0.212	0.128	0.128
Effective size $\nu_{H}$
Uniform $1/L$	2.75	2.57	2.33

Table 4: Effective sizes under three residual anchors.

Figure 1: Within-size Spearman correlations between spectral and distribution-recovery rankings.

Addition	Examined	MNLI-m	SNLI	$\alpha$ NLI
$4\to 5$	54,348	2,116	309	1,127
$8\to 9$	48,000	332	441	1,085
$16\to 17$	32,000	0	7	447
$24\to 25$	15,992	0	0	85
$31\to 32$	32	0	0	0

Table 5: Stable conflicting member additions by panel size.

Figure 2: Residual geometry relative to human sampling.

Figure 3: Projection of consensus and orthogonal directions in a 10-model panel.

Figure 4: Expected variance of a random-subpanel mean.

Figure 5: Random-subpanel

\nu_{H}

by panel size.

Figure 6: Observed presentation changes and an independent-transition reference.

Measurement	CC-1000
$\nu_{H}$	2.838
$\nu_{\mathrm{MSE}}$ (empirical)	1.490
$\nu_{\mathrm{MSE}}$ (finite annotation)	1.549
$n_{\mathrm{eff}}$	1.677
$\gamma_{\mathrm{co}}$	57.1%

Table 6: Measurements for the external CC-1000 check.

Figure 7: Enumeration and candidate panels in the $(\mathrm{acc},\nu_{H})$ plane at $k=7$ .

Dataset	$k$	acc	$\nu_{H}$	$\Delta$ acc	$\Delta\nu_{H}$	$\Delta$ acc	$\Delta\nu_{H}$
		Baseline top- $k$		Rule A		Rule D
MNLI-m	5	0.7325	2.732	$+0.80$	$+27.6$	$+1.00$	$+10.8$
MNLI-m	7	0.7215	3.085	$+0.20$	$+24.8$	$+2.35$	$+18.7$
SNLI	5	0.8685	2.495	$+0.25$	$+56.0$	$+1.50$	$+40.6$
SNLI	7	0.8590	2.759	$+1.10$	$+52.3$	$+2.70$	$+34.5$
$\alpha$ NLI	5	0.9380	2.651	$+0.20$	$+43.9$	$+0.80$	$+23.8$

Table 7: Accuracy and

\nu_{H}

changes for panels selected by rules A and D.

Dataset	$k$	$\mathcal{E}$	$\nu_{\mathrm{MSE}}$	A	D	E	$\Delta\bar{\omega}_{A}$
		Baseline top- $k$		$\Delta\nu_{\mathrm{MSE}}$ (%)
MNLI-m	5	0.2135	2.125	$+19.9$	$+7.5$	$+19.9$	$+5.7$
MNLI-m	7	0.2066	2.196	$+13.4$	$+14.6$	$+15.4$	$+4.4$
SNLI	5	0.1367	2.487	$+31.0$	$+29.6$	$+31.2$	$+26.6$
SNLI	7	0.1317	2.581	$+33.2$	$+22.9$	$+34.4$	$+18.9$
$\alpha$ NLI	5	0.0600	2.768	$+11.3$	$+10.6$	$+15.9$	$+32.2$

Table 8: Distributional recovery for panels selected by rules A, D, and E.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Human-reference label count

M

and the resulting

\nu_{H}

Dataset	$k$	Baseline and selected panels (parenthesized values are single-judge accuracy ranks)
MNLI-m	5	$S_{0}$ : KimiK3, KimiK2.5, o4-mini, KimiK2.6, Qwen3.7-plus; A : $-$ KimiK2.5(2), $-$ Qwen3.7-plus(5), $+$ GPT5.6-terra(18), $+$ Claude-haiku4.5(24); D : $-$ o4-mini(3), $-$ Qwen3.7-plus(5), $+$ Claude-opus5(13), $+$ GPT5.6-terra(18); E : same as A.
MNLI-m	7	$S_{0}$ : KimiK3, KimiK2.5, o4-mini, KimiK2.6, Qwen3.7-plus, Qwen3-max, GLM5.3; A : $-$ KimiK2.5(2), $-$ Qwen3.7-plus(5), $+$ DeepSeekV3.2(15), $+$ Claude-haiku4.5(24); D : $-$ Qwen3-max(6), $-$ GLM5.3(7), $+$ GPT5.6-terra(18), $+$ Claude-haiku4.5(24); E : $-$ KimiK2.5(2), $-$ Qwen3-max(6), $+$ DeepSeekV3.2(15), $+$ Claude-haiku4.5(24).
SNLI	5	$S_{0}$ : Gemini3.1-pro, Claude-opus5, Qwen3.7-plus, Qwen3.8-max, GLM5.1; A : $-$ Gemini3.1-pro(1), $-$ Qwen3.8-max(4), $+$ Claude-haiku4.5(30), $+$ DeepSeekV3.2(32); D : $-$ Claude-opus5(2), $-$ Qwen3.8-max(4), $+$ GPT5.4(21), $+$ DeepSeekV3.2(32); E : $-$ Qwen3.7-plus(3), $-$ Qwen3.8-max(4), $+$ KimiK2.6(23), $+$ GPT5.6-terra(29).
SNLI	7	$S_{0}$ : Gemini3.1-pro, Claude-opus5, Qwen3.7-plus, Qwen3.8-max, GLM5.1, Gemini3.6-flash, Qwen3.5-plus; A : $-$ Gemini3.1-pro(1), $-$ Qwen3.8-max(4), $+$ Claude-haiku4.5(30), $+$ DeepSeekV3.2(32); D : $-$ Qwen3.8-max(4), $-$ Gemini3.6-flash(6), $+$ GLM5.2(11), $+$ GPT5.6-terra(29); E : $-$ Qwen3.8-max(4), $-$ Gemini3.6-flash(6), $+$ Claude-haiku4.5(30), $+$ DeepSeekV3.2(32).
$\alpha$ NLI	5	$S_{0}$ : Grok4.6, Grok4.5, Qwen3.8-max, Claude-opus5, Gemini3.1-pro; A : $-$ Grok4.5(2), $-$ Gemini3.1-pro(5), $+$ GPT5.6-terra(24), $+$ DeepSeekV3.2(32); D : $-$ Grok4.5(2), $-$ Qwen3.8-max(3), $+$ DeepSeekV4-flash(12), $+$ MiniMaxM3(21); E : $-$ Grok4.5(2), $-$ Qwen3.8-max(3), $+$ Gemini2.5-pro(18), $+$ o4-mini(25).
$\alpha$ NLI	7	$S_{0}$ : Grok4.6, Grok4.5, Qwen3.8-max, Claude-opus5, Gemini3.1-pro, Gemini3.6-flash, Gemini3.7-flash; A : $-$ Grok4.5(2), $-$ Gemini3.6-flash(6), $+$ GPT5.6-terra(24), $+$ DeepSeekV3.2(32); D : $-$ Gemini3.6-flash(6), $-$ Gemini3.7-flash(7), $+$ GPT5.6-terra(24), $+$ DeepSeekV3.2(32); E : $-$ Gemini3.6-flash(6), $-$ Gemini3.7-flash(7), $+$ o4-mini(25), $+$ DeepSeekV3.2(32).

Appendix

Table A2: Baseline and selected panels for the eight dataset–size cases.

Binary-error Pearson matrix $\Phi$
Dataset	Signed summary	PR
MNLI-m	1.971	3.536
SNLI	2.227	4.383
$\alpha$ NLI	1.999	3.792
Human-residual Gram matrix $C$
MNLI-m	2.201	4.228

Appendix

Table C1: Dependence summaries under binary-error and human-residual representations.

Figure D1: (a) Individual agreement with human gold. (b) Observed error counts and an unconditional-independence reference.

Figure D2: Subpanel search in the fixed model pool. (a) Best candidates by size. (b) Half-split search results.

Dataset	Random	Diverse	Difference
MNLI-m	3.384	3.382	-0.003
SNLI	4.509	4.615	+0.105
$\alpha$ NLI	4.468	4.577	+0.108

Appendix

Table D1: Median

\nu_{H}

for random and family-diverse 9-judge panels.

Figure D3: Within-size rank correlations between fixed item halves for PR and distributional error

\mathcal{E}

Dataset	Items	$\nu_{H}$	$\nu_{\mathrm{MSE}}$	Ratio	Share
MNLI-m	999	4.220	2.269	1.86	96.4%
	1,596	4.242	2.278	1.86	96.4%
SNLI	1,000	6.650	3.739	1.78	94.0%
	1,513	6.737	3.804	1.77	93.9%
$\alpha$ NLI	995	6.283	3.335	1.88	94.5%
	1,525	6.287	3.149	2.00	94.3%

Appendix

Table D2: Effective sizes for the replacement panel on retained samples and full item sets.

Figure D4: Enumeration and candidate panels in the $(\mathrm{acc},\nu_{H})$ plane at $k=5$ .

Explore similar work

May 28, 2026cs.CL

Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes' worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation algorithms helps -- established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across prompt variants, temperatures, chain-of-thought reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.

Guneet Kohli

Apple

Llm-As-A-Judge Judges Large Language Model Evaluation Majority Voting Textual Dataset

Ask PDF

May 31, 2026cs.CL

A Finite-Calibration Regime Map for LLM Judge Panels

We study when LLM judge panels should be calibrated with low-dimensional stackers versus joint output tables under finite human-label budgets. Low-dimensional stackers have small estimation cost but miss interactions, whereas joint-table calibrators can represent interactions but pay for cell counts and unseen patterns. We cast this tradeoff as a finite-calibration regime map and instantiate it as Finite-Calibration Panel Selection, a deployable validation selector over judge path, prefix size, and aggregator family with table and parametric estimation diagnostics. On RewardBench, LLMBar, SummEval, and Arena100K with a seven-judge pool including DeepSeek V4 Flash, scalar/reliability aggregation wins 16 of 20 real dataset--budget cells, indicating that current judge outputs are often additive or redundant. Controlled calibration-growth data show the complementary regime: additive labels remain scalar-favored, whereas a six-way interaction selects a larger joint table and its test MSE drops from 0.224 to 0.061 once unseen mass vanishes. Thus the practical question is not ``how many judges?'' but whether the next judge's information is estimable under the available human labels.

Bin Zhu, Yanghui Rao

School of Computer Science and Engineering Sun Yat-sen University

Judges Large Language Model Judges Token Budget Allocation

Ask PDF

Aug 7, 2026cs.AI

Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

LLM judge panels are a standard evaluation tool, but prior work reports highly correlated panel errors: nine judges provide roughly the effective information of two independent ones, and aggregation closes only a small fraction of the gap. A natural remedy--a signal from a different evidence source, e.g., executing a test suite--produced no distinguishable change in the panel's effective-vote count at scale (-0.04, 95% CI [-0.10, +0.02]). Aggregate dependence and conditional decision utility are different questions. Elementary majority arithmetic fixes the affected set for single-ballot substitution: only decisions with a one-vote margin can change. The empirical question is whether panel error rates rise and useful substitutions concentrate there. They do: the entire accuracy gain concentrates on these pivotal queries, where it is large (+10.4 to +23.3 percentage points across three headline configurations), and is exactly zero elsewhere. We confirm the pattern across three code benchmarks and four panel sizes (a 9-judge extension and 56 dependent subsampling checks, gain +6.5 to +16.1 percentage points). On HumanEval+/MBPP+, a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the signal on 16.2% of queries; signal-only remains stronger at 87.60%. Thus population-level dependence diagnostics and margin-stratified utility are complementary, and the affected-set characterization yields a call-reduction rule for any specified single-ballot substitution policy.

Yang Shu

Zhejiang University, Hangzhou, China

Majority Voting Large Language Model Judges Equal Error Rate

Ask PDF