How Many Humans Are 32 LLM Judges Worth?

Chao Li; Yingying Yu; Yunfeng Li

cs.CLSep 18, 2026

How Many Humans Are 32 LLM Judges Worth?

Authors: Chao Li, Yingying Yu, Yunfeng Li

Organizations: Tsinghua University · University College London · Hebei University of Economics and Business

Abstract

A panel's human-equivalent size is target-specific. Matching a fixed 32-judge panel to empirical human label distributions on three ChaosNLI tasks yields two distinct effective sizes: distributional-error matching gives $ν_{\mathrm{MSE}}=2.304$ , $3.750$ , and $3.445$ , whereas spectral matching gives $ν_H=4.242$ , $6.459$ , and $6.499$ , a gap of $1.72$ -- $1.89\times$ ; a binary-error diagnostic credits the same panels with only $1.971$ -- $2.227$ effective votes. Extrapolating the distributional-error curve at fixed squared mean residual, mean member variance, and normalized mean covariance gives asymptotes of $2.392$ , $3.990$ , and $3.655$ , with 32 judges already reaching $94.0$ -- $96.3\%$ . An exact spectral identity explains the gap: error depends on member energy and on the orientation of residual variation relative to averaging, information that the participation ratio (PR) discards. A realizable hard-label construction confirms that higher spectral diversity can coexist with worse distribution recovery even under equal member energies and nonnegative correlations, and the consensus direction retains $γ_{\mathrm{co}}=43.8\%$ , $33.7\%$ , and $35.9\%$ of centered residual variance. An external check on CC-1000, a 1,000-item Civil Comments subset with a different panel, gives $ν_H=2.84$ . For panel choice, we establish an existence result and one feasible path: exhaustive enumeration at $k\in\{5,7\}$ shows that panels beating the accuracy-top- $k$ baseline on both accuracy and $ν_H$ always exist, and greedily swapping at most two members reaches $24.8$ -- $56.0\%$ higher $ν_H$ at $0.10$ -- $1.10$ percentage points higher accuracy. Our dataset and code are available at https://github.com/Chao1208/32judges-votes.

Figures & tables

Quantity	Measurement target	Answers the question
$\nu_{H}$	PR of the normalized residual Gram matrix, matched to independent draws from $h$ .	How many independent draws have comparable spectral diversity?
$\nu_{\mathrm{MSE}}$	Squared error of the panel label distribution relative to $h$ , matched to the same draws.	How many independent draws have comparable distributional error?
$\gamma_{\mathrm{co}}$	Share of centered residual variance along the equal-weight judge direction.	How much centered variation survives equal-weight averaging?
$n_{\mathrm{eff}}$	Signed-correlation summary of binary errors relative to the supplied gold label.	How many effective votes does the binary-error diagnostic report?

Table 1: Measurement targets in the panel study.

Component	Scope and treatment
Baseline	$32\times 1{,}000\times 3=96{,}000$ records; 95,994 parseable. The 6 failures (0.00625%) retain flagged placeholders: 1/0/5 by task.
Human reference	100 labels per item; $h_{i}$ is their empirical frequency vector.
Panel audit	54,139 unique member sets per task; 150,372 fixed member additions. Shared judges and items make these overlapping comparisons.
Item stability	Fixed 500/500 halves; compares stability within observed data. Failed-item sensitivity is recorded separately.
Presentation	122,000 records, including cache reuse; its own original-response snapshot. Complete-case sets contain 489/492/492 items and 31/31/29 judges.
CC-1000 check	1,000 items, 32 judges, and 176,251 human toxicity votes; score-only public release.

Table 2: Collection and analysis design.

Measurement	MNLI-m	SNLI	$\alpha$ NLI
$n_{\mathrm{eff}}$	1.97	2.23	2.00
$\nu_{H}$	4.24	6.46	6.50
$\nu_{\mathrm{MSE}}$	2.30	3.75	3.44
$\nu_{H}/\nu_{\mathrm{MSE}}$	1.84	1.72	1.89
$\mathcal{E}$	0.19691	0.09068	0.04825
$\gamma_{\mathrm{co,all}}$	43.8%	33.7%	35.9%

Table 3: Measurement values for the same 32-judge panels on ChaosNLI.

Mean squared inner product $\bar{q}$
Anchor $A$	MNLI-m	SNLI	$\alpha$ NLI
Uniform $1/L$	0.412	0.531	0.676
Mode $A_{\mathrm{mode}}$	0.425	0.284	0.287
Distribution $h$	0.212	0.128	0.128
Effective size $\nu_{H}$
Uniform $1/L$	2.75	2.57	2.33

Table 4: Effective sizes under three residual anchors.

Figure 1: Within-size Spearman correlations between spectral and distribution-recovery rankings.

Addition	Examined	MNLI-m	SNLI	$\alpha$ NLI
$4\to 5$	54,348	2,116	309	1,127
$8\to 9$	48,000	332	441	1,085
$16\to 17$	32,000	0	7	447
$24\to 25$	15,992	0	0	85
$31\to 32$	32	0	0	0

Table 5: Stable conflicting member additions by panel size.

Figure 2: Residual geometry relative to human sampling.

Figure 3: Projection of consensus and orthogonal directions in a 10-model panel.

Figure 4: Expected variance of a random-subpanel mean.

Figure 5: Random-subpanel

\nu_{H}

by panel size.

Figure 6: Observed presentation changes and an independent-transition reference.

Measurement	CC-1000
$\nu_{H}$	2.838
$\nu_{\mathrm{MSE}}$ (empirical)	1.490
$\nu_{\mathrm{MSE}}$ (finite annotation)	1.549
$n_{\mathrm{eff}}$	1.677
$\gamma_{\mathrm{co}}$	57.1%

Table 6: Measurements for the external CC-1000 check.

Figure 7: Enumeration and candidate panels in the $(\mathrm{acc},\nu_{H})$ plane at $k=7$ .

Dataset	$k$	acc	$\nu_{H}$	$\Delta$ acc	$\Delta\nu_{H}$	$\Delta$ acc	$\Delta\nu_{H}$
		Baseline top- $k$		Rule A		Rule D
MNLI-m	5	0.7325	2.732	$+0.80$	$+27.6$	$+1.00$	$+10.8$
MNLI-m	7	0.7215	3.085	$+0.20$	$+24.8$	$+2.35$	$+18.7$
SNLI	5	0.8685	2.495	$+0.25$	$+56.0$	$+1.50$	$+40.6$
SNLI	7	0.8590	2.759	$+1.10$	$+52.3$	$+2.70$	$+34.5$
$\alpha$ NLI	5	0.9380	2.651	$+0.20$	$+43.9$	$+0.80$	$+23.8$

Table 7: Accuracy and

\nu_{H}

changes for panels selected by rules A and D.

Dataset	$k$	$\mathcal{E}$	$\nu_{\mathrm{MSE}}$	A	D	E	$\Delta\bar{\omega}_{A}$
		Baseline top- $k$		$\Delta\nu_{\mathrm{MSE}}$ (%)
MNLI-m	5	0.2135	2.125	$+19.9$	$+7.5$	$+19.9$	$+5.7$
MNLI-m	7	0.2066	2.196	$+13.4$	$+14.6$	$+15.4$	$+4.4$
SNLI	5	0.1367	2.487	$+31.0$	$+29.6$	$+31.2$	$+26.6$
SNLI	7	0.1317	2.581	$+33.2$	$+22.9$	$+34.4$	$+18.9$
$\alpha$ NLI	5	0.0600	2.768	$+11.3$	$+10.6$	$+15.9$	$+32.2$

Table 8: Distributional recovery for panels selected by rules A, D, and E.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Human-reference label count

M

and the resulting

\nu_{H}

Dataset	$k$	Baseline and selected panels (parenthesized values are single-judge accuracy ranks)
MNLI-m	5	$S_{0}$ : KimiK3, KimiK2.5, o4-mini, KimiK2.6, Qwen3.7-plus; A : $-$ KimiK2.5(2), $-$ Qwen3.7-plus(5), $+$ GPT5.6-terra(18), $+$ Claude-haiku4.5(24); D : $-$ o4-mini(3), $-$ Qwen3.7-plus(5), $+$ Claude-opus5(13), $+$ GPT5.6-terra(18); E : same as A.
MNLI-m	7	$S_{0}$ : KimiK3, KimiK2.5, o4-mini, KimiK2.6, Qwen3.7-plus, Qwen3-max, GLM5.3; A : $-$ KimiK2.5(2), $-$ Qwen3.7-plus(5), $+$ DeepSeekV3.2(15), $+$ Claude-haiku4.5(24); D : $-$ Qwen3-max(6), $-$ GLM5.3(7), $+$ GPT5.6-terra(18), $+$ Claude-haiku4.5(24); E : $-$ KimiK2.5(2), $-$ Qwen3-max(6), $+$ DeepSeekV3.2(15), $+$ Claude-haiku4.5(24).
SNLI	5	$S_{0}$ : Gemini3.1-pro, Claude-opus5, Qwen3.7-plus, Qwen3.8-max, GLM5.1; A : $-$ Gemini3.1-pro(1), $-$ Qwen3.8-max(4), $+$ Claude-haiku4.5(30), $+$ DeepSeekV3.2(32); D : $-$ Claude-opus5(2), $-$ Qwen3.8-max(4), $+$ GPT5.4(21), $+$ DeepSeekV3.2(32); E : $-$ Qwen3.7-plus(3), $-$ Qwen3.8-max(4), $+$ KimiK2.6(23), $+$ GPT5.6-terra(29).
SNLI	7	$S_{0}$ : Gemini3.1-pro, Claude-opus5, Qwen3.7-plus, Qwen3.8-max, GLM5.1, Gemini3.6-flash, Qwen3.5-plus; A : $-$ Gemini3.1-pro(1), $-$ Qwen3.8-max(4), $+$ Claude-haiku4.5(30), $+$ DeepSeekV3.2(32); D : $-$ Qwen3.8-max(4), $-$ Gemini3.6-flash(6), $+$ GLM5.2(11), $+$ GPT5.6-terra(29); E : $-$ Qwen3.8-max(4), $-$ Gemini3.6-flash(6), $+$ Claude-haiku4.5(30), $+$ DeepSeekV3.2(32).
$\alpha$ NLI	5	$S_{0}$ : Grok4.6, Grok4.5, Qwen3.8-max, Claude-opus5, Gemini3.1-pro; A : $-$ Grok4.5(2), $-$ Gemini3.1-pro(5), $+$ GPT5.6-terra(24), $+$ DeepSeekV3.2(32); D : $-$ Grok4.5(2), $-$ Qwen3.8-max(3), $+$ DeepSeekV4-flash(12), $+$ MiniMaxM3(21); E : $-$ Grok4.5(2), $-$ Qwen3.8-max(3), $+$ Gemini2.5-pro(18), $+$ o4-mini(25).
$\alpha$ NLI	7	$S_{0}$ : Grok4.6, Grok4.5, Qwen3.8-max, Claude-opus5, Gemini3.1-pro, Gemini3.6-flash, Gemini3.7-flash; A : $-$ Grok4.5(2), $-$ Gemini3.6-flash(6), $+$ GPT5.6-terra(24), $+$ DeepSeekV3.2(32); D : $-$ Gemini3.6-flash(6), $-$ Gemini3.7-flash(7), $+$ GPT5.6-terra(24), $+$ DeepSeekV3.2(32); E : $-$ Gemini3.6-flash(6), $-$ Gemini3.7-flash(7), $+$ o4-mini(25), $+$ DeepSeekV3.2(32).

Appendix

Table A2: Baseline and selected panels for the eight dataset–size cases.

Binary-error Pearson matrix $\Phi$
Dataset	Signed summary	PR
MNLI-m	1.971	3.536
SNLI	2.227	4.383
$\alpha$ NLI	1.999	3.792
Human-residual Gram matrix $C$
MNLI-m	2.201	4.228

Appendix

Table C1: Dependence summaries under binary-error and human-residual representations.

Figure D1: (a) Individual agreement with human gold. (b) Observed error counts and an unconditional-independence reference.

Figure D2: Subpanel search in the fixed model pool. (a) Best candidates by size. (b) Half-split search results.

Dataset	Random	Diverse	Difference
MNLI-m	3.384	3.382	-0.003
SNLI	4.509	4.615	+0.105
$\alpha$ NLI	4.468	4.577	+0.108

Appendix

Table D1: Median

\nu_{H}

for random and family-diverse 9-judge panels.

Figure D3: Within-size rank correlations between fixed item halves for PR and distributional error

\mathcal{E}

Dataset	Items	$\nu_{H}$	$\nu_{\mathrm{MSE}}$	Ratio	Share
MNLI-m	999	4.220	2.269	1.86	96.4%
	1,596	4.242	2.278	1.86	96.4%
SNLI	1,000	6.650	3.739	1.78	94.0%
	1,513	6.737	3.804	1.77	93.9%
$\alpha$ NLI	995	6.283	3.335	1.88	94.5%
	1,525	6.287	3.149	2.00	94.3%

Appendix

Table D2: Effective sizes for the replacement panel on retained samples and full item sets.

Figure D4: Enumeration and candidate panels in the $(\mathrm{acc},\nu_{H})$ plane at $k=5$ .

Explore similar work

Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
May 28, 2026Guneet KohliLlm-As-A-Judge Judges
Apple
A Finite-Calibration Regime Map for LLM Judge Panels
May 31, 2026Bin Zhu, Yanghui RaoJudges Large Language Model Judges
School of Computer Science and Engineering Sun Yat-sen University
Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps
Aug 7, 2026Yang ShuMajority Voting Large Language Model Judges
Zhejiang University, Hangzhou, China

May 28, 2026, cs.CLJ/K move · Enter open · S save

Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

Guneet Kohli

Apple

Llm-As-A-Judge Judges Large Language Model Evaluation Majority Voting Textual Dataset

Open paper AskPDF arXiv

LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes' worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation algorithms helps -- established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across prompt variants, temperatures, chain-of-thought reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.

Abstract

Explore similar work

Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

A Finite-Calibration Regime Map for LLM Judge Panels

Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps