JudgeMoE: Distributional Aggregation for LLM-as-a-Judge

Yiqi Liu; Joseph James; Yang Wang; Kun Zhao; Chenghao Xiao; Chenghua Lin

cs.CLOct 5, 2026

JudgeMoE: Distributional Aggregation for LLM-as-a-Judge

Authors: Yiqi Liu, Joseph James, Yang Wang, Kun Zhao, Chenghao Xiao, Chenghua Lin

Organizations: The University of Manchester · The University of Sheffield · University of Pittsburgh · Shanghai University of Finance and Economics

Abstract

When an LLM judge scores an output, its score distribution retains uncertainty and disagreement information that is lost after scalar compression. We introduce JudgeMoE, a lightweight aggregator that assigns example-specific weights to cached judge score distributions and fuses them before computing a final score. A protocol study shows that score-range choice is unstable across judge--dataset settings and that soft scoring usually outperforms hard decoding. On the original 10-cell benchmark, JudgeMoE improves mean Spearman over uniform log pooling by $+0.079$ . Applying the same configuration to six additional cells yields a $+0.0393$ mean gain over the strongest local single judge across 16 cells, with positive differences in 12/16 cells and a one-sided Wilcoxon signed-rank $p=0.0091$ . Validation-based analyses further show that the preferred aggregation method depends on the task and judge pool.

Figures & tables

Figure 1: Three levels of information retention for the same judge score distributions. Hard-score aggregation uses only decoded labels, and soft-score aggregation uses expected scores. JudgeMoE pools the full score distributions before scoring, retaining confidence, entropy, near-tie mass, and bucket-level agreement during aggregation.

Dataset	Task	Size
SummEval	Summarization	6,400
TopicalChat	Dialogue quality	2,160
ROSCOE-eSNLI	Reasoning quality	151
HelpSteer2	Helpfulness scoring	101,620 main
WMT MQM	Translation quality	150,347
MT-Bench-human	Pairwise pref.	2,575 eval

Table 1: Datasets used in the paper. Direct-score datasets are evaluated with judge-score correlations, while pairwise datasets are used for auxiliary supervision and preference evaluation. The MT-Bench-human count is the filtered non-tie evaluation set; the SFT row lists the validation set used for pairwise evaluation and the training subset used for the auxiliary pairwise loss. Split-level counts appear in Appendix Table 9 .

Figure 2: Protocol sensitivity. Left: the best score range varies across dataset–model cells. Right: soft scoring usually improves over hard decoding across valid runs and scales.

Comparison	Baseline	JudgeMoE	$\Delta$	95% CI	one-sided $p$	two-sided $p$
Uniform log	0.304	0.383	+0.079	[0.039, 0.119]	0.005	0.010
Cov-log	0.341	0.383	+0.043	[-0.001, 0.085]	0.053	0.105
Best single	0.341	0.383	+0.042	[-0.007, 0.088]	0.053	0.105

Table 2: Main 10-cell baseline comparisons for the selected 6-judge JudgeMoE checkpoint. JudgeMoE shows its largest and statistically significant gain over uniform log pooling. Comparisons to covariance-aware fixed pooling and the best single-judge baseline have positive mean gaps, but their confidence intervals cross zero and are not statistically significant. Confidence intervals are paired bootstrap intervals over cells. The last two columns report one-sided and two-sided paired Wilcoxon signed-rank test p-values over cells, without multiplicity correction. “Best single” is the strongest single judge within each evaluated cell.

Follow-up	Hom. JudgeMoE	Het. JudgeMoE	Best single	Validation decision
SummEval + BERTSc.	0.646	0.670	0.517	keep expert
HelpSteer2 + RM	0.533	0.545	0.390	repair pool
WMT + trans. BERTSc.	0.239	0.133	0.190	keep LLM pool
WMT + COMET	0.247	0.162	0.4326	use standalone

Table 3: Focused heterogeneous follow-ups on compact deployment settings. All entries are test Spearman correlations with human labels; higher is better. BERTSc. denotes BERTScore and RM denotes the reward model. The validation decision is chosen using validation labels before test reporting. “Keep expert” selects the heterogeneous pool, “repair pool” selects a validation-repaired subset, “keep LLM pool” rejects the added expert, and “use standalone” keeps the external metric outside JudgeMoE.

Observed pattern	Choice
Protocol variation	Select protocol
Strong fixed pool	Fixed pooling
Learned-weight gain	Full JudgeMoE
Weak judge	Subset + retraining
Strong external evaluator	Standalone evaluator

Table 4: Validation-guided deployment choices supported by the protocol study, main aggregation experiments, TopicalChat and WMT analyses, and heterogeneous-expert experiments.

Figure 3: Per-cell pattern for the main direct-score benchmark. JudgeMoE improves over the best single judge in most direct-score cells, but the per-cell breakdown also identifies failures on TopicalChat and WMT MQM@1–5 (Appendix Table 27 ). TopicalChat is a case where covariance-aware fixed pooling already outperforms JudgeMoE, while WMT MQM@1–5 motivates sparse repair: retraining JudgeMoE on the validation-selected sparse subset improves Spearman from 0.0376 to 0.1648 (Tables 11 and 53 ).

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Prompt content
Instruction	Evaluator role + criterion
Context	Source / prompt / dialogue
Candidate	Output to score
Reference	Optional task reference
Output	Scalar score in specified range

Appendix

Table 5: Shared prompt components for direct-score judging. Dataset-specific wording varies, while all prompts use the same scalar-score output format.

Task family	Default direct-score template skeleton
Summarization	You are an expert summarization evaluator. Evaluate the [aspect] of the generated summary. [Aspect-specific criterion.] SOURCE DOCUMENT: {source} GENERATED SUMMARY: {candidate} Return only a score from {low} to {high}.
Dialogue	You are an expert dialogue response evaluator. Evaluate the [aspect] of the assistant response. CONTEXT: {source} ASSISTANT RESPONSE: {candidate} Return only a score from {low} to {high}.
Assistant response	You are an expert evaluator of assistant responses. Evaluate the [aspect] of the response. USER REQUEST: {source} ASSISTANT RESPONSE: {candidate} Return only a score from {low} to {high}.
Reasoning	You are an expert evaluator of reasoning chains. Evaluate the overall quality of the reasoning chain. Consider correctness, completeness, and coherence. PROBLEM OR CONTEXT: {source} REASONING CHAIN: {candidate} Return only a score from {low} to {high}.
Translation	You are an expert translation evaluator. Evaluate the overall translation quality. Consider adequacy and fluency. SOURCE SEGMENT: {source} TRANSLATION: {candidate} [optional HUMAN REFERENCE] Return only a score from {low} to {high}.
FLASK-style	You are an expert evaluator of assistant responses. Evaluate the candidate response using the provided criterion and scoring rubric. INSTRUCTION: {source} CANDIDATE RESPONSE: {candidate} CRITERION: {criterion} [reference/rubric blocks] Return only a score from {low} to {high}.

Appendix

Table 6: Default direct-score prompt templates. All task families use the same scalar-score output format; prompt-style variants are evaluated separately in Table 16 .

Experiment	Paper content
Protocol study	Scale + soft/hard scoring
Aggregation	Main results + ablations
Seed sweep	6-judge / 3-judge variation
Pairwise study	Direct + pairwise training
Large judges	Quality + efficiency
WMT MQM	Pool failure + sparse retraining

Appendix

Table 7: Experiment groups and their corresponding analyses in the paper.

Family	Aggregation input	Full distribution	Example-specific weights	Interaction
Single-judge methods	Score / rationale / checklist	No	No	No
Jury / debate	Votes / answers / rationales	Usually no	Sometimes	Yes
Latent-reliability models	Labels / preferences	No	Judge-level	No
Scalar weighting	Scalar scores	No	Yes	No
Fixed opinion pooling	Score distributions	Yes	No	No
JudgeMoE	Cached score distributions	Yes	Yes	No

Appendix

Table 8: Comparison of related evaluator aggregation methods. JudgeMoE combines full score distributions with example-specific judge weighting.

Dataset	Train	Dev	Valid	Test	Total	Note
Direct-score corpus
All direct-score data	212,953	15,912	5,190	31,813	265,868	–
HelpSteer2	–	–	–	–	106,810	Train / validation preserved
SummEval	–	–	–	–	6,400	Human scores retained
TopicalChat	–	–	–	–	2,160	Dialogue aspects retained
ROSCOE	–	–	–	–	151	Diagnostic subset

Appendix

Table 9: Dataset composition and split counts for the released evaluation data. Licenses and use restrictions follow the original datasets and are recorded in the release manifest.

Item	Setting	Reference
Architecture	DeepSets, $d_{h}{=}128$ , dropout 0.1, log pool, $T{=}0.7$ , $\varepsilon{=}10^{-6}$	Table 13
Optimization	Adam, lr $8{\times}10^{-4}$ , batch 128, 20 epochs	Released logs
Objective	Huber loss; auxiliary ranking loss with margin 0.0	Section 3.3, Table 42
Seeds	13; sweep: 13, 17, 29	Table 41
Features	$K$ score probabilities + scalar summaries	Table 18
Splits	Original split labels; exact example-ID intersection	Table 9

Appendix

Table 10: Key settings and reproducibility references for the main aggregation experiments. The selected configuration is unchanged for the six additional evaluation cells.

Dataset	Scale	Mainline	3-judge	Best single	Best judge	Uniform log	Cov-log
SummEval	1–5	0.5134	0.4967	0.4687	Gemma-2-9B	0.4420	0.4313
SummEval	1–10	0.6101	0.6043	0.5657	Gemma-3-12B	0.6401	0.6494
TopicalChat	1–5	0.4357	0.4297	0.4683	Gemma-2-9B	0.3108	0.4928
TopicalChat	1–10	0.4270	0.3925	0.4458	Gemma-2-9B	0.3700	0.4708
ROSCOE	1–5	0.2870	0.2841	0.2641	Gemma-3-12B	0.2975	0.2142
ROSCOE	1–10	0.2041	0.1381	0.0773	Ministral-3-8B	0.0644	0.0516

Appendix

Table 11: Per-dataset breakdown for the main aggregation systems. The gains vary across dataset and scale cells: the main gains come from SummEval at 1–5, HelpSteer, and WMT MQM at 1–10, while TopicalChat and SummEval at 1–10 remain challenging against strong fixed baselines. ROSCOE is reported as a small-sample diagnostic cell and is interpreted cautiously because it contains only 151 examples.

Temperature	Avg. Sp.	$\Delta$ vs. single	Wins / 10
$T{=}1.0$	0.3828	0.0414	2
$T{=}0.7$	0.3832	0.0419	4
$T{=}1.3$	0.3729	0.0315	4

Appendix

Table 12: Temperature sensitivity for DeepSets + log pooling. The values are taken from the 3-way temperature comparison on the same 10 dataset–scale cells.

Learned aggregation configuration	Avg. Sp.
Linear + log pool	0.3511
FFN + linear pool	0.3505
Low-rank linear + log	0.3546
DeepSets + log pool	0.3828
DeepSets + log, $T{=}0.7$	0.3832
3-judge efficient	0.3656

Appendix

Table 13: Learned-aggregation selection on the original 10-cell suite. The selected mainline is DeepSets gating with tempered log pooling at

T{=}0.7

; Table 2 uses this fixed configuration for the main baseline comparisons.

Gate	Linear pooling	Log pooling
FFN	$0.3623\pm 0.0016$	$0.3751\pm 0.0004$
DeepSets	$0.3831\pm 0.0026$	$\mathbf{0.4068\pm 0.0016}$

Appendix

Table 14:

2\times 2

gate and pooling comparison on the original 10 cells. All four settings use the same judge manifests, feature construction, pooling temperature, 30-epoch training schedule, and three random seeds. Values are mean Spearman

\pm

standard deviation. This experiment uses a separate training schedule from the reported mainline checkpoint, so the absolute values are interpreted within this table.

Selector	Equal-cell Sp.	Weighted Sp.
Validation-selected protocol	0.1865	0.1674
Fixed 1–5 soft	0.1486	0.0269
Fixed 1–10 soft	0.1813	0.1002
Best protocol by test performance	0.2569	0.1681

Appendix

Table 15: Held-out protocol study over 24 dataset and model cells. The validation-selected protocol outperforms fixed 1–5 and 1–10 soft scoring on equal-cell Spearman. The selected protocols span all five tested score ranges, showing that protocol sensitivity is not limited to one fixed range.

Style	Rows	Avg. soft Sp.	Wins vs. default
Default	12	0.3230	0/12
Rubric	12	0.2275	0/12
Critical	12	0.2968	4/12

Appendix

Table 16: Reduced prompt-style protocol check over two judges, three datasets, two scales, and three phrasing variants. A non-default prompt is best in 4/12 dataset–model–scale cells. The rubric wording is uniformly weaker than the default prompt, while the critical wording helps in a small subset of cases, including Qwen on WMT MQM (+0.1158 at 1–5 and +0.3023 at 1–10).

Pool	Judges	Mean $\rho$	JudgeMoE Sp.	Wins vs. default
Default 2	2	0.7955	0.4049	–
Protocol-diverse 6	6	0.8472	0.3883	3 / 3

Appendix

Table 17: Protocol-diverse pool pilot on summeval , topical_chat , and wmt_mqm at 1--5 and 1--10 . The 6-judge pool augments a default 2-judge pool ( Qwen2.5-7B , gemma-2-9B ) with rubric and critical prompt variants of the same two models. Mean residual correlation increases, and average JudgeMoE performance drops slightly, indicating that added protocol diversity can fail to yield useful complementarity.

Feature set	Avg. Sp.	$\Delta$ vs. single	Wins / 10
Full current	0.3832	0.0419	6
Distribution + scores	0.3811	0.0398	2
Distribution only	0.3746	0.0332	2

Appendix

Table 18: Feature ablation for the main DeepSets + log-pool configuration. Most of the signal already lies in the judge score distributions, but the full scalar summary bundle provides a small and consistent gain over using distributions alone.

Input features	Mean Sp.	SD
Hard score only	0.2634	0.0028
Soft score only	0.3100	0.0088
Hard and soft scores	0.3671	0.0027
Probability-derived scalar summaries	0.3776	0.0053

Appendix

Table 19: Performance with different levels of access to judge outputs on the original 10 cells.

Cell	Full-6 $\bar{\rho}$	Strong-3 $\bar{\rho}$
Average (10 cells)	0.7775	0.7071
SummEval 1–10	0.7611	0.5860
TopicalChat 1–5	0.8705	0.7875
TopicalChat 1–10	0.9467	0.9150
WMT MQM 1–10	0.5883	0.5696

Appendix

Table 20: Mean off-diagonal residual correlation

\rho

among judge errors, computed from normalized soft scores relative to human labels. The large correlations on TopicalChat imply that most remaining error is shared across judges, which is consistent with Proposition 3 and helps explain why fixed covariance-aware pooling is already strong there.

Cell	Best test family	Selected	Correct
ROSCOE 1–10	sparse	sparse	yes
ROSCOE 1–5	fixed	fixed	yes
SummEval 1–10	fixed	fixed	yes
SummEval 1–5	sparse	fixed	no
TopicalChat 1–10	fixed	fixed	yes
TopicalChat 1–5	fixed	fixed	yes

Appendix

Table 21: Leave-one-cell-out diagnostic for the deployment heuristic. We perform a coarse threshold sweep over

\rho

, leave-one-out gain, and the fixed-vs-full margin on the eight direct-score cells with held-out validation and test splits. All eight folds recover the same selector: choose sparse when the leave-one-out gain is at least 0.14; otherwise choose fixed when

\rho\geq 0.5

and the fixed pool is already at least as strong as full JudgeMoE on validation; otherwise choose full. This rule matches the best test family on 6/8 cells.

Strategy	Avg. test Sp.	Weighted test Sp.
Best single (valid-selected)	0.2748	0.1860
Always fixed pool	0.3374	0.1319
Always full JudgeMoE	0.2975	0.1655
Always sparse variant	0.3497	0.1536
Best family by validation performance	0.3402	0.1650
LOO heuristic selector	0.3495	0.1694

Appendix

Table 22: Validation-only strategy selection on the eight direct-score cells with held-out validation and test splits. Thresholds are selected on the remaining cells and then applied to the held-out test cell. The resulting selector improves over always using full JudgeMoE and over always using the strongest fixed pool on equal-cell average test Spearman, while sample-weighted averages remain dominated by the large WMT cells and therefore stay close to the best family by test performance.

Strategy	Avg. test Sp.	Weighted test Sp.
Best single (valid-selected)	0.2748	0.1860
Full fixed pool	0.3374	0.1319
Full JudgeMoE	0.3154	0.1665
Valid-selected subset fixed	0.3133	0.2113
Valid-selected family	0.3732	0.2213
Best family by test performance	0.3789	0.2215

Appendix

Table 23: Validation-only subset selection on the eight held-out direct-score cells. The best fixed subset uses 3.38 judges on average and is smaller than the full six-judge pool in all eight cells. A validation-selected family over best single, full fixed pooling, full JudgeMoE, and subset-fixed pooling nearly matches the best family selected using test performance.

Validation signal	Recommended choice
High residual correlation; covariance-aware pooling already strong	Fixed covariance-aware pooling
Large leave-one-out gain or large subset/full-pool gap	Subset selection + retraining
Remaining gain from example-specific weighting	Full JudgeMoE

Appendix

Table 24: Validation signals and corresponding deployment choices. Appendix Table 21 reports the leave-one-cell-out evaluation.

Stage	Validation signal	Selection
Protocol	Scale / extraction differences	Select protocol
Family	Fixed / full / single performance	Select aggregation family
Subset	Subset performance	Select judge subset
Subset retraining	Subset / full-pool difference	Retrain on selected subset

Appendix

Table 25: Validation procedure for protocol, aggregation-family, and judge-subset selection. Supporting results are reported in Appendix Tables 15 , 21 , 22 , and 23 , and in Table 53 .

Subset	Weighted Sp.	$\Delta$ vs. single	$\Delta$ vs. cov-log
All 10 cells	0.3052	0.0413	0.0549
Exclude ROSCOE	0.3052	0.0413	0.0548
Exclude WMT	0.5393	0.1346	0.0961
Non-HelpSteer	0.1549	-0.0241	0.0236

Appendix

Table 26: Sample-weighted summaries for the mainline 6-judge JudgeMoE system. On the full 10-cell suite, the weighted bootstrap 95% interval is

[-0.0630,0.1399]

against the best single judge and

[0.0109,0.1014]

against covariance-aware pooling. Excluding ROSCOE leaves the weighted means essentially unchanged.

Cell	$n$	JudgeMoE	Best single	$\Delta$	95% CI	$p$
helpsteer2_train@1-10	101620	0.5386	0.3849	0.1537 †	[0.1502, 0.1572]	0.0100
helpsteer2_train@1-5	101620	0.5426	0.4087	0.1339 †	[0.1303, 0.1373]	0.0100
roscoe_esnli@1-10	151	0.2041	0.0773	0.1268	[-0.0240, 0.2675]	0.1294
roscoe_esnli@1-5	151	0.2870	0.2641	0.0228	[-0.0063, 0.0465]	0.1791
summeval@1-10	6400	0.6101	0.5657	0.0444 †	[0.0260, 0.0653]	0.0100
summeval@1-5	6400	0.5134	0.4687	0.0448 †	[0.0319, 0.0591]	0.0100

Appendix

Table 27: Per-cell paired bootstrap significance for the main 10-cell aggregation comparison against the best single judge, using the exact per-example predictions of the reported mainline checkpoint. This supplements the aggregate 10-cell Wilcoxon test by showing which benchmark cells individually support the positive JudgeMoE trend.

System	ECE $\downarrow$	MAE $\downarrow$
JudgeMoE	0.0588	0.1890
Best single	0.2120	0.2782
Uniform log	0.1785	0.2718
Cov-log	0.2214	0.2976

Appendix

Table 28: Continuous-score calibration check on the original 10 direct-score cells. ECE uses 10 equal-width bins over normalized predictions; within each bin we compute the absolute gap between mean prediction and mean human score, then average these gaps by bin frequency. Values are equal-cell averages, matching the main Spearman aggregation.

Dataset	JudgeMoE	Best single	Uniform log	Cov-log
SummEval	0.0661	0.3840	0.1621	0.2210
TopicalChat	0.0559	0.2640	0.1385	0.2118
ROSCOE	0.1053	0.2161	0.2693	0.3381
HelpSteer2	0.0133	0.0483	0.0859	0.0791
WMT MQM	0.0534	0.1476	0.2368	0.2568

Appendix

Table 29: Per-dataset ECE breakdown, averaged over the two main scales for each dataset. The aggregate calibration gain spans the full benchmark set: JudgeMoE has the lowest ECE on all five dataset averages in this diagnostic.

Comparison	Mean $\Delta$	SD Δ	Power @10	Cells for 80%
vs. uniform log	+0.0794	0.0692	0.98	5
vs. cov-log	+0.0427	0.0729	0.58	19
vs. best single	+0.0419	0.0812	0.49	24

Appendix

Table 30: First-order power note for the original 10-cell paired comparisons. We use the observed cell-level mean and standard deviation of the paired deltas with a one-sided

\alpha{=}0.05

normal approximation. The table explains why the 10-cell benchmark is sufficient for the uniform-log control but underpowered for the smaller gaps against covariance-aware pooling and the best single judge.

Component	Selection setup	Reported use
Protocol study	Five score ranges with hard/soft extraction; validation-based protocol selection over 24 cells	Select scoring protocol
Main architecture	Five architecture/pooling variants, temperature sweep, and feature ablation on the original 10 cells	Select 6-judge DeepSets + log pooling
Compact 3-judge pool	Candidate triples from protocol-study screening	Efficiency setting
WMT subset selection	Top- $k$ subsets and leave-one-out selection on WMT validation data	Retrain selected subsets
General subset selector	All 63 nonempty judge subsets over eight validation/test cells	Select judge subset and aggregation family
Heterogeneous experts	One task-aligned expert added at a time using task-specific train or validation data	Evaluate heterogeneous pools and standalone experts

Appendix

Table 31: Selection procedures for the main experiment groups. The 6-judge architecture selected on the original 10 cells is reused for the HelpSteer2 validation, FLASK, WildBench, and heterogeneous evaluations.

Cell	JudgeMoE	Single	Cov-log	$\Delta$ single	$\Delta$ cov-log
HelpSteer2 val.@1–5	0.4394	0.3851	0.4212	+0.0543	+0.0182
HelpSteer2 val.@1–10	0.4562	0.3805	0.4358	+0.0756	+0.0204
12-cell avg.	0.3940	0.3483	0.3552	+0.0457	+0.0388

Appendix

Table 32: Held-out HelpSteer2-validation extension of the selected mainline checkpoint. These two cells were added only after the original 10-cell architecture sweep, so they evaluate the frozen mainline checkpoint without rerunning architecture search. Unlike the FLASK support check, both new HelpSteer2-validation cells are positive against the best single judge and covariance-aware log pooling. The table reports the two new cells together with the resulting 12-cell averages; the corresponding one-sided paired Wilcoxon tests are

p{=}0.0212

against the best single judge and

p{=}0.0461

against covariance-aware log pooling.

Cell	JudgeMoE	Single	Cov-log	$\Delta$ single	$\Delta$ cov-log
FLASK@1–5	0.5452	0.4897	0.5452	+0.0555	-0.0001
FLASK@1–10	0.4364	0.4148	0.4449	+0.0216	-0.0085
12-cell avg.	0.4011	0.3598	0.3663	+0.0413	+0.0349

Appendix

Table 33: Secondary FLASK support check for the selected mainline checkpoint. These two cells were added only after the original 10-cell architecture sweep, so they evaluate the frozen mainline checkpoint without rerunning architecture search. FLASK@1–5 ties covariance-aware log pooling up to rounding, consistent with the mixed fixed-pool picture. The table reports the two new FLASK cells together with the resulting 12-cell averages; the corresponding one-sided paired Wilcoxon tests are

p{=}0.0261

against the best single judge and

p{=}0.1018

against covariance-aware log pooling. We use FLASK mainly as out-of-domain support for the best-single trend; HelpSteer2 validation remains the primary fixed-pool expansion check.

Setting	Comparison	Left	Right	$\Delta$	95% CI	$p$
SummEval 1–10 + BERTScore	Het. JudgeMoE vs. best single	0.6703	0.5172	+0.1531	[0.1104, 0.1944]	0.0007
SummEval 1–10 + BERTScore	Het. JudgeMoE vs. uniform log	0.6703	0.1930	+0.4773	[0.4219, 0.5320]	0.0007
SummEval 1–10 + BERTScore	Het. JudgeMoE vs. hom. JudgeMoE	0.6703	0.6460	+0.0242	[-0.0026, 0.0509]	0.0733
HelpSteer2 1–10 + reward	Het. JudgeMoE vs. best single	0.5454	0.3900	+0.1553	[0.1468, 0.1644]	0.0025
HelpSteer2 1–10 + reward	Het. JudgeMoE vs. uniform log	0.5454	0.3102	+0.2352	[0.2239, 0.2472]	0.0025
HelpSteer2 1–10 + reward	Het. JudgeMoE vs. hom. JudgeMoE	0.5454	0.5330	+0.0124	[0.0066, 0.0189]	0.0025

Appendix

Table 34: Paired bootstrap significance for the strongest heterogeneous follow-ups. “Left” and “Right” are Spearman correlations;

\Delta

is left minus right. Intervals and

p

-values come from paired bootstrap over matched examples.

Pool	Best single	Uniform log	Cov-log	JudgeMoE
3-judge LLM pool	0.1898	0.2031	0.1988	0.2468
3-judge LLM + COMET	0.4326	0.3926	0.3794	0.1620

Appendix

Table 35: WMT MQM@1–10 follow-up with COMET as a task-aligned heterogeneous expert. COMET is a much stronger standalone evaluator than any constituent LLM judge, but appending it to the mixed pool degrades JudgeMoE sharply.

Setting	Best single	Full JudgeMoE	$\Delta$ vs. single	min frac $>$ single	min frac $\geq 90\%$ gain	min frac $\geq 95\%$ gain
SummEval 1–10 + BERTScore (repaired)	0.5172	0.6682	0.1510	0.05	0.40	1.00
HelpSteer2 1–10 + reward	0.3900	0.5670	0.1769	0.05	0.70	0.70

Appendix

Table 36: Sample-efficiency summary for the two strongest heterogeneous positives. Fractions refer to the portion of the training split used to fit JudgeMoE; the gain-recovery columns normalize against the full-data gain over the best single judge. Both settings beat the best single judge with only 5% of training data, but the reward-model append saturates more slowly than the BERTScore expert.

Setting	Expert	mean $\Delta$ vs. hom. abs.-err.	mean $\Delta$ vs. single abs.-err.	top-gate rate	Role
SummEval 1–10 + BERTScore	BERTScore	0.0744	0.3167	0.1203	frequent expert
HelpSteer2 1–10 + reward	Skywork reward	0.0031	0.0321	0.0000	soft calibration

Appendix

Table 37: Compact qualitative summary for the strongest heterogeneous positives. The top-gate rate measures how often the added expert becomes the highest-weight judge on a test example. BERTScore acts like a genuine conditional expert on SummEval, whereas the reward model on HelpSteer2 behaves more like a soft calibration signal.

Setting	Strong single	Best single	Hom. JudgeMoE	Het. JudgeMoE	$\Delta$ Het.	$\Delta$ Hom.
SummEval@1–10	Qwen2.5-72B	0.5172	0.6460	0.6703	0.1779	0.1536
HelpSteer2@1–10	Qwen2.5-72B	0.3900	0.5330	0.5454	0.0716	0.0593
WildBench@1–10	Qwen2.5-72B	0.3520	0.1719	0.0854	0.0822	0.1688

Appendix

Table 38: Aligned strong single-judge follow-ups using Qwen2.5-72B-Instruct on the same evaluation splits as the heterogeneous pilots. JudgeMoE remains stronger on the two positive follow-up cells, while WildBench remains difficult for all methods.

Setting	Groups	Avg. cand.	Best group score	Het.	Hom.	Single	$\Delta$ Het.
summeval best-of- $N$	100	9.3200	0.9979	0.9365	0.9446	0.9194	-0.0081

Appendix

Table 39: Offline best-of-

N

SummEval case study, grouping candidate summaries by source article. The repaired heterogeneous pool still beats the best single judge, while the repaired homogeneous pool remains stronger on this reranking-style proxy.

Component	Setting
Direct-score target	Human scores normalized to $[0,1]$
Direct loss	Huber loss on the fused score
Pairwise target	Binary preference converted to $\{-1,+1\}$
Pairwise loss	Margin-ranking loss with tuned auxiliary weight and fixed margin $m{=}0.0$
Large sweeps	JSONL-based resume support for interrupted runs
Subset comparisons	Exact example-ID overlap for sample-aligned evaluation

Appendix

Table 40: Implementation details relevant to reproduction.

Setting	Mean Sp.	Std.	$\Delta$ vs. single
Mainline 6-judge	0.3826	0.0013	0.0413
Efficient 3-judge	0.3638	0.0028	0.0245

Appendix

Table 41: Seed stability for the strongest 6-judge and 3-judge settings. The small standard deviations show that the two reported deployment settings are stable across random seeds.

$\lambda$	Direct Sp.	$\Delta$ pair acc.
0.1	0.3725	0.0360
0.5	0.4192	0.0462
1.0	0.4190	0.0431

Appendix

Table 42: Joint training with pairwise auxiliary loss in the separate 3-judge pairwise-ablation setting (MT-Bench-human and Summarize-from-Feedback validation, each at 1–5 and 1–10). The direct Spearman values summarize this 4-row setting, distinct from the 10-cell 6-judge mainline benchmark. Moderate pairwise weight improves pairwise accuracy without hurting direct-score quality.

Scale	Dataset	Base acc.	Joint acc.
1–5	MT-Bench-human	0.6369	0.6823
1–5	Summarize-from-Feedback	0.5140	0.6054
1–10	MT-Bench-human	0.7200	0.7250
1–10	Summarize-from-Feedback	0.5994	0.6244

Appendix

Table 43: Per-dataset pairwise gains for the best joint-training setting. The joint model improves on both pairwise datasets at both scales, which is why the paper treats pairwise supervision as a useful auxiliary signal for direct-score training.

Scale	Best single	Uniform log	Direct	Joint	Pair-only
1–5	0.5482	0.5385	0.5411	0.5495	0.5112
1–10	0.5674	0.5578	0.5601	0.5507	0.5026

Appendix

Table 44: Supplementary OOD evaluation on 3,114 English non-tie Arena human preferences. The pattern is consistent with the main pairwise story: joint training helps at 1–5, pairwise-only remains weak at both scales, and at 1–10 the strongest single judge is already difficult to improve upon.

Scale	Best single	Uniform log	Direct	Joint	Pair-only
1–5	0.4870	0.4479	0.4917	0.4656	0.4562
1–10	0.5201	0.4870	0.5036	0.5012	0.5130

Appendix

Table 45: Supplementary OOD evaluation on a more recent, length-controlled Arena Expert subset with 844 English A/B comparisons. The benchmark is filtered to contexts that fit the local three-judge pool without OOM failures (‘source

\leq

3000‘ chars; responses

\leq

6000 chars). The result is more conservative than Arena Human Preference: direct-only is the strongest learned setting at 1–5, while at 1–10 the best single judge remains strongest.

Scale	Best single	Uniform log	Direct	Joint	Pair-only
1–5	0.5605	0.5446	0.5361	0.5259	0.4952
1–10	0.5821	0.5685	0.5821	0.5657	0.4832

Appendix

Table 46: Supplementary OOD evaluation on a recent length-controlled PPE-Human-Preference-V1 subset with 1,759 English A/B comparisons. The benchmark is filtered to ‘source

\leq

4000‘ chars and responses ‘

\leq

8000‘ chars. The result is more conservative than Arena Human Preference: at 1–5 the best single judge remains strongest, and at 1–10 direct-only only matches the best single judge while joint training no longer helps.

Scale	Best single	Uniform log	Cov-log	JudgeMoE	Hard
1–5	0.2285	0.2183	0.2638	0.1955	0.0804
1–10	0.2351	0.2650	0.2589	0.2709	-0.0157

Appendix

Table 47: Supplementary direct-score evaluation on a modern open-ended WildBench v2-hard subset. We use the released train split, take the provided GPT-4 reference response as the candidate answer, and keep a length-controlled subset of 246 examples (‘prompt

\leq

6000‘ chars; candidate ‘

\leq

8000‘ chars). The pattern remains conditional: covariance-aware log pooling is strongest at 1–5, while the 6-judge JudgeMoE mainline becomes strongest at 1–10.

Standalone evaluator	Cells	Standalone	JudgeMoE	JMoE wins
DeepSeek-V3.2	4	0.5710	0.4966	1/4
Qwen2.5-72B	4	0.5362	0.4966	1/4

Appendix

Table 48: Aligned comparisons with larger standalone evaluators on the same evaluation subsets. These evaluators are included as standalone deployment options in the validation analysis.

System	Active params / token	Relative compute
Local single judge	7–12B dense	0.19–0.32 $\times$
Local 3-judge efficient pool	28B dense across separate passes	0.76 $\times$
Local 6-judge mainline pool	53B dense across separate passes	1.43 $\times$
DeepSeek-V3.2-Exp	37B activated, 671B total	1.00 $\times$

Appendix

Table 49: Hardware-independent first-order inference-compute proxy. For autoregressive inference, weight-matrix FLOPs scale approximately linearly with active parameters and processed tokens, so under the same evaluation workload the relative compute is well approximated by active-parameter counts per token. Under the 1K-example SummEval 1–5 workload used in our deployment note (roughly 474K processed tokens), this proxy corresponds to about 26.5 PFLOPs for the local 3-judge pool, 50.2 PFLOPs for the local 6-judge pool, and 35.1 PFLOPs for DeepSeek-V3.2-Exp. The six local judges are independent and can be run in parallel; in the same probe, single-judge jobs took 16.8–27.4 minutes, so ideal six-way parallel latency is bounded by the slowest job. The sequential sum was 134.4 minutes.

Type	Cell	Example	Human	JudgeMoE	Cov-log	Top gate	Judge scores	Gate weights
+	summeval@1-5	S1499-relevance	0.833	0.825	0.344	L3	[.11,1,.50,.32,.01,.43]	[.08,.25,.35,.15,.04,.13]
+	helpsteer2@1-10	H10914-helpfulness	1.000	0.901	0.524	G3	[.57,.90,.52,.50,.50,.55]	[0,.98,0,.01,0,0]
+	wmt-mqm@1-10	W66502	0.800	0.798	0.564	G2	[.77,.64,.54,.57,.52,.56]	[1,0,0,0,0,0]
+	roscoe-esnli@1-10	R30	1.000	0.822	0.580	G3	[.56,.90,.50,.54,.49,.60]	[.18,.41,0,.17,0,.23]
-	topical-chat@1-5	T12-Engaging	0.333	0.898	0.390	G3	[.39,1,.51,.39,.42,.34]	[.10,.31,.15,.03,.16,.24]

Appendix

Table 50: Illustrative per-example cases where learned aggregation improves over (+) or fails against (-) fixed cov-log pooling. Judge-score and gate-weight vectors use the order G2/G3/L3/Min/Q2.5/Q3.5. Examples are selected automatically by the largest absolute-error change in the indicated direction. Across the 10 direct-score cells, the JudgeMoE prediction is closer to the human score than cov-log in 75.5% of examples on an equal-cell basis (84.6% example-weighted).

Scale	Cand.	$\rho_{s}$ (valid,test)	Sel. valid	Sel. test	Test rank
1-5	120	0.973	0.143	0.153	1
1-10	120	0.974	0.213	0.232	3

Appendix

Table 51: Validation–test stability for the WMT sparse fixed-pool search. Candidates include all single judges and all non-single subsets under uniform-log and cov-log pooling. The selected candidate is chosen by validation Spearman; its test rank is computed among the same candidates by test Spearman.

Scale	Judge	Judge Sp.	MAE	Mean gate	Top-gate	Diagnostic readout
1-5	G2	0.141	0.186	0.373	46.1%	best single; largest gate mass
1-5	G3	0.019	0.256	0.244	21.7%	secondary evidence
1-5	L3	-0.030	0.269	0.221	19.2%	secondary evidence
1-5	Min	0.086	0.299	0.140	12.7%	secondary evidence
1-5	Q2.5	-0.148	0.651	0.006	0.3%	known contaminant; gate almost removes it
1-5	Q3.5	0.014	0.400	0.017	0.0%	secondary evidence

Appendix

Table 52: WMT per-judge gate behavior connecting sparse-repair theory to the learned gate. Judge Spearman/MAE provide an empirical precision proxy; mean gate and top-gate rates are computed from the learned full-pool gate. Rank correlation between per-judge Spearman and mean gate is 1-5: 0.71, 1-10: 1.00. These correlations use only six judges and are diagnostic. The gate identifies weak judges, but at 1–5 sparse retraining is still needed because coarse scores leave too little coherent signal in the contaminated full pool.

Scale / subset	System or diagnostic	Spearman
1–5 (full)	Best single judge	0.1413
1–5 (full)	Top-2 sparse log pool	0.1473
1–5 (full)	Sparse JudgeMoE (2 selected judges)	0.1648
1–5 (full)	Full-6 uniform log pool	0.0016
1–5 (full)	Mainline JudgeMoE	0.0376
1–10 (full)	Best single judge	0.1797

Appendix

Table 53: Deeper WMT MQM failure-and-recovery analysis. At 1–5, naive all-judge fusion degrades, but held-out-validation-selected sparse composition recovers the signal and a sparse JudgeMoE trained on the selected subset performs best. At 1–10, better score resolution already makes the full 6-judge JudgeMoE competitive, while the sparse 4-judge JudgeMoE retains most of that performance with a cleaner pool.

Figure 4: Equal-cell reliability diagram for the main direct-score benchmark. For each system and each bin, we average the bin-wise predicted score and human score across the 10 benchmark cells, so the plot complements the equal-cell ECE table and the example-weighted summary.

Figure 5: Compact visualization of the WMT MQM failure mode. At 1–5, sparse composition and dropping one weak judge both improve over naive full-pool aggregation, and the sparse retraining follow-up shows that a cleaned judge pool can repair the failure. At 1–10, better score resolution lifts both sparse pooling and the learned mainline model, with sparse retraining retaining most of the mainline gain.

Explore similar work

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2Llm-As-A-Judge Large Language Model Evaluation
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
May 25, 2026Delip Rao, Chris Callison-BurchLarge Language Model Judges Judgement
University of Pennsylvania
LLM Judge Validation Under Sparse Overlap: From Inference to Design
Sep 25, 2026Junxuan Li, Arko Mukherjee, Soumyabrata PalLarge Language Model Judges Token Budget Allocation
Adobe · Adobe Research

Feb 9, 2026, cs.LGJ/K move · Enter open · S save

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Jitian Zhao, Changho Shin, Tzu-Heng Huang, Satya Sai Srinath Namburi GNVV+1

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

Llm-As-A-Judge Large Language Model Evaluation

Open paper AskPDF arXiv

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.

Abstract

Explore similar work

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

LLM Judge Validation Under Sparse Overlap: From Inference to Design