Where Do LLM Ensemble Gains Come From? Decomposing Candidate Value, Diversity, and Aggregation

Donghwan Kim

cs.CLOct 13, 2026

Where Do LLM Ensemble Gains Come From? Decomposing Candidate Value, Diversity, and Aggregation

Authors: Donghwan Kim

Organizations: Independent Researcher

Abstract

LLM ensemble gains are often measured against a single baseline, but the value attributed to additional candidates can change with the comparison used. We study an LLM aggregator under four matched conditions: Direct (no candidates), Single-slot (one candidate in a multi-slot interface), Duplicate (one candidate repeated), and Multi-candidate (separately generated candidates). Under either non-diverse control, overall candidate gain (CIV) splits algebraically into an additional-candidate contrast (DSV) and a candidate-context contrast (CCV); the split, but not CIV, depends on the chosen control. On MultiHopRAG model diversity (k=2), the repetition gap Rep=-8.33 pp [-13.25, -3.75] quantifies the difference between gains measured against Duplicate and Single-slot. On a matched FRAMES stronger-panel extension (n=200, k=3, aggregator output limit 4096), paired Delta Rep=+9.67 pp [+6.00, +13.50]; within panels, Rep changes sign (original -6.33; stronger +3.33). In FRAMES evidence aggregation, candidate-context contrasts remain strongly negative under both non-diverse controls. Our results show why ensemble evaluations should state the reference control before attributing gains to additional candidates.

Figures & tables

Figure 1: Four aggregation conditions for panel size

k{=}3

(comparisons, not an execution order). Reader-facing names are primary; experiment IDs appear in parentheses. Single-slot and Duplicate accuracies are averaged over candidate index

i

; Single-slot does not average over slot position in the primary Acc. Separately generated Multi-candidate answers may coincide. Neither non-diverse control identifies pure causal diversity. Formal estimands are defined in Section 3 .

Bench.	Src.	CIV	DSV dup	CCV E
FRAMES	S	$-0.8$ [ $-5.2$ , $4.0$ ]	$0.0$ [ $-2.7$ , $3.0$ ]	n/a
FRAMES	M	$+0.7$ [ $-10.3$ , $12.0$ ]	$+1.8$ [ $-2.5$ , $6.8$ ]	n/a
FRAMES	E	$-15.9$ [ $-22.6$ , $-9.4$ ]	$+1.9$ [ $-0.9$ , $4.6$ ]	$-17.8$ [ $-24.1$ , $-11.3$ ]
MultiHop	S	$-6.0$ [ $-11.3$ , $-0.7$ ]	$+4.3$ [ $0.7$ , $8.0$ ]	n/a
MultiHop	M	$-3.0$ [ $-9.0$ , $3.0$ ]	$+9.3$ [ $5.0$ , $14.0$ ]	n/a
MultiHop	E	$-13.3$ [ $-20.7$ , $-6.3$ ]	$+0.1$ [ $-2.0$ , $2.1$ ]	$-13.4$ [ $-20.3$ , $-6.6$ ]

Table 1: Main Luna breakdown (pp, 95% CIs). DSV is gain over Duplicate (

\mathrm{DSV}_{\mathrm{dup}}

; C3b control). FRAMES M uses a pairwise

k{=}2

unit (30 distinct items), not FRAMES

n{=}200

. Evidence CCV uses mean-duplicate C3b.

Panel	Direct	Single-slot	Duplicate	Multi-cand.	Rep
Original	46.5	45.2	38.8	41.5	$-6.33$ [ $-9.33$ , $-3.67$ ]
Stronger	46.5	55.3	58.7	62.0	$+3.33$ [ $+0.83$ , $+5.83$ ]

Table 2: Absolute arm accuracies (%) and Rep (pp) on matched FRAMES

n{=}200

(aggregator output limit 4096). Direct (C2) is shared across panels. Columns: Direct / Single-slot / Duplicate / Multi-candidate (experiment IDs C2 / C1 slot / C3b / C1). Rep

=

Duplicate

-

Single-slot. Absolute accuracies are rounded to one decimal; Rep uses the unrounded matched contrast. Rep CIs are item-paired bootstrap (

B{=}2000

, seed 2026).

	$\Delta$ CIV	$\Delta$ DSV dup	$\Delta$ DSV slot	$\Delta$ Rep
Pooled $n{=}200$	$\mathbf{+20.50}$	$\mathbf{+0.67}$	$\mathbf{+10.33}$	$\mathbf{+9.67}$
	$[14.00,27.00]$	$[-6.00,7.00]$	$[3.67,16.67]$	$[6.00,13.50]$

Table 3: Matched FRAMES stronger-minus-original effects (pp;

k{=}3

, aggregator output limit 4096). Pooled

n{=}200

is primary.

\Delta

CCV

{}_{\mathrm{dup}}=+19.83

[+15.17,+24.67]

and

\Delta

CCV

{}_{\mathrm{slot}}=+10.17

[+6.33,+14.17]

are stated in text and Figure 3 . Cohort diagnostics are in Appendix D .

Figure 2: Left: absolute MultiHop model-diversity contrasts (gain over Duplicate, gain over Single-slot, and Rep). Right: matched FRAMES stronger-minus-original changes (

n{=}200

, 4096-token aggregation). Both panels show paired 95% CIs.

Figure 3: Accounting split of the same

\Delta

CIV under Duplicate and Single-slot (not a causal path diagram). Duplicate:

\Delta

CCV

{}_{\mathrm{dup}}=+19.83

\Delta

DSV

{}_{\mathrm{dup}}=+0.67

; Single-slot:

\Delta

CCV

{}_{\mathrm{slot}}=+10.17

\Delta

DSV

{}_{\mathrm{slot}}=+10.33

. Both sum to

\Delta

CIV

=+20.50

;

\Delta

Rep

=+9.67

links the DSV values. Paired CIs are in Table 3 .

Figure 4: Left: restoring raw evidence beside candidates (accuracy %). Right: FRAMES evidence candidate-context contrasts under Single-slot and Duplicate at 4096 tokens (point estimates). CIV (Multi-candidate

-

Direct) does not change between those two baselines by definition.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Appendix A. Historical primary split figure. DSV

=\mathrm{DSV}_{\mathrm{dup}}

. CIV and DSV dup show paired 95% CIs. Evidence CCV has a CI; sampling/model CCV markers are identity points without separate CIs.

Dataset	Source	$n$	$k$	Tokens	Unit	Status
FRAMES	S	200	3	1024	item	Primary (A)
FRAMES	M	30	2	1024	pairwise	Primary (A)
FRAMES	E	180	3	1024	item	Primary (A)
MultiHop	S/M/E	100	2/3	1024	item/pair	Primary (A)
FRAMES	stronger	200	3	4096	item	Extension (B)

Appendix

Table 4: Core experiment design (verified against frozen configs). FRAMES M uses 30 distinct items with pairwise comparison/averaging, not the generic FRAMES

n{=}200

pool. The matched stronger-candidate row combines the original

95

-item cohort and a nonoverlapping

105

-item complement; it is a later controlled extension, not two independent benchmarks.

Analysis	Status	Notes
S/M/E: CIV, DSV dup , CCV	Confirmatory (A)	Historical primary
4096 checks; RAW+candidates; stronger-panel extension; MultiHop Rep/slot; controlled answer-multiplicity; Qwen; explanations	Extension (B)	Fixed before outcomes
Slot $n{=}60$ ; compact rationale	Diagnostic (C)	Not primary
Human audit $n{=}60$	Completed (B)	Stratified; no recalibration

Appendix

Table 5: Analysis hierarchy.

Cohort	$\Delta$ CIV	$\Delta$ DSV dup	$\Delta$ DSV slot	$\Delta$ Rep
Original cohort $n{=}95$	$+17.89$	$-1.05$	$+10.53$	$+11.58$
	$[9.47,26.32]$	$[-10.18,8.42]$	$[1.75,20.00]$	$[6.67,16.84]$
Compl. $n{=}105$	$+22.86$	$+2.22$	$+10.16$	$+7.94$
	$[13.33,31.43]$	$[-6.67,10.48]$	$[0.95,18.73]$	$[2.86,13.33]$

Appendix

Table 6: Matched FRAMES stronger-minus-original cohort diagnostics (pp). Same protocol cohorts, not independent benchmarks.

Quantity	Value
Rater agreement (3-class)	$52/60{=}86.7\%$ ; $\kappa{=}0.746$
Human vs. judge agreement	$50/60{=}83.3\%$ ; $\kappa{=}0.645$
TP / FP / TN / FN	$16$ / $0$ / $34$ / $10$

Appendix

Table 7: Human audit summary (stratified

n{=}60

	Human Correct	Human Incorrect
Judge Correct	TP ${=}16$	FP ${=}0$
Judge Incorrect	FN ${=}10$	TN ${=}34$

Appendix

Table 8: Confusion matrix (positive

=

judge Correct; reference

=

final human).

Figure 6: Appendix F. Cost summary. Accuracy versus deployment dollar cost. Costs are total deployment costs (USD) on a 100-item FRAMES overlap subset of the historical panels; judge costs are excluded. Because policy construction and model averaging differ from Table 1 , these accuracies are not directly comparable to Table 1 estimates. Here

D_{S}

denotes sampling diversity: three samples per generator, averaged across the three generators. Filled markers are non-dominated within each panel. Policies use different token limits; C2 PRO also changes model setup.

Bench.	Estimand	Est. (pp)	95% CI
FRAMES	CIV E	$-15.7$	[ $-28.7$ , $-2.8$ ]
FRAMES	DSV E	$+1.5$	[ $-3.7$ , $+7.4$ ]
FRAMES	DSV M	$+6.7$	[ $0.0$ , $+13.3$ ]
MultiHop	CIV E	$-15.0$	[ $-27.2$ , $-3.9$ ]
MultiHop	DSV E	$+0.2$	[ $-2.4$ , $+2.6$ ]
MultiHop	DSV M	$+5.6$	[ $+0.6$ , $+10.6$ ]

Appendix

Table 9: Kimi robustness subsets (not the main Luna confirmatory panel).

Bench.	Candidate configuration	Acc.	vs. Luna Direct (C2)
FRAMES	Luna Direct (C2)	44.5%	n/a
FRAMES	Gemini High	61.5%	higher
FRAMES	Grok High	52.0%	higher
FRAMES	Sonnet High	50.0%	higher
MultiHop	Luna Direct (C2)	87.0%	n/a
MultiHop	Gemini High	95.0%	higher

Appendix

Table 10: Standalone task accuracy vs. Luna Direct (C2) on the earlier standalone-candidate evaluation stack (FRAMES Luna Direct

=44.5\%

). This is not the matched stronger-panel Direct (C2) in Table 2 (

46.5\%

at aggregator output limit 4096).

Contrast	FRAMES	MultiHop
R1 $-$ A0 DSV	$-1.50$ [ $-8.17$ , $+5.00$ ]	$+2.33$ [ $-4.00$ , $+8.67$ ]
R2 $-$ R1 DSV	$+0.83$ [ $-5.17$ , $+6.67$ ]	$-4.67$ [ $-10.67$ , $+0.67$ ]

Appendix

Table 11: Post-hoc exploratory compact-rationale contrasts (

K{=}200

). This exploratory analysis was defined after the full-explanation results. The primary estimates and conclusions remain unchanged.

Language Model Ensembles

Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.

See all 135 papers

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM Evaluation LLM-as-a-Judge
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
Mixture of Complementary Agents for Robust LLM Ensemble
May 21, 2026Yichi Zhang, Kevin Lu, Yuang Zhang +3Language Model Ensembles Multi-LLM Collaboration
DIMACS, Rutgers University · Department of Mathematics, Rutgers University · Department of Computer Science, George Mason University +1
Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
Jul 22, 2026Donghwan KimLLM Evaluation Language Model Ensembles
Aidentyx Inc., San Jose, CA, USA

Feb 9, 2026J/K move · Enter open · S save

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Jitian Zhao, Changho Shin, Tzu-Heng Huang, Satya Sai Srinath Namburi GNVV+1

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Language Model Ensembles

Open paper AskPDF arXiv

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.

Abstract

Language Model Ensembles

Explore similar work

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Mixture of Complementary Agents for Robust LLM Ensemble

Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles