When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines
Authors: Artem Maryanskyy, Dmitry Budnikov, Alibek T. Kaliyev
Organizations: Uber Technologies, Sunnyvale, CA 94085, USA · Laboratories of Electrical, Thermal Technologies and Energy Saving, Federal Scientific Agroengineering Center VIM, 109428 Moscow, Russia · Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA
Multi-agent LLM pipelines produce contradictory evidence on whether team diversity improves output quality: heterogeneous Mixture-of-Agents teams outperform single models, yet homogeneous Self-MoA teams consistently win under synthesis-based aggregation. We propose a resolution by identifying the selection bottleneck -- a crossover threshold in aggregation quality that determines whether diversity helps or hurts. Under this model, we obtain a closed-form crossover threshold s∗ (Proposition 1) that separates the regimes where diversity helps and hurts. In a targeted experiment spanning 42 tasks across 7 categories (N=210), a diverse team with judge-based selection achieves a win rate of 0.810 against a single-model baseline, while a homogeneous team scores 0.512 -- near chance (Glass's Δ=2.07). Judge-based selection outperforms MoA-style synthesis by ΔWR=+0.631 -- the synthesis approach is preferred over the baseline in zero of 42 tasks by the judge panel. A decoupled evaluation with independent judges confirms all directional findings (Spearman ρ=0.90). Exploratory evidence suggests that including a weaker model improves performance while reducing cost (p<10−4, not pre-registered). Our results suggest that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines.
Figures & tables
Figure 1: The selection bottleneck. (a) Theoretical output quality Q(T,s) as a function of selector quality s . The diverse team (blue) crosses the homogeneous baseline (gray) at the crossover threshold s∗ (dashed vertical). Below s∗ , diversity hurts; above it, diversity helps. (b) Empirical operating regimes from V4 data. Judge-based selection operates well above s∗ (WR = 0.810), majority vote sits near s∗ (WR = 0.496), and MoA-style synthesis falls far below it (WR = 0.179). The homogeneous baseline (WR = 0.512, dotted) is shown for reference.
Cell
Agents
Selector
Tests
div_strong+judge
Opus + GPT-5.4 + Gem.-Pro
Judge panel
Reference
homo_opus+judge
Opus ×3
Judge panel
Diversity effect
div_mixed+judge
Opus + Gem.-Pro + Haiku
Judge panel
Weak-model effect
div_strong+vote
Opus + GPT-5.4 + Gem.-Pro
Majority vote
Judge vs. vote
div_strong+synth
Opus + GPT-5.4 + Gem.-Pro
MoA synthesis
Selection vs. synthesis
Table 1: Experimental design. Five cells test specific hypotheses by varying one factor relative to the reference cell. Each cell is evaluated on all 42 tasks ( N=42 per cell, N=210 total).
Cell
Original WR
Decoupled WR (2J)
Decoupled WR (3J)
div_mixed+judge
0.929
0.726
0.722
div_strong+judge
0.810
0.611
0.500
div_strong+vote
0.496
0.506
0.389
homo_opus+judge
0.512
0.500 †
0.000 †
div_strong+synth
0.179
0.312
0.119
† All pairwise verdicts are ties; independent judges cannot distinguish homogeneous outputs.
Table 2: Original vs. decoupled win rates. Decoupled WR uses an independent 2-judge panel (Gemini 2.0 Flash + GLM-5); 3-judge panel values (including degenerate GPT-4o-mini, 99.6% tie rate) shown for completeness. All directional contrasts survive; Spearman ρ=0.90 (2-judge panel).
Figure 2: BT-corrected consensus win rates for all five experimental cells, averaged over 42 tasks. High performance appears only when diversity and judge-based selection are combined. The synthesis cell (MoA) performs worst, falling well below the single-model baseline.
Cell
Sonnet
GPT-5m
DeepSeek
Mean
κˉ
div_strong+judge
0.958
0.756
0.595
0.770
0.095
homo_opus+judge
0.500
0.518
0.619
0.546
0.667
div_mixed+judge
0.994
0.893
0.631
0.839
0.175
div_strong+vote
0.492
0.496
0.512
0.500
0.236
div_strong+synth
0.234
0.131
0.345
0.237
0.263
Table 3: Per-judge win rates by cell. κˉ is the mean pairwise Cohen’s κ measuring inter-rater agreement.
Figure 3: Per-task diversity advantage ( Δ WR = diverse_strong+judge minus homo_opus+judge ) across all 42 tasks. Positive values favor diversity. 38 of 42 tasks show a positive effect (4 ties, 0 negative; Clopper–Pearson 95% CI: [0.774, 0.973]). Error bars are approximate 95% CIs.
OLS (HC3)
MixedLM
Intercept ( homo_opus+judge )
0.512∗∗∗ (0.008)
0.512∗∗∗ (0.024)
diverse_strong+judge
+0.298∗∗∗ (0.024)
+0.298∗∗∗ (0.035)
diverse_mixed+judge
+0.417∗∗∗ (0.022)
+0.417∗∗∗ (0.035)
diverse_strong+synth
−0.333∗∗∗ (0.029)
−0.333∗∗∗ (0.035)
diverse_strong+vote
−0.016 (0.037)
−0.016 (0.035)
Task var. ( σ^2 )
—
≈0
Table 4: Regression results. Reference category: homo_opus+judge (intercept). All effects stable across specifications; task variance ≈0 . ∗∗∗p<0.001 .
Configuration
Rel. Cost
BT-WR
95% CI
div_mixed+judge †
1.0×
0.929
[0.887, 0.964]
div_strong+judge
2.5×
0.810
[0.768, 0.851]
homo_opus+judge
2.0×
0.512
[0.500, 0.530]
div_strong+vote
1.9×
0.496
[0.425, 0.563]
div_strong+synth
1.9×
0.179
[0.127, 0.234]
Table 5: Relative cost vs. quality across experimental cells ( 1.0× = cheapest cell). diverse_mixed+judge is both the best-performing and cheapest configuration. † Exploratory (not pre-registered).