You're Hired: Strategic Model Selection for LLM Collaboration

Zongwan Cao; Ziyuan Yang; Shangbin Feng; Michael Duan; Skyler Hallinan; Bingbing Wen; Lucy Lu Wang; Yulia Tsvetkov

cs.CLSep 30, 2026

You're Hired: Strategic Model Selection for LLM Collaboration

Authors: Zongwan Cao, Ziyuan Yang, Shangbin Feng, Michael Duan, Skyler Hallinan, Bingbing Wen, Lucy Lu Wang, Yulia Tsvetkov

Organizations: University of Washington · University of Southern California · Allen Institute for AI

Abstract

While multi-agent and model collaboration algorithms gain traction to combine the strengths of diverse Large Language Models (LLMs), existing systems remain bottlenecked on pre-defined and hand-crafted model pools. In this work, we investigate the problem of model selection in multi-LLM systems. We propose and systematically evaluate a taxonomy of 9 selection algorithms ranging from diversity of model descriptions, capability-aware behavioral diversity, and LLM-based recruiters. We conduct extensive experiments across two candidate pools of 10 and 32 models, deployed in four model collaboration algorithms, and evaluated across tasks spanning math, coding, QA, and reasoning. Results demonstrate that successful selection algorithms greatly outperform random or heuristics-based teams such as merely selecting the models with top individual performance, by up to 36.1% across settings. Specifically, capability- and training-based selection strategies alleviate selection variance and achieve the best performance, which we recommend to employ before deploying real-world multi-LLM systems. Further analysis reveals that larger candidate pools pose greater challenges to shallow selection heuristics, while algorithms grounded in interacting with candidate models and understanding model capability robustly filter out misaligned, unsafe models, as well as generalizing to novel, out-of-distribution tasks. Together, we establish that principled and informed team selection is critical and present strong model selection algorithms for assembling effective multi-LLM systems.

Figures & tables

Figure 1: Overview of our work. Given a candidate model pool, we study how to select a team for downstream collaboration. Our selectors consist of standard baselines, stated diversity, capability-aware-diversity, hybrid approaches, and LLM-based recruiters. The selected models are then composed into a team and evaluated across diverse collaboration paradigms.

Selector $\sigma$	Signal $\mathcal{Z}$	Procedure
I. Standard Baselines
Random	None	Random sampling
Top size	Parameter count	Direct ranking
Top solo score	Solo performance	Direct ranking
II. Stated Diversity
Description diversity	Model description	Exact (Alg. 1 )

Table 1: Selection strategy taxonomy.

Standard Baselines
	Prompt Routing			Multiagent Refine			DARE-TIES			LLM-Blender
Selection Method	GSM8K	TQA	MBPP	GSM8K	TQA	MBPP	GSM8K	TQA	MBPP	GSM8K	TQA	MBPP
Single Best Model	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3
Random	$50.5_{\pm 10.0}$	$48.0_{\pm 5.7}$	$51.7_{\pm 6.5}$	$45.4_{\pm 2.6}$	$42.8_{\pm 6.1}$	$42.7_{\pm 4.4}$	$45.2_{\pm 13.4}$	$59.8_{\pm 4.7}$	$52.4_{\pm 7.5}$	$59.8_{\pm 7.1}$	$48.0_{\pm 5.4}$	$53.8_{\pm 7.3}$
Top Solo Score	$62.8_{\pm 1.5}$	$43.9_{\pm 2.0}$	$45.2_{\pm 2.2}$	$60.2_{\pm 1.6}$	$50.2_{\pm 2.0}$	$36.1_{\pm 2.2}$	$31.4_{\pm 1.5}$	$63.9_{\pm 1.9}$	$46.4_{\pm 2.2}$	73.9 ±1.4	$51.9_{\pm 2.0}$	$47.0_{\pm 2.3}$
Stated Diversity

Table 2: Main collaborative performance results across all datasets and mechanisms for Pool 1 (top) and Pool 2 (bottom). The highest mean score in each column per pool is bolded . Cells are colored to highlight the efficacy of adaptive selection based on mean performance: Light Green indicates the team beats the Random baseline, Medium Green indicates the team beats all standard baselines ( Random , Top Solo , and Top Size when applicable), and Dark Green indicates the team beats all standard baselines plus the Single Best Model .

Panel of Figure 2 — Figure 2: Average fraction of clean models in selected teams across contamination levels and randomized injection trials. The red dashed line marks the Random baseline.

Panel of Figure 3 — Figure 3: Pool-size scaling on Pool 2. Expanding the uncurated pool without strict capability constraints causes team performance to saturate or even degrade.

Panel of Figure 4 — Figure 4: Pool diversity scaling on Pool 2. Evaluating a 32-slot pool across varying compositions of $k$ distinct models $\times$ $m$ copies. Naive diversity ( Desc. ) saturates as $k$ increases, while behavioral diversity ( IRT-net ) degrades at $32\times 1$ , risking the inclusion of low-quality outliers.

	CoCoNot		GPQA-Diamond
Selector	Routing	Blender	Routing	Blender
Random	$44.3_{\pm 6.9}$	$26.1_{\pm 9.2}$	$22.4_{\pm 2.3}$	$26.9_{\pm 11.0}$
IRT-Net (most)	$38.6_{\pm 3.0}$	$29.7_{\pm 3.0}$	$19.2_{\pm 7.5}$	$33.3_{\pm 9.0}$
Idio. (most)	$45.3_{\pm 3.0}$	$25.1_{\pm 3.0}$	$17.2_{\pm 7.5}$	$\mathbf{40.4}_{\pm 9.5}$
SFT	$\mathbf{48.6}_{\pm 3.0}$	$\mathbf{33.1}_{\pm 3.0}$	$\mathbf{28.3}_{\pm 9.0}$	$34.3_{\pm 9.0}$

Table 3: OOD evaluation on Pool 2. The highest mean score in each column per pool is bolded.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

		Pool 1		Pool 2
Category	Direction	Max-Min	Average	Max-Min	Average
Stated Diversity	Most	0.621	0.606	0.703	0.697
	Least	0.491	0.488	0.559	0.332
Combined	Most	0.569	0.592	0.336	0.428
	Least	0.584	0.574	0.530	0.209
Nested	Most	0.551	0.580	0.643	0.541

Appendix

Table 4: Max-min vs. average dispersion score comparison. Tasks are evaluated using LLM-Blender on TruthfulQA. Higher scores per setting are in bold .

Dataset	Task type	Dev set size	Test set size
GSM8K ( Cobbe et al., 2021 )	exact match	200	1000
TruthfulQA ( Lin et al., 2022 )	multiple choice	200	617
MBPP ( Austin et al., 2021 )	coding	487	487

Appendix

Table 5: Main evaluation datasets

Dataset	Task type	Note
BBH_tiny ( Suzgun et al., 2023 )	exact match	150-question random sample of BBH
MMLU_Redux_tiny ( Gema et al., 2025 )	multiple choice	150-question random sample of MMLU-Redux
HumanEval ( Chen et al., 2021 )	coding	114-question sample of HumanEval

Appendix

Table 6: Datasets used to train the SFT Classifier.

Dataset	Task type	Size
ARC-Challenge ( Clark et al., 2018 )	multiple choice	1,172
BBH ( Suzgun et al., 2023 )	exact match	1,000
MMLU-Redux ( Gema et al., 2025 )	multiple choice	1,000
GPQA-Diamond ( Rein et al., 2024 )	multiple choice	99
MATH ( Hendrycks et al., 2021 )	exact match	956
MedQA ( Jin et al., 2021 )	multiple choice	637

Appendix

Table 7: Datasets used to build the performance-profile signal.

Dataset	Task type	Size
GPQA-Diamond ( Rein et al., 2024 )	multiple choice	99
CoCoNot ( Brahman et al., 2024 )	refusal judgment	1,000

Appendix

Table 8: Datasets used in the out-of-distribution generalization experiment.

Panel of Figure 5 — Figure 5: Round-by-round robustness of every selection method against the theoretical Random baseline (dashed gray line). The $y$ -axis is the fraction of clean models in the final selected team, and the $x$ -axis shows the number of malicious models in the 10-candidate pool. Methods that fall below the dashed line are actively selecting malicious models at a higher rate than blind chance.

Variant	Specialization
yuru_qw_code_alpaca	Code generation and programming tasks
yuru_qw_cot	Chain-of-thought, step-by-step reasoning
yuru_qw_flan_v2	Diverse instruction-following across task types
yuru_qw_gemini_alpaca	General-purpose instruction-following
yuru_qw_lima	High-quality, concise, well-aligned responses
yuru_qw_oasst1	Open-ended assistant-style conversation

Appendix

Table 9: Pool 1: 10 Qwen2.5-7B checkpoints, differing only in fine-tuning corpus

ID	Original source	Size (B)	Specialization
parti_0	chtmp223/Qwen2.5-7B-CLIPPER	7 (Qwen2.5)	Narrative claim verification against full book text
parti_1	chengq9/ToolRL-Qwen2.5-3B	3 (Qwen2.5)	RL-trained for tool use and parameter filling
parti_2	AgentFlow/agentflow-planner-7b	7 (unspecified)	Online agent planning via Flow-GRPO
parti_3	nanami/ladder-last16L-llama3.1-8b-instruct-sft	8 (Llama-3.1)	Efficient transformer architecture research
parti_4	viswavi/qwen2.5-rlcf	7 (Qwen2.5)	Instruction-following via preference tuning on WildChecklists
parti_5	milli19/promptmii-llama3.1-8b-instruct	8 (Llama-3.1)	RL-trained to induce instructions

Appendix

Table 10: Pool 2: 32 models distilled from independently contributed systems. Base architecture is read from the original repository name where stated; “unspecified” means neither the name nor the source paper states it, and we did not assume one.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,050 papers

Sep 29, 2026

Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection

The performance ceiling of an LLM team is constrained not only by individual model capabilities, but also by inter-member error resonance and predictive differences. Although heterogeneous teaming is often observed to be effective in practice, existing approaches lack complementarity metrics that are computable, interpretable, and optimizable, leaving team composition to rely on heuristics. We propose a heterogeneity-driven team selection framework that performs offline profiling to characterize individual capability along with two complementary signals: one captures decorrelation in error patterns to reduce co-failures, while the other measures divergence in predictive behavior to capture strategy diversity. We formulate team selection as a standardized quality--complementarity combinatorial objective and apply an efficient greedy search to select a small team from a candidate pool. Experiments across multiple benchmarks demonstrate that our framework consistently outperforms quality-only baselines under controlled candidate pools and team sizes, establishing reusable selection principles for multi-LLM systems.

Liangyu Teng, Hengsong Liu, Juncen Guo +4

Fudan Institute on Networking Systems of AI

Language Model Ensembles Ensemble Diversity Multi-LLM Collaboration

Ask PDF

May 21, 2026

Mixture of Complementary Agents for Robust LLM Ensemble

Multi-AI collaboration, such as ensembling or debating large language models (LLMs), is a promising paradigm for aggregating information and boosting performance. A foundational step in these pipelines is to feed the responses of several proposer LLMs into a summarizer LLM, which synthesizes a better answer. However, choosing which proposers to include is non-trivial. Existing approaches primarily focus either on accuracy (picking the strongest models) or diversity (ensuring variety), and often overlook the interactions among proposers and with the summarizer. We reframe proposer selection as a combinatorial selection problem akin to feature selection, where the value of an LLM lies in its complementarity with others. However, directly applying standard feature-selection algorithms is impractical in the LLM setting due to prohibitive time complexity. Motivated by this limitation, we explore an extensive range of computationally feasible, greedy-style selection algorithms that assess complementarity using a small labeled set. Our experiments validate complementarity as a guiding principle for proposer selection and identify methods that achieve the best performance-cost trade-offs in practice.

Yichi Zhang, Kevin Lu, Yuang Zhang +3

DIMACS, Rutgers University · Department of Mathematics, Rutgers University · Department of Computer Science, George Mason University +1

Language Model Ensembles Multi-LLM Collaboration

Ask PDF

Mar 20, 2026

When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines

Multi-agent LLM pipelines produce contradictory evidence on whether team diversity improves output quality: heterogeneous Mixture-of-Agents teams outperform single models, yet homogeneous Self-MoA teams consistently win under synthesis-based aggregation. We propose a resolution by identifying the selection bottleneck -- a crossover threshold in aggregation quality that determines whether diversity helps or hurts. Under this model, we obtain a closed-form crossover threshold

s^*

(Proposition 1) that separates the regimes where diversity helps and hurts. In a targeted experiment spanning 42 tasks across 7 categories (

N=210

), a diverse team with judge-based selection achieves a win rate of 0.810 against a single-model baseline, while a homogeneous team scores 0.512 -- near chance (Glass's

Δ= 2.07

). Judge-based selection outperforms MoA-style synthesis by

Δ_{\mathrm{WR}} = +0.631

-- the synthesis approach is preferred over the baseline in zero of 42 tasks by the judge panel. A decoupled evaluation with independent judges confirms all directional findings (Spearman

ρ= 0.90

). Exploratory evidence suggests that including a weaker model improves performance while reducing cost (

p < 10^{-4}

, not pre-registered). Our results suggest that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines.

Artem Maryanskyy, Dmitry Budnikov, Alibek T. Kaliyev

Uber Technologies, Sunnyvale, CA 94085, USA · Laboratories of Electrical, Thermal Technologies and Energy Saving, Federal Scientific Agroengineering Center VIM, 109428 Moscow, Russia · Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA

Multi-Agent LLM Systems Multi-LLM Collaboration Ensemble Diversity Ensemble Learning LLM Agent Evaluation

Ask PDF

Abstract

Multi-Agent LLM Systems

Explore similar work

Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection

Mixture of Complementary Agents for Robust LLM Ensemble

When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines