You're Hired: Strategic Model Selection for LLM Collaboration

Zongwan Cao; Ziyuan Yang; Shangbin Feng; Michael Duan; Skyler Hallinan; Bingbing Wen; Lucy Lu Wang; Yulia Tsvetkov

cs.CLSep 30, 2026

You're Hired: Strategic Model Selection for LLM Collaboration

Authors: Zongwan Cao, Ziyuan Yang, Shangbin Feng, Michael Duan, Skyler Hallinan, Bingbing Wen, Lucy Lu Wang, Yulia Tsvetkov

Organizations: University of Washington · University of Southern California · Allen Institute for AI

Abstract

While multi-agent and model collaboration algorithms gain traction to combine the strengths of diverse Large Language Models (LLMs), existing systems remain bottlenecked on pre-defined and hand-crafted model pools. In this work, we investigate the problem of model selection in multi-LLM systems. We propose and systematically evaluate a taxonomy of 9 selection algorithms ranging from diversity of model descriptions, capability-aware behavioral diversity, and LLM-based recruiters. We conduct extensive experiments across two candidate pools of 10 and 32 models, deployed in four model collaboration algorithms, and evaluated across tasks spanning math, coding, QA, and reasoning. Results demonstrate that successful selection algorithms greatly outperform random or heuristics-based teams such as merely selecting the models with top individual performance, by up to 36.1% across settings. Specifically, capability- and training-based selection strategies alleviate selection variance and achieve the best performance, which we recommend to employ before deploying real-world multi-LLM systems. Further analysis reveals that larger candidate pools pose greater challenges to shallow selection heuristics, while algorithms grounded in interacting with candidate models and understanding model capability robustly filter out misaligned, unsafe models, as well as generalizing to novel, out-of-distribution tasks. Together, we establish that principled and informed team selection is critical and present strong model selection algorithms for assembling effective multi-LLM systems.

Figures & tables

Figure 1: Overview of our work. Given a candidate model pool, we study how to select a team for downstream collaboration. Our selectors consist of standard baselines, stated diversity, capability-aware-diversity, hybrid approaches, and LLM-based recruiters. The selected models are then composed into a team and evaluated across diverse collaboration paradigms.

Selector $\sigma$	Signal $\mathcal{Z}$	Procedure
I. Standard Baselines
Random	None	Random sampling
Top size	Parameter count	Direct ranking
Top solo score	Solo performance	Direct ranking
II. Stated Diversity
Description diversity	Model description	Exact (Alg. 1 )

Table 1: Selection strategy taxonomy.

Standard Baselines
	Prompt Routing			Multiagent Refine			DARE-TIES			LLM-Blender
Selection Method	GSM8K	TQA	MBPP	GSM8K	TQA	MBPP	GSM8K	TQA	MBPP	GSM8K	TQA	MBPP
Single Best Model	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3	65.6 ±1.5	59.3 ±2.0	47.0 ±2.3
Random	$50.5_{\pm 10.0}$	$48.0_{\pm 5.7}$	$51.7_{\pm 6.5}$	$45.4_{\pm 2.6}$	$42.8_{\pm 6.1}$	$42.7_{\pm 4.4}$	$45.2_{\pm 13.4}$	$59.8_{\pm 4.7}$	$52.4_{\pm 7.5}$	$59.8_{\pm 7.1}$	$48.0_{\pm 5.4}$	$53.8_{\pm 7.3}$
Top Solo Score	$62.8_{\pm 1.5}$	$43.9_{\pm 2.0}$	$45.2_{\pm 2.2}$	$60.2_{\pm 1.6}$	$50.2_{\pm 2.0}$	$36.1_{\pm 2.2}$	$31.4_{\pm 1.5}$	$63.9_{\pm 1.9}$	$46.4_{\pm 2.2}$	73.9 ±1.4	$51.9_{\pm 2.0}$	$47.0_{\pm 2.3}$
Stated Diversity

Table 2: Main collaborative performance results across all datasets and mechanisms for Pool 1 (top) and Pool 2 (bottom). The highest mean score in each column per pool is bolded . Cells are colored to highlight the efficacy of adaptive selection based on mean performance: Light Green indicates the team beats the Random baseline, Medium Green indicates the team beats all standard baselines ( Random , Top Solo , and Top Size when applicable), and Dark Green indicates the team beats all standard baselines plus the Single Best Model .

Panel of Figure 2 — Figure 2: Average fraction of clean models in selected teams across contamination levels and randomized injection trials. The red dashed line marks the Random baseline.

Panel of Figure 3 — Figure 3: Pool-size scaling on Pool 2. Expanding the uncurated pool without strict capability constraints causes team performance to saturate or even degrade.

Panel of Figure 4 — Figure 4: Pool diversity scaling on Pool 2. Evaluating a 32-slot pool across varying compositions of $k$ distinct models $\times$ $m$ copies. Naive diversity ( Desc. ) saturates as $k$ increases, while behavioral diversity ( IRT-net ) degrades at $32\times 1$ , risking the inclusion of low-quality outliers.

	CoCoNot		GPQA-Diamond
Selector	Routing	Blender	Routing	Blender
Random	$44.3_{\pm 6.9}$	$26.1_{\pm 9.2}$	$22.4_{\pm 2.3}$	$26.9_{\pm 11.0}$
IRT-Net (most)	$38.6_{\pm 3.0}$	$29.7_{\pm 3.0}$	$19.2_{\pm 7.5}$	$33.3_{\pm 9.0}$
Idio. (most)	$45.3_{\pm 3.0}$	$25.1_{\pm 3.0}$	$17.2_{\pm 7.5}$	$\mathbf{40.4}_{\pm 9.5}$
SFT	$\mathbf{48.6}_{\pm 3.0}$	$\mathbf{33.1}_{\pm 3.0}$	$\mathbf{28.3}_{\pm 9.0}$	$34.3_{\pm 9.0}$

Table 3: OOD evaluation on Pool 2. The highest mean score in each column per pool is bolded.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

		Pool 1		Pool 2
Category	Direction	Max-Min	Average	Max-Min	Average
Stated Diversity	Most	0.621	0.606	0.703	0.697
	Least	0.491	0.488	0.559	0.332
Combined	Most	0.569	0.592	0.336	0.428
	Least	0.584	0.574	0.530	0.209
Nested	Most	0.551	0.580	0.643	0.541

Appendix

Table 4: Max-min vs. average dispersion score comparison. Tasks are evaluated using LLM-Blender on TruthfulQA. Higher scores per setting are in bold .

Dataset	Task type	Dev set size	Test set size
GSM8K ( Cobbe et al., 2021 )	exact match	200	1000
TruthfulQA ( Lin et al., 2022 )	multiple choice	200	617
MBPP ( Austin et al., 2021 )	coding	487	487

Appendix

Table 5: Main evaluation datasets

Dataset	Task type	Note
BBH_tiny ( Suzgun et al., 2023 )	exact match	150-question random sample of BBH
MMLU_Redux_tiny ( Gema et al., 2025 )	multiple choice	150-question random sample of MMLU-Redux
HumanEval ( Chen et al., 2021 )	coding	114-question sample of HumanEval

Appendix

Table 6: Datasets used to train the SFT Classifier.

Dataset	Task type	Size
ARC-Challenge ( Clark et al., 2018 )	multiple choice	1,172
BBH ( Suzgun et al., 2023 )	exact match	1,000
MMLU-Redux ( Gema et al., 2025 )	multiple choice	1,000
GPQA-Diamond ( Rein et al., 2024 )	multiple choice	99
MATH ( Hendrycks et al., 2021 )	exact match	956
MedQA ( Jin et al., 2021 )	multiple choice	637

Appendix

Table 7: Datasets used to build the performance-profile signal.

Dataset	Task type	Size
GPQA-Diamond ( Rein et al., 2024 )	multiple choice	99
CoCoNot ( Brahman et al., 2024 )	refusal judgment	1,000

Appendix

Table 8: Datasets used in the out-of-distribution generalization experiment.

Panel of Figure 5 — Figure 5: Round-by-round robustness of every selection method against the theoretical Random baseline (dashed gray line). The $y$ -axis is the fraction of clean models in the final selected team, and the $x$ -axis shows the number of malicious models in the 10-candidate pool. Methods that fall below the dashed line are actively selecting malicious models at a higher rate than blind chance.

Variant	Specialization
yuru_qw_code_alpaca	Code generation and programming tasks
yuru_qw_cot	Chain-of-thought, step-by-step reasoning
yuru_qw_flan_v2	Diverse instruction-following across task types
yuru_qw_gemini_alpaca	General-purpose instruction-following
yuru_qw_lima	High-quality, concise, well-aligned responses
yuru_qw_oasst1	Open-ended assistant-style conversation

Appendix

Table 9: Pool 1: 10 Qwen2.5-7B checkpoints, differing only in fine-tuning corpus

ID	Original source	Size (B)	Specialization
parti_0	chtmp223/Qwen2.5-7B-CLIPPER	7 (Qwen2.5)	Narrative claim verification against full book text
parti_1	chengq9/ToolRL-Qwen2.5-3B	3 (Qwen2.5)	RL-trained for tool use and parameter filling
parti_2	AgentFlow/agentflow-planner-7b	7 (unspecified)	Online agent planning via Flow-GRPO
parti_3	nanami/ladder-last16L-llama3.1-8b-instruct-sft	8 (Llama-3.1)	Efficient transformer architecture research
parti_4	viswavi/qwen2.5-rlcf	7 (Qwen2.5)	Instruction-following via preference tuning on WildChecklists
parti_5	milli19/promptmii-llama3.1-8b-instruct	8 (Llama-3.1)	RL-trained to induce instructions

Appendix

Table 10: Pool 2: 32 models distilled from independently contributed systems. Base architecture is read from the original repository name where stated; “unspecified” means neither the name nor the source paper states it, and we did not assume one.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,050 papers

Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection
Sep 29, 2026Liangyu Teng, Hengsong Liu, Juncen Guo +4Language Model Ensembles Ensemble Diversity
Fudan Institute on Networking Systems of AI
Mixture of Complementary Agents for Robust LLM Ensemble
May 21, 2026Yichi Zhang, Kevin Lu, Yuang Zhang +3Language Model Ensembles Multi-LLM Collaboration
DIMACS, Rutgers University · Department of Mathematics, Rutgers University · Department of Computer Science, George Mason University +1
When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines
Mar 20, 2026Artem Maryanskyy, Dmitry Budnikov, Alibek T. KaliyevMulti-Agent LLM Systems Multi-LLM Collaboration
Uber Technologies, Sunnyvale, CA 94085, USA · Laboratories of Electrical, Thermal Technologies and Energy Saving, Federal Scientific Agroengineering Center VIM, 109428 Moscow, Russia · Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA

Sep 29, 2026J/K move · Enter open · S save

Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection

Liangyu Teng, Hengsong Liu, Juncen Guo, Jingyu Zhang+3

Fudan Institute on Networking Systems of AI

Language Model Ensembles Ensemble Diversity Multi-LLM Collaboration

Open paper AskPDF arXiv

The performance ceiling of an LLM team is constrained not only by individual model capabilities, but also by inter-member error resonance and predictive differences. Although heterogeneous teaming is often observed to be effective in practice, existing approaches lack complementarity metrics that are computable, interpretable, and optimizable, leaving team composition to rely on heuristics. We propose a heterogeneity-driven team selection framework that performs offline profiling to characterize individual capability along with two complementary signals: one captures decorrelation in error patterns to reduce co-failures, while the other measures divergence in predictive behavior to capture strategy diversity. We formulate team selection as a standardized quality--complementarity combinatorial objective and apply an efficient greedy search to select a small team from a candidate pool. Experiments across multiple benchmarks demonstrate that our framework consistently outperforms quality-only baselines under controlled candidate pools and team sizes, establishing reusable selection principles for multi-LLM systems.

Abstract

Multi-Agent LLM Systems

Explore similar work

Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection

Mixture of Complementary Agents for Robust LLM Ensemble

When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines