MoCo: A One-Stop Shop for Model Collaboration Research

Shangbin Feng; Yuyang Bai; Ziyuan Yang; Yike Wang; Zhaoxuan Tan; Jiajie Yan; Zhenyu Lei; Wenxuan Ding

cs.CLJan 29, 2026

MoCo: A One-Stop Shop for Model Collaboration Research

Authors: Shangbin Feng, Yuyang Bai, Ziyuan Yang, Yike Wang, Zhaoxuan Tan, Jiajie Yan, Zhenyu Lei, Wenxuan Ding,

Organizations: University of Washington · Texas A&M University · University of Notre Dame · University of Virginia ·

Abstract

Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.

Figures & tables

Figure 1: MoCo is a comprehensive library for model collaboration research. Download MoCo , write a config file specifying model collaboration setups (models, data, hardware, etc.), execute and compare diverse model collaboration algorithms with MoCo .

	Model Pool 1: Specialized LMs							Model Pool 2: General-Purpose LMs
	QA	math	reason	safety	code	IF	avg.	QA	math	reason	safety	code	IF	avg.
Best Single	0.588	0.774	0.418	0.586	0.553	1.387	0.549	0.504	0.834	0.413	0.495	0.614	8.402	0.582
Cascade	0.597	0.805	0.387	0.631	0.597	1.385	0.565	0.540	0.831	0.423	0.457	0.632	9.289	0.591
Graph Routing	0.610	0.780	0.387	0.638	0.579	1.676	0.565	0.592	0.836	0.487	0.567	0.790	7.516	0.644
Prompt Routing	0.596	0.803	0.411	0.571	0.597	1.381	0.558	0.579	0.837	0.461	0.557	0.754	10.276	0.649
Switch Generation	0.587	0.733	0.382	0.652	0.412	-1.060	0.493	0.491	0.860	0.481	0.521	0.395	17.234	0.625

Table 1: Performance of API-level , text-level , logit-level , and weight-level model collaboration algorithms over two model pool settings and six evaluation domains. Improvements over the best single model without collaboration are highlighted in orange . IF stands for instruction following: we normalize its scores with min-max standardization to 0-1 and calculate the macro-average across evaluation domains for the “avg.” column. Best in bold and second-best in underline . * indicates that these methods operate with objective tasks: the open-ended generation CocoNot dataset is not included in their performance on the safety domain. / indicates that the collaboration method is not compatible with this model/data setting (e.g., model merging requires models to share the same architecture and only works with model pool 1). Results show that diverse model collaboration approaches improve over individual models in 61.0% (model, evaluation) settings, with text-level and weight-level collaboration algorithms as generally stronger.

Figure 2: Scaling the number of models in model collaboration systems and evaluating on reasoning, QA, and safety domains. We observe a consistent upward trend that further improves over the best single model, with text-level and weight-level methods being more scalable and benefiting from a larger pool of diverse models. This indicates that by scaling up model collaboration, we could build bottom-up compositional AI systems where the components are small but the system is large.

Figure 3: Impact of model pool diversity on collaboration performance. The x-axis shows the configurations of model pool diversity:

1\times 8,2\times 4,4\times 2

and

8\times 1

. Results demonstrate that model collaboration benefits from increased diversity among participating models, indicating the need for model specialization.

Figure 4: For problems where none of the LLMs could solve individually, what percentage of them are solvable with the model collaboration system, across diverse tasks and collaboration strategies. We observe consistent collaborative emergence across settings with an average of 18.5%, indicating that many model collaboration algorithms do not merely offer a union of existing capabilities: new skills emerge in the collaborative system of multiple models that solve problems where individual models struggle to.

Figure 5: Employing random, prompt-based, or description-based strategies to select 3 models out of 8 for collaboration. Both strategies outperform the random baseline and no collaboration, indicating the importance of model selection strategies and highlighting the need for future research.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	mmlu-redux	gsm8k	bbh
w/o model 1	0.572	0.729	0.539
w/o model 2	0.581	0.759	0.561
w/o model 3	0.589	0.822	0.565
w/o model 4	0.593	0.828	0.588
w/o model 5	0.598	0.872	0.598
avg	0.587	0.802	0.570

Appendix

Table 2: Leave-one-out analysis to study the sensitivity of model collaboration, specifically the multiagent debate approach, to minor changes in model composition. An average standard deviation of 0.030 shows that it is mostly robust.

Dataset	Source	Size
Dataset	Source	dev	test
AGIeval	( Zhong et al., 2024 )	1156	1156
Alpaca	( Dubois et al., 2023 )	2000	1000
ARC-Challenge	( Clark et al., 2018 )	299	1172
BigBench-hard	( Suzgun et al., 2023 )	1000	1000
BLEND	( Myung et al., 2024 )	1000	1000

Appendix

Table 3: Datasets Statistics.

Figure 6: Collaborative emergence on the General-purpose QA domain.

Figure 7: Collaborative emergence on the Safety domain.

Figure 8: Collaborative emergence on the Coding domain.

Method	Training	Inference	Notes
Cascade	/	$2Dm\cdot\sum_{i=1}^{n}\frac{k_{i}}{2^{i-1}}$	Each level defer 50%
Graph Router	$2nDm\cdot\max(k_{i})$	$2Dm\cdot\max(k_{i})$	Omit GNN training
Prompt Router	/	$2Dm\cdot(k_{r}+\max(k_{i}))$	Router $k_{r}$
LLM Router	$2nDm\cdot\max(k_{i})+6rDM\cdot k_{r}$	$2Dm\cdot(k_{r}+\max(k_{i}))$	Router $k_{r}$
Switch Generate	$6rDm\cdot k_{s}$	$2Dm\cdot\bigl(k_{s}/\textit{patch}+\max(k_{i})\bigr)$	Switcher $k_{s}$ , rounds $r$ , path size patch
Mentor Collab	$2nDm\cdot\max(k_{i})$	$2Dm\cdot\mathrm{mean}(k_{i})$	Omit MLP training

Appendix

Table 4: Collaboration Methods: Training and Inference FLOPs Complexity Analysis.

D

denotes the dataset size,

m

the maximum tokens length and the model pool

\mathcal{K}=\{k_{i}\}^{n}_{1}

contains

n

models. Each forward pass produces 2 FLOPs per parameter, and each backward pass produces 6 FLOPs per parameter.

Multi-LLM Collaboration

Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.

See all 79 papers

COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference
Sep 22, 2026Norah Alballa, Wenxuan Zhang, Salma Kharrat +4LLM Inference Efficiency LLM Routing
KAUST · LUMS
Collaborative Learning and Reasoning over Model Slices
Oct 9, 2026Yuheng Wang, Runde Yang, Zhuohan Bao +9Multi-LLM Collaboration Multi-Agent Reasoning
School of Artificial Intelligence, Shanghai Jiao Tong University
Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM
Apr 20, 2026Hang Zeng, Xiangyu Liu, Yong Hu +5LLM Inference Efficiency Small Language Models
Shanghai Jiao Tong University, Shanghai, China · WeChat Tencent, Beijing, China · State University of New York at Buffalo, New York, United States

Sep 22, 2026J/K move · Enter open · S save

COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

Norah Alballa, Wenxuan Zhang, Salma Kharrat, Fares Fourati+3

KAUST · LUMS

LLM Inference Efficiency LLM Routing Multi-LLM Collaboration

Open paper AskPDF arXiv

No single Large Language Model (LLM) is uniformly reliable across queries, motivating multi-model inference systems that either route among models or combine their outputs. However, routing stops after selecting an initial model, while dense collaboration invokes peers on every query. We show that collaboration is non-monotonic: peers can recover failures that no model solves alone, but can also corrupt initially correct answers. We introduce COMED (Controlled Model Escalation for Multi-LLM Deliberation), a post-anchor controller for selective cross-model collaboration. COMED uses anchor self-consistency, router margin, and a lightweight peer probe to accept confident answers, verify ambiguous cases, and escalate only when collaboration is likely beneficial. We formalize this trade-off with a rescue-harm decomposition showing that selective collaboration improves when rescued errors outweigh collaboration-induced harms. Across medical, scientific, and general reasoning benchmarks, COMED improves fixed and routed anchors in all 16 open-weight settings, with gains up to +10.7 percentage points on MedQA while invoking fewer models and using fewer decoded tokens than dense collaboration. On HLE with frontier models, COMED improves GPT-5.5 from 23.1% to 28.1%, outperforming dense collaboration and achieving the best results.

Abstract

Multi-LLM Collaboration

Explore similar work

COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

Collaborative Learning and Reasoning over Model Slices

Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM