Organizations: University of Washington · Texas A&M University · University of Notre Dame · University of Virginia · New York University · University of Illinois Urbana-Champaign · Harvard University · Washington University in St. Louis · University of California Irvine · Stanford University
Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.
Figures & tables
Figure 1: MoCo is a comprehensive library for model collaboration research. Download MoCo , write a config file specifying model collaboration setups (models, data, hardware, etc.), execute and compare diverse model collaboration algorithms with MoCo .
Model Pool 1: Specialized LMs
Model Pool 2: General-Purpose LMs
QA
math
reason
safety
code
IF
avg.
QA
math
reason
safety
code
IF
avg.
Best Single
0.588
0.774
0.418
0.586
0.553
1.387
0.549
0.504
0.834
0.413
0.495
0.614
8.402
0.582
Cascade
0.597
0.805
0.387
0.631
0.597
1.385
0.565
0.540
0.831
0.423
0.457
0.632
9.289
0.591
Graph Routing
0.610
0.780
0.387
0.638
0.579
1.676
0.565
0.592
0.836
0.487
0.567
0.790
7.516
0.644
Prompt Routing
0.596
0.803
0.411
0.571
0.597
1.381
0.558
0.579
0.837
0.461
0.557
0.754
10.276
0.649
Switch Generation
0.587
0.733
0.382
0.652
0.412
-1.060
0.493
0.491
0.860
0.481
0.521
0.395
17.234
0.625
Table 1: Performance of API-level , text-level , logit-level , and weight-level model collaboration algorithms over two model pool settings and six evaluation domains. Improvements over the best single model without collaboration are highlighted in orange . IF stands for instruction following: we normalize its scores with min-max standardization to 0-1 and calculate the macro-average across evaluation domains for the “avg.” column. Best in bold and second-best in underline . * indicates that these methods operate with objective tasks: the open-ended generation CocoNot dataset is not included in their performance on the safety domain. / indicates that the collaboration method is not compatible with this model/data setting (e.g., model merging requires models to share the same architecture and only works with model pool 1). Results show that diverse model collaboration approaches improve over individual models in 61.0% (model, evaluation) settings, with text-level and weight-level collaboration algorithms as generally stronger.
Figure 2: Scaling the number of models in model collaboration systems and evaluating on reasoning, QA, and safety domains. We observe a consistent upward trend that further improves over the best single model, with text-level and weight-level methods being more scalable and benefiting from a larger pool of diverse models. This indicates that by scaling up model collaboration, we could build bottom-up compositional AI systems where the components are small but the system is large.
Figure 3: Impact of model pool diversity on collaboration performance. The x-axis shows the configurations of model pool diversity: 1×8,2×4,4×2 and 8×1 . Results demonstrate that model collaboration benefits from increased diversity among participating models, indicating the need for model specialization.
Figure 4: For problems where none of the LLMs could solve individually, what percentage of them are solvable with the model collaboration system, across diverse tasks and collaboration strategies. We observe consistent collaborative emergence across settings with an average of 18.5%, indicating that many model collaboration algorithms do not merely offer a union of existing capabilities: new skills emerge in the collaborative system of multiple models that solve problems where individual models struggle to.
Figure 5: Employing random, prompt-based, or description-based strategies to select 3 models out of 8 for collaboration. Both strategies outperform the random baseline and no collaboration, indicating the importance of model selection strategies and highlighting the need for future research.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
mmlu-redux
gsm8k
bbh
w/o model 1
0.572
0.729
0.539
w/o model 2
0.581
0.759
0.561
w/o model 3
0.589
0.822
0.565
w/o model 4
0.593
0.828
0.588
w/o model 5
0.598
0.872
0.598
avg
0.587
0.802
0.570
Appendix
Table 2: Leave-one-out analysis to study the sensitivity of model collaboration, specifically the multiagent debate approach, to minor changes in model composition. An average standard deviation of 0.030 shows that it is mostly robust.
Dataset
Source
Size
dev
test
AGIeval
( Zhong et al., 2024 )
1156
1156
Alpaca
( Dubois et al., 2023 )
2000
1000
ARC-Challenge
( Clark et al., 2018 )
299
1172
BigBench-hard
( Suzgun et al., 2023 )
1000
1000
BLEND
( Myung et al., 2024 )
1000
1000
Appendix
Table 3: Datasets Statistics.
Figure 6: Collaborative emergence on the General-purpose QA domain.
Figure 7: Collaborative emergence on the Safety domain.
Figure 8: Collaborative emergence on the Coding domain.
Method
Training
Inference
Notes
Cascade
/
2Dm⋅∑i=1n2i−1ki
Each level defer 50%
Graph Router
2nDm⋅max(ki)
2Dm⋅max(ki)
Omit GNN training
Prompt Router
/
2Dm⋅(kr+max(ki))
Router kr
LLM Router
2nDm⋅max(ki)+6rDM⋅kr
2Dm⋅(kr+max(ki))
Router kr
Switch Generate
6rDm⋅ks
2Dm⋅(ks/patch+max(ki))
Switcher ks , rounds r , path size patch
Mentor Collab
2nDm⋅max(ki)
2Dm⋅mean(ki)
Omit MLP training
Appendix
Table 4: Collaboration Methods: Training and Inference FLOPs Complexity Analysis. D denotes the dataset size, m the maximum tokens length and the model pool K={ki}1n contains n models. Each forward pass produces 2 FLOPs per parameter, and each backward pass produces 6 FLOPs per parameter.
Multi-LLM collaboration studies how large language models share work, exchange information, and combine outputs to improve capability, reliability, or cost.
No single Large Language Model (LLM) is uniformly reliable across queries, motivating multi-model inference systems that either route among models or combine their outputs. However, routing stops after selecting an initial model, while dense collaboration invokes peers on every query. We show that collaboration is non-monotonic: peers can recover failures that no model solves alone, but can also corrupt initially correct answers. We introduce COMED (Controlled Model Escalation for Multi-LLM Deliberation), a post-anchor controller for selective cross-model collaboration. COMED uses anchor self-consistency, router margin, and a lightweight peer probe to accept confident answers, verify ambiguous cases, and escalate only when collaboration is likely beneficial. We formalize this trade-off with a rescue-harm decomposition showing that selective collaboration improves when rescued errors outweigh collaboration-induced harms. Across medical, scientific, and general reasoning benchmarks, COMED improves fixed and routed anchors in all 16 open-weight settings, with gains up to +10.7 percentage points on MedQA while invoking fewer models and using fewer decoded tokens than dense collaboration. On HLE with frontier models, COMED improves GPT-5.5 from 23.1% to 28.1%, outperforming dense collaboration and achieving the best results.
Large language models (LLMs) excel across diverse tasks, benefiting from knowledge priors, in-context reasoning, and test-time computation. Despite these strengths, single-model reasoning remains vulnerable to error propagation, as later steps depend on the model's earlier outputs, leaving reasoning blind spots difficult to independently reassess. To address this limitation, we develop collaborative learning, encompassing the construction of heterogeneous collaborators and the learning of effective cross-model collaboration. Specifically, model slicing replaces a large model with smaller, heterogeneous models under a comparable parameter budget, enabling prior reasoning to be reassessed across distinct model parameterizations. Pyramid supervision combines protocol distillation with step-level supervision to refine reasoning and interaction decisions throughout collaboration. We evaluate our approach across four task categories: multi-hop question answering, mathematical reasoning, code generation, and open-ended constrained generation. The trained heterogeneous models achieve comparable or better task performance than the original large model and established multi-agent baselines, including reinforcement-learning-based methods, while also achieving higher inference efficiency than the multi-agent baselines and exhibiting differentiated collaboration patterns across tasks. These findings show that learned collaboration turns model heterogeneity into complementary reasoning capabilities.
Yuheng Wang, Runde Yang, Zhuohan Bao +9
School of Artificial Intelligence, Shanghai Jiao Tong University
Personal intelligence requires strong capabilities under strict privacy and efficiency constraints. Unlike cloud-first intelligence, which relies on powerful but costly and privacy-vulnerable large language models (LLMs), we focus on local-first intelligence, emphasizing on-device small language models (SLMs), which enable efficient, private inference but have limited capacity. To reconcile the capability gap with privacy and efficiency, we propose a dynamic collaboration framework, where an SLM learns to proactively decide how to consult an LLM during multi-step reasoning, while the LLM provides adaptive feedback instead of acting as a passive tool, modeling them as heterogeneous agents under information asymmetry. We further systematically investigate how collaboration strategies are shaped by SLM and LLM capabilities as well as efficiency and privacy constraints. Evaluation results reveal a distinct scaling effect: stronger SLMs become more self-reliant, while stronger LLMs enable fewer and more informative interactions. In addition, the learned dynamic collaboration strategies significantly outperform various baselines and transfer robustly to unseen LLMs as well as to real-world personal tasks without additional training.
Hang Zeng, Xiangyu Liu, Yong Hu +5
Shanghai Jiao Tong University, Shanghai, China · WeChat Tencent, Beijing, China · State University of New York at Buffalo, New York, United States