Toward Collective Model Intelligence: Characterizing and Coordinating Models

SinapisAI Team; Jian-Dong Liu; Qun Dai; Zhendong Du; Guannan Lai; Hao-Yu Liu; Yi-Fan Ma; Zi-Yu Mao

cs.AIOct 17, 2026

Toward Collective Model Intelligence: Characterizing and Coordinating Models

Authors: SinapisAI Team, Jian-Dong Liu, Qun Dai, Zhendong Du, Guannan Lai, Hao-Yu Liu, Yi-Fan Ma, Zi-Yu Mao,

Organizations: SinapisAI

Abstract

Independently developed AI models accumulate diverse and complementary capabilities, but a growing model collection does not automatically realize Collective AI. Addressing this gap requires understanding model capabilities with limited evaluation, organizing complementary models to collaborate on diverse tasks, and executing model calls efficiently under resource constraints. We present a systematic approach to pursuing Collective Model Intelligence through three connected components. First, Model Eval constructs evidence-based capability profiles by combining query-level performance estimation with cross-domain prediction, reducing the need for exhaustive evaluation. Second, Mio (Models in One) organizes complementary model candidates and supports task-conditioned, cost-aware selection and collaboration as task requirements evolve. Third, Mos (Model Serving OS) separates model selection from execution and provides endpoint scheduling and shared model deployment and distribution. Experiments show lower capability-estimation errors under limited evaluation. Across nine public benchmarks, Mio 1.0 Lite achieves the highest average score and lowest average cost among compared methods, with per-request costs 73.7-97.5% lower than those of the best-scoring alternatives. A prewarmed shared cache accelerates model-file retrieval by up to 23.3 times. This work lays a foundation for Collective Model Intelligence by characterizing and coordinating the diverse capabilities of an expanding model universe.

Figures & tables

Figure 1: Individual models accumulate knowledge and skills across developers and domains. Collective Model Intelligence seeks additional value by characterizing, organizing, and coordinating these capabilities. The illustrated functions provide a foundation within a broader vision that includes growth of the model population and experience-driven collective evolution.

Figure 2: Collective Model Intelligence Framework architecture.

Figure 3: Model organization workflow: register in Model Space, make callable through Mos, characterize through Model Eval, and review and publish the candidate pool through Mio.

State	Supporting record	Consequence
Registered	Identity, source, version and access configuration	Discoverable in Model Space.
Callable	Compatible endpoint and readiness evidence	Available for execution under its access policy.
Characterized	Published, scoped capability evidence	Can support comparisons or capability-based decisions.
Mio candidate	Published pool snapshot and selection support	Included in the pool used for request-level selection.

Table 1: Model resource states and their supporting records at a recorded snapshot.

Figure 4: Model serving workflow: select a model through Mio or the caller, schedule and execute through Mos, and return results. The retry design preserves model identity and residual constraints.

Figure 5: Budgeted capability characterization by Model Eval. Mos query results and a fixed reference snapshot guide profile updates and further queries. Profiles retain evidence and uncertainty.

Figure 6: Model organization, selection and collaboration in Mio. Model Eval profiles and Model Space inform a complementary candidate pool; each decision selects a feasible model for Mos execution and updates task context and resource budgets. Capability-profile refresh follows separate scoring and evidence review by Model Eval.

Figure 7: SLO-aware endpoint scheduling with model identity. High-priority requests bypass batch collection; low-priority requests enter a collection window when all eligible endpoints are busy. Endpoint selection combines capacity checks, SLO-aware screening, and Pareto-based ranking, with sequential failover over a bounded set of ranked candidates when the failure type and response progress permit.

Figure 8: Distributed elastic model cache orchestrated by Fluid.

Figure 9: Quality–cost trade-offs on six benchmarks. Costs use logarithmic axes. Shading marks combinations dominated by Mio; dashed lines connect observed individual-model Pareto points.

Table 11

Figure 10: Limited-query score estimation with shared IRT-selected queries on held-out evaluation models. Lower MAE is better.

Figure 11: Limited-query score estimation with stratified sampling. Lower MAE is better.

Figure 12: Prediction MAE on unmeasured domains using shared exact anchor scores. (a) Measured-domain budget, averaged equally over four historical missingness levels. (b) Historical model–domain missingness at

B=4

measured domains per model. Lower MAE is better.

Scenario	Models	Train	Standard test	Robust test	Robust changes
General Text	63	3,000	3,688	2,894	Semantic rewrite, language variation, long-context noise
Multimodal	45	4,000	4,600	1,900	Text rewrite, context noise, image variation, combined perturbation
Chinese	10	7,590	3,900	2,000	Semantic rewrite, English translation, prefix noise

Table 5: RobustRouteBench candidate pools, training sizes, and final-test populations after excluding all development source groups.

Method	General Text		Multimodal		Chinese		Macro nAUC
	Standard	Robust	Standard	Robust	Standard	Robust
Mio selector	0.5924	0.5328	0.7077	0.6128	0.7653	0.6051	0.6360
MLP	0.5884	0.5231	0.6299	0.5918	0.7656	0.6108	0.6183
EmbedLLM-RouteJudge	0.5863	0.5304	0.5510	0.5869	0.7629	0.6087	0.6044
Avengers-Pro	0.6022	0.2865	0.6239	0.5680	0.7660	0.6054	0.5753
Eagle	0.6045	0.3359	0.6095	0.5264	0.7642	0.5982	0.5731

Table 6: RobustRouteBench nAUC across all 13 implementations, ordered by macro nAUC.

Scenario	Robustness type	$Q_{\mathrm{peak}}$	Rank	nAUC	Rank
General Text	Semantic rewrite	0.5124	1	0.4829	1
	Language variation	0.4913	1	0.4487	2
	Long-context noise	0.6747	1	0.6529	1
Multimodal	Text rewrite	0.6455	1	0.6438	1
	Context noise	0.6219	1	0.6021	1
	Image variation	0.6289	1	0.6247	1

Table 7: Mio selector scores and ranks among 13 implementations for each robustness type. Multimodal metrics are computed after pointwise averaging of the three run curves.

Figure 13: Batch model-loading completion time: direct OSS versus a prewarmed Fluid cache for three models. Solid: Fluid; dashed: OSS. Logarithmic vertical axes; equally spaced concurrency levels.

Figure 14: Mean per-instance model-loading time for three models. Solid: Fluid; dashed: OSS. Logarithmic vertical axes; equally spaced concurrency levels.

Model	Batch mean	Per-instance mean	Batch peak	Per-instance peak
hayai-ocr-v2	$8.73\times$	$9.04\times$	$20.67\times$	$19.90\times$
s1-mini	$9.39\times$	$9.68\times$	$16.70\times$	$16.36\times$
Ornith-1.0-9B	$12.00\times$	$10.84\times$	$22.56\times$	$23.31\times$

Table 8: Mean and peak Fluid speedup over direct OSS access.

Cost-Aware Inference

Cost-aware inference allocates models, computation, and serving resources to meet quality goals at lower latency, energy use, and monetary cost.

See all 282 papers

MoCo: A One-Stop Shop for Model Collaboration Research
Jan 29, 2026Shangbin Feng, Yuyang Bai, Ziyuan Yang +17Multi-LLM Collaboration
University of Washington · Texas A&M University · University of Notre Dame +7
CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion
Jul 6, 2026Adam Fisch, Daniel Deutsch, Joshua Maynez +5Language Model Generation Evaluation Matrix Completion
Google DeepMind · Google Research
AgentBaton: Efficient Model Collaboration in Agentic Tasks
Oct 25, 2026Junlong Li, Jindong Jiang, Yonggan Fu +6LLM Agent Efficiency LLM Agent Orchestration
The Hong Kong University of Science and Technology · Project Leads · University of Illinois Urbana-Champaign

Jan 29, 2026J/K move · Enter open · S save

MoCo: A One-Stop Shop for Model Collaboration Research

Shangbin Feng, Yuyang Bai, Ziyuan Yang, Yike Wang+16

University of Washington · Texas A&M University · University of Notre Dame+7

Multi-LLM Collaboration

Open paper AskPDF arXiv

Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.

Abstract

Cost-Aware Inference

Explore similar work

MoCo: A One-Stop Shop for Model Collaboration Research

CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion

AgentBaton: Efficient Model Collaboration in Agentic Tasks