Competence-Aware Orchestration for LLM-Based Multi-Agent Systems

Songmiao Wang; Yuchen Wang; Chenglin Yu; Hongxia Yang; Ming Li

cs.AIOct 9, 2026

Competence-Aware Orchestration for LLM-Based Multi-Agent Systems

Authors: Songmiao Wang, Yuchen Wang, Chenglin Yu, Hongxia Yang, Ming Li

Organizations: The Hong Kong Polytechnic University

Abstract

LLM-based multi-agent systems (MAS) have emerged as a promising paradigm for enhancing complex reasoning through role specialization and collaboration, yet a key challenge remains: how can collaboration be adapted to a backbone's competence on the current query? This problem matters because more collaboration is not always better--richer coordination may help on difficult cases, but it also incurs additional token cost and redundant interactions. We propose a two-stage competence-aware framework for automatic MAS orchestration. It first models the LLM backbone's task-conditioned competence for the current query, and then uses this signal to adapt role activation and the communication graph. Across six benchmarks, our method achieves an average score of 93.81%. Experiments with six LLM backbones of varying competence levels further show that our method adjusts collaboration according to the inferred competence: for higher-competence backbones, it improves average accuracy from 87.22 to 89.97 while reducing token cost by 36.26%; for lower-competence backbones, it yields a further average gain of 4.58 points over existing methods. The same trend also holds on unseen backbones.

Figures & tables

Figure 1: Comparison between existing MAS construction methods and our method.

Figure 2: Overview of the proposed competence-aware orchestration framework. Stage I learns a competence representation from the backbone’s capability fingerprint and the current query under success-rate supervision. Stage II uses this representation to guide role activation and communication graph generation.

Method	MMLU	GSM8K	AQuA	MultiArith	SVAMP	HumanEval	Average
Vanilla	80.30	82.43	71.16	92.77	86.32	71.56	80.76
CoT	81.64 $\uparrow$ 1.34	86.81 $\uparrow$ 4.38	73.21 $\uparrow$ 2.05	93.12 $\uparrow$ 0.35	87.33 $\uparrow$ 1.01	74.75 $\uparrow$ 3.19	82.81 $\uparrow$ 2.05
SC (CoT)	83.30 $\uparrow$ 3.00	81.86 $\downarrow$ 0.57	75.79 $\uparrow$ 4.63	94.30 $\uparrow$ 1.53	88.44 $\uparrow$ 2.12	79.87 $\uparrow$ 8.31	83.93 $\uparrow$ 3.17
Chain	82.66 $\uparrow$ 2.36	88.28 $\uparrow$ 5.85	74.38 $\uparrow$ 3.22	93.42 $\uparrow$ 0.65	86.95 $\uparrow$ 0.63	81.57 $\uparrow$ 10.01	84.54 $\uparrow$ 3.78
Tree	80.93 $\uparrow$ 0.63	85.28 $\uparrow$ 2.85	71.59 $\uparrow$ 0.43	93.54 $\uparrow$ 0.77	89.09 $\uparrow$ 2.77	80.74 $\uparrow$ 9.18	83.53 $\uparrow$ 2.77
Complete	82.59 $\uparrow$ 2.29	79.76 $\downarrow$ 2.67	72.86 $\uparrow$ 1.70	94.82 $\uparrow$ 2.05	84.28 $\downarrow$ 2.04	79.07 $\uparrow$ 7.51	82.23 $\uparrow$ 1.47

Table 1: Performance comparison (%) on six benchmarks.

Setting	HumanEval	AQuA	MMLU	Avg.	Tokens / Task
Full	92.56	87.14	92.81	90.84	2360.45
w/o task	90.91	85.71	88.24	88.29 $\downarrow$ 2.55	2758.02 $\uparrow$ 16.84%
w/o model	87.60	87.62	88.24	87.82 $\downarrow$ 3.02	6010.64 $\uparrow$ 154.64%
w/ desc.	88.43	85.24	89.54	87.74 $\downarrow$ 3.10	4736.68 $\uparrow$ 100.67%

Table 2: Downstream ablation of competence modeling.

Figure 3: Number of agents in the generated MAS on efficiency-oriented target backbones.

Method	GLM 5.1 ⋆	Gemini 3.1 Pro ⋆	GPT-4o	Avg.
Vanilla	83.01	81.70	80.30	81.67
G-Designer	88.89	85.62	87.14	87.22
Ours	89.53	87.58	92.81	89.97 $\downarrow$ 36.26%

Table 3: Results on MMLU with performance-oriented target backbones.

Method	Qwen3-14B	Llama3.1-8B	Gemma-3-12B	Avg.
Vanilla	77.79	63.40	66.67	69.29
G-Designer	84.31	67.32	70.58	74.07
Ours	87.58	70.59	77.78	78.65

Table 4: Results on MMLU with efficiency-oriented target backbones.

Figure 4: Case studies on MMLU. Left: changing the task while fixing the backbone (Llama-3.1-8B). Right: changing the backbone from Llama-3.1-8B to GPT-4o while fixing the task. check/cross marks indicate answer correctness.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Source datasets
code_logic	HumanEval
quantitative_reasoning	AQuA
basic_math	GSM8K, MultiArith, SVAMP
general_knowledge	MMLU

Appendix

Table 5: Domain assignment used in probe-set construction.

Dataset	Answer Type	Metric	#Test	License
MMLU	Multi-choice	Acc.	153	MIT License
GSM8K	Number	Acc.	1,319	MIT License
MultiArith	Number	Acc.	600	Unspecified
SVAMP	Number	Acc.	1,000	MIT License
AQuA	Multi-choice	Acc.	254	Apache-2.0
HumanEval	Code	Pass@1	164	MIT License

Appendix

Table 6: Dataset descriptions and statistics.

Family	Backbones
Phi	Phi-3.5-mini-instruct
Qwen2.5	Qwen2.5-Coder-7B-Instruct , Qwen2.5-Math-7B-Instruct
DeepSeek	deepseek-v3.2
Gemma	gemma-3-12b-it
GPT	gpt-4o-mini , gpt-5
Llama	llama-3.1-8b-instruct , llama-3.2-1b-instruct , llama-3.2-3b-instruct

Appendix

Table 7: LLM backbones used for proxy-supervision data construction.

Backbone	Total Tokens	Avg. Tokens / Probe
Gemini 3.1 Pro	28,576	595.3
GLM 5.1	78,716	1640.0
GPT-4o	12,228	254.7
Average	39,840	830.0

Appendix

Table 8: One-time token cost of Capability Fingerprinting for the three performance-oriented backbones in Section 4.4 .

Explore similar work

CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments
Jun 4, 2026Jiaju Chen, Bo Sun, Yuxuan Lu +3Multi-Agent LLM Systems Multi-Agent Coordination
Northeastern University · Microsoft Research Asia
CONCAT: Consensus- and Confidence-Driven Ad Hoc Teaming for Efficient LLM-Based Multi-Agent Systems
May 28, 2026Ziyang Ma, Dingyi Zhang, Sichu Liang +4Multi-Agent System Optimization Multi-Agent Orchestration
Southeast University · Huawei Technologies Ltd
MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
Oct 3, 2026Qizhi Chu, Zekai Yu, Sijie Wen +5Multi-Agent LLM Systems Multi-Agent Collaboration
Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University

Jun 4, 2026, cs.CLJ/K move · Enter open · S save

CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments

Jiaju Chen, Bo Sun, Yuxuan Lu, Yun Wang+2

Northeastern University · Microsoft Research Asia

Multi-Agent LLM Systems Multi-Agent Coordination LLM Agent Evaluation Multi-Agent Collaboration

Open paper AskPDF arXiv

Multi-agent systems (MAS) built on large language models have shown growing promise, with their effectiveness resting on agents' ability to coordinate through text-based channels much as human teams do. Yet recent study suggests that MAS often falter not because agents lack individual task-solving ability, but because they lack collaborative competence: the capacity to establish common ground, maintain shared task understanding, balance individual and collective incentives, and repair misalignment as interaction unfolds. Decades of research in Computer-Supported Cooperative Work have characterized these requirements for human teams coordinating under constrained communication, yet existing MAS evaluations focus mainly on task outcomes or single-agent proficiency in reasoning, planning, and tool use. To enable a systematic analysis of agents' collaborative competence in MAS, we introduce CollabSim, a configurable simulation framework that combines a theory-grounded definition of collaborative capabilities, controlled manipulation of interaction conditions, and action-level probing of agents' internal states. Experiments across four LLMs show that CollabSim can capture condition effects, separate model performance patterns, and reveal task-dependent effects of agent design.

Abstract

Explore similar work

CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments

CONCAT: Consensus- and Confidence-Driven Ad Hoc Teaming for Efficient LLM-Based Multi-Agent Systems

MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability