LLM-based multi-agent systems (MAS) have emerged as a promising paradigm for enhancing complex reasoning through role specialization and collaboration, yet a key challenge remains: how can collaboration be adapted to a backbone's competence on the current query? This problem matters because more collaboration is not always better--richer coordination may help on difficult cases, but it also incurs additional token cost and redundant interactions. We propose a two-stage competence-aware framework for automatic MAS orchestration. It first models the LLM backbone's task-conditioned competence for the current query, and then uses this signal to adapt role activation and the communication graph. Across six benchmarks, our method achieves an average score of 93.81%. Experiments with six LLM backbones of varying competence levels further show that our method adjusts collaboration according to the inferred competence: for higher-competence backbones, it improves average accuracy from 87.22 to 89.97 while reducing token cost by 36.26%; for lower-competence backbones, it yields a further average gain of 4.58 points over existing methods. The same trend also holds on unseen backbones.
Figures & tables
Figure 1: Comparison between existing MAS construction methods and our method.
Figure 2: Overview of the proposed competence-aware orchestration framework. Stage I learns a competence representation from the backbone’s capability fingerprint and the current query under success-rate supervision. Stage II uses this representation to guide role activation and communication graph generation.
Method
MMLU
GSM8K
AQuA
MultiArith
SVAMP
HumanEval
Average
Vanilla
80.30
82.43
71.16
92.77
86.32
71.56
80.76
CoT
81.64 ↑ 1.34
86.81 ↑ 4.38
73.21 ↑ 2.05
93.12 ↑ 0.35
87.33 ↑ 1.01
74.75 ↑ 3.19
82.81 ↑ 2.05
SC (CoT)
83.30 ↑ 3.00
81.86 ↓ 0.57
75.79 ↑ 4.63
94.30 ↑ 1.53
88.44 ↑ 2.12
79.87 ↑ 8.31
83.93 ↑ 3.17
Chain
82.66 ↑ 2.36
88.28 ↑ 5.85
74.38 ↑ 3.22
93.42 ↑ 0.65
86.95 ↑ 0.63
81.57 ↑ 10.01
84.54 ↑ 3.78
Tree
80.93 ↑ 0.63
85.28 ↑ 2.85
71.59 ↑ 0.43
93.54 ↑ 0.77
89.09 ↑ 2.77
80.74 ↑ 9.18
83.53 ↑ 2.77
Complete
82.59 ↑ 2.29
79.76 ↓ 2.67
72.86 ↑ 1.70
94.82 ↑ 2.05
84.28 ↓ 2.04
79.07 ↑ 7.51
82.23 ↑ 1.47
Table 1: Performance comparison (%) on six benchmarks.
Setting
HumanEval
AQuA
MMLU
Avg.
Tokens / Task
Full
92.56
87.14
92.81
90.84
2360.45
w/o task
90.91
85.71
88.24
88.29 ↓ 2.55
2758.02 ↑ 16.84%
w/o model
87.60
87.62
88.24
87.82 ↓ 3.02
6010.64 ↑ 154.64%
w/ desc.
88.43
85.24
89.54
87.74 ↓ 3.10
4736.68 ↑ 100.67%
Table 2: Downstream ablation of competence modeling.
Figure 3: Number of agents in the generated MAS on efficiency-oriented target backbones.
Method
GLM 5.1 ⋆
Gemini 3.1 Pro ⋆
GPT-4o
Avg.
Vanilla
83.01
81.70
80.30
81.67
G-Designer
88.89
85.62
87.14
87.22
Ours
89.53
87.58
92.81
89.97 ↓ 36.26%
Table 3: Results on MMLU with performance-oriented target backbones.
Method
Qwen3-14B
Llama3.1-8B
Gemma-3-12B
Avg.
Vanilla
77.79
63.40
66.67
69.29
G-Designer
84.31
67.32
70.58
74.07
Ours
87.58
70.59
77.78
78.65
Table 4: Results on MMLU with efficiency-oriented target backbones.
Figure 4: Case studies on MMLU. Left: changing the task while fixing the backbone (Llama-3.1-8B). Right: changing the backbone from Llama-3.1-8B to GPT-4o while fixing the task. check/cross marks indicate answer correctness.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Source datasets
code_logic
HumanEval
quantitative_reasoning
AQuA
basic_math
GSM8K, MultiArith, SVAMP
general_knowledge
MMLU
Appendix
Table 5: Domain assignment used in probe-set construction.
Multi-agent systems (MAS) built on large language models have shown growing promise, with their effectiveness resting on agents' ability to coordinate through text-based channels much as human teams do. Yet recent study suggests that MAS often falter not because agents lack individual task-solving ability, but because they lack collaborative competence: the capacity to establish common ground, maintain shared task understanding, balance individual and collective incentives, and repair misalignment as interaction unfolds. Decades of research in Computer-Supported Cooperative Work have characterized these requirements for human teams coordinating under constrained communication, yet existing MAS evaluations focus mainly on task outcomes or single-agent proficiency in reasoning, planning, and tool use. To enable a systematic analysis of agents' collaborative competence in MAS, we introduce CollabSim, a configurable simulation framework that combines a theory-grounded definition of collaborative capabilities, controlled manipulation of interaction conditions, and action-level probing of agents' internal states. Experiments across four LLMs show that CollabSim can capture condition effects, separate model performance patterns, and reveal task-dependent effects of agent design.
Although large language model (LLM) based multi-agent systems (MAS) show their capability to solve complex tasks and achieve higher performance over single agent systems, they lead to huge computational overheads because of heavy communication between agents. Previous research has made efforts to train a sparse multi-agent graph or fine-tune a planner to orchestrate the workflow better. However, such extra training processes introduce computational costs and limit MAS to specific domains, therefore compromising their generalizability. In this paper, we propose CONCAT, a training-free multi-agent collaboration framework based on CONsensus and Confidence-driven Ad hoc Teaming to efficiently organize agent interactions. Specifically, agents are clustered based on their initial answers, and leaders of each cluster are selected based on the agents' confidence. Then, a heuristic function based on the Theory of Mind is designed to predict the collaboration benefits between every two leaders according to their answers and confidence. Finally, an ad hoc multi-agent network is organized after evicting a percentage of communications based on the predicted benefits. Experiments across three LLMs and three benchmarks show that CONCAT achieves up to 2.02x higher efficiency (accuracy/latency ratio) than LLM-Debate and outperforms training-aware methods such as AgentDropout, while reducing average latency by 50.1% on Qwen2.5-14B-Instruct, without any task-specific training.
Large language models (LLMs) have progressively evolved into the core of autonomous agents. Building on this progress, LLM-based multi-agent systems (MAS) coordinate multiple agents into a synergistic team to accomplish complex tasks that exceed the capabilities of individual agents. The effectiveness of such systems depends not only on the agents themselves, but also on how collaboration mechanisms are designed and organized. Note that real-world collaboration is typically partially observable, where each agent can only access partial information about the environment due to physical or privacy-related constraints. However, many existing multi-agent benchmarks assume global observability, and leave limited support for systematically evaluating collaboration mechanisms. To bridge this gap, we introduce MASBench, a multi-agent collaboration benchmark designed under partially observable constraints. It is organized into three progressive task categories: Reasoning, Scheduling, and Game. Through this structure, we progressively evaluate three representative collaboration mechanisms: Protocol, Memory, and Routing. MASBench further provides deterministic evaluation metrics, including performance score, communication cost, and cost effectiveness, to characterize both collaboration outcomes and communication overhead. Experiments across diverse LLM backbones and mechanism configurations offer empirical guidance for effective MAS design. Code is available at: https://github.com/BUPT-GAMMA/MASBench
Qizhi Chu, Zekai Yu, Sijie Wen +5
Beijing University of Posts and Telecommunications · Shanghai Jiao Tong University · Tsinghua University