Self-Adapting Group of Experts for Multi-Agent Reasoning

Mohammad Atif Quamar; Nurbek Tastan; Karthik Nandakumar; Junpei Komiyama

cs.AISep 28, 2026

Self-Adapting Group of Experts for Multi-Agent Reasoning

Authors: Mohammad Atif Quamar, Nurbek Tastan, Karthik Nandakumar, Junpei Komiyama

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Michigan State University · RIKEN AIP

Abstract

Multi-agent systems bring together language model agents with different roles to propose, review, and refine solutions. Each agent's response depends on its model's capabilities, the reasoning strategy defined by its system prompt, and the information in its input context. Existing frameworks often adapt communication by changing this context while leaving individual prompts fixed, even when a problem calls for different skills. We study whether agents' initial responses can identify a strategy better suited to the current problem and guide its transfer to other agents. To address this, we introduce SAGE (Self-Adapting Group of Experts), a training-free framework that uses answer agreement, prefix consistency, and reciprocal peer review to select a strategy donor. SAGE transfers the selected donor's reasoning strategy to the other agents while preserving their original roles. This transfer uses only the agents' original system prompts, without access to the problem or generated solutions. After strategy adaptation, agents exchange responses through a dynamic, sparse directed acyclic graph that routes information from higher-scoring agents to lower-scoring agents. Experiments across multiple agent backbones and reasoning benchmarks show that SAGE achieves higher average accuracy than the evaluated baselines. Our code is available at https://github.com/atifquamar07/sage.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SAGE . (1) $N$ agents answer the query $x$ independently. Each response is scored by answer agreement plus prefix consistency, and reciprocal peer review between higher- and lower-scoring agents selects a strategy donor (here, agent A). (2) Every other agent’s role prompt is rewritten to include the donor’s reasoning guidance while keeping its role; the rewriter sees only the two prompts, not the problem or any response. (3) Agents revise their answers along a sparse directed acyclic graph (DAG), reading the updated answers of up to $K$ higher-scoring parents; scores and the DAG are rebuilt after every round. (4) A weighted vote over initial answers, retained reviews, and round answers, with higher weight for each stage’s leader (gold rings), selects the final answer $\widehat{a}$ .

Qwen2.5-1.5B-Instruct
Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$\underline{28.96{\scriptstyle\,\pm\,2.78}}$	$\underline{53.64}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23$
MOC	$70.20{\scriptstyle\,\pm\,1.25}$	$71.20{\scriptstyle\,\pm\,0.92}$	$44.67{\scriptstyle\,\pm\,0.83}$	$34.27{\scriptstyle\,\pm\,1.22}$	$50.20{\scriptstyle\,\pm\,1.73}$	$26.94{\scriptstyle\,\pm\,2.04}$	$49.58$
MAD-M 2	$72.00{\scriptstyle\,\pm\,2.42}$	$\underline{73.80{\scriptstyle\,\pm\,0.80}}$	$62.33{\scriptstyle\,\pm\,0.61}$	$\underline{34.80{\scriptstyle\,\pm\,0.53}}$	$52.20{\scriptstyle\,\pm\,1.04}$	$25.25{\scriptstyle\,\pm\,2.67}$	$53.40$
G-Designer	$71.93{\scriptstyle\,\pm\,0.12}$	$72.47{\scriptstyle\,\pm\,0.64}$	$60.13{\scriptstyle\,\pm\,1.80}$	$34.13{\scriptstyle\,\pm\,0.95}$	$52.67{\scriptstyle\,\pm\,1.22}$	$27.78{\scriptstyle\,\pm\,5.13}$	$53.19$

Table 1: Main results on Qwen2.5-1.5B and Ministral-3-3B. Comparison of SAGE with single-agent and multi-agent baselines across six reasoning benchmarks. We report accuracy (%) as mean

\pm

sample standard deviation over three runs; AVG denotes the macro-average across benchmarks. Bold and underlined values indicate the best and second-best results within each backbone, respectively, including ties.

Figure 2: Effect of scaling the number of agents. We evaluate SAGE across six reasoning benchmarks with Qwen2.5-1.5B-Instruct (top) and Ministral-3-3B-Instruct (bottom) as the number of agents increases from 4 to 9. Accuracy generally improves with more agents on both backbones.

Figure 3: Backbone scaling. We compare Single and SAGE on GSM-Hard and GPQA-Diamond using Qwen2.5-Instruct models ranging from 0.5B to 72B parameters. The table reports mean

\pm

sample SD over three runs, with the higher mean in bold; the plots show the same results with evenly spaced model sizes.

Figure 4: Accuracy by team composition. Accuracy (%) for nine-agent teams on GSM8K and MMLU (one run). Q/M/P counts Qwen/Ministral/Phi agents; H denotes the mixed team. AVG is the benchmark mean; bold/underlined values mark best/second best.

Figure 5: Message corruption. SAGE and SelfOrg across five benchmarks with three of nine agents corrupted. Bars show means; error bars show sample SD over three runs.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SAGE
Active agents / parent limit / rounds	$N=4$ , $K=2$ , $T=3$
Prefix fraction / score weight	$\tau=0.60$ , $\lambda=0.50$
Review budget per score stratum	$m=2$
Initial, revision, and review temperature	$0.50$
Prefix / rewrite temperature	$0.60$ / $0.20$
Leader / other occurrence vote weight	$1.5$ / $1.0$

Appendix

Table 2: Main-study implementation settings. Rounds are counted after initialization. Shared rows apply to both backbones; decoding parameters apply across stages with the listed temperature overrides. Routing degree

K

and decoding top-

k

are distinct.

Method	MMMU-Pro	MathVista
Single	$40.40{\scriptstyle\,\pm\,0.20}$	$\underline{57.80{\scriptstyle\,\pm\,0.69}}$
CoT	$37.07{\scriptstyle\,\pm\,0.70}$	$51.27{\scriptstyle\,\pm\,0.92}$
SelfOrg	$\underline{41.40{\scriptstyle\,\pm\,2.11}}$	$55.47{\scriptstyle\,\pm\,1.29}$
SAGE	$\mathbf{42.40}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.80}}$	$\mathbf{61.53}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$

Appendix

Table 3: Vision-language reasoning. We compare SAGE with single-agent and SelfOrg on MMMU-Pro and MathVista using Qwen2.5-VL-3B-Instruct. SAGE achieves the best overall results on both benchmarks (left) and leads on easy and hard MMMU-Pro questions (right). Values are percentages, reported as mean

\pm

sample SD over three runs. Bold and underlined values indicate the best and second-best results, respectively.

Qwen2.5-1.5B
Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$28.96{\scriptstyle\,\pm\,2.78}$	$53.64{\scriptstyle\,\pm\,0.14}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23{\scriptstyle\,\pm\,0.80}$
SAGE-NoRewrite	$76.80{\scriptstyle\,\pm\,0.53}$	$75.60{\scriptstyle\,\pm\,1.25}$	$67.80{\scriptstyle\,\pm\,1.51}$	$37.21{\scriptstyle\,\pm\,1.31}$	$52.37{\scriptstyle\,\pm\,0.83}$	$27.49{\scriptstyle\,\pm\,1.01}$	$56.21{\scriptstyle\,\pm\,1.07}$
SAGE	$\mathbf{78.47}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.01}}$	$\mathbf{77.33}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.17}}$	$\mathbf{69.07}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.64}}$	$\mathbf{39.00}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.06}}$	$54.33{\scriptstyle\,\pm\,1.63}$	$\mathbf{31.65}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{3.04}}$	$\mathbf{58.31}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.20}}$
Ministral-3-3B

Appendix

Table 4: Prompt-rewriting ablation on both main-study backbones. Accuracy (%) is reported as mean

\pm

standard deviation over three runs; AVG first averages the six benchmark accuracies within each run. SAGE includes rewriting, while SAGE-NoRewrite retains every agent’s original prompt. Single and CoT are the results in Table 1 . Bold marks the highest mean within each backbone, including ties.

Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Qwen2.5-1.5B
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$28.96{\scriptstyle\,\pm\,2.78}$	$53.64{\scriptstyle\,\pm\,0.14}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23{\scriptstyle\,\pm\,0.80}$
SAGE-RandomDonor	$77.80{\scriptstyle\,\pm\,0.48}$	$75.20{\scriptstyle\,\pm\,0.98}$	$68.20{\scriptstyle\,\pm\,0.65}$	$37.73{\scriptstyle\,\pm\,1.21}$	$52.27{\scriptstyle\,\pm\,1.89}$	$29.14{\scriptstyle\,\pm\,3.54}$	$56.72{\scriptstyle\,\pm\,1.02}$
SAGE	$\mathbf{78.47}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.01}}$	$\mathbf{77.33}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.17}}$	$\mathbf{69.07}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.64}}$	$\mathbf{39.00}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.06}}$	$54.33{\scriptstyle\,\pm\,1.63}$	$\mathbf{31.65}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{3.04}}$	$\mathbf{58.31}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.20}}$

Appendix

Table 5: Donor-selection ablation on Qwen2.5-1.5B-Instruct. Accuracy (%) is reported as mean

\pm

standard deviation over three runs; AVG first averages the six benchmark accuracies within each run. Single, CoT, and SAGE are the results in Table 1 . SAGE-RandomDonor transfers the strategy of a randomly chosen agent instead of the donor selected by peer review. Bold marks the highest mean in each column.

Qwen2.5-1.5B
Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$28.96{\scriptstyle\,\pm\,2.78}$	$53.64{\scriptstyle\,\pm\,0.14}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23{\scriptstyle\,\pm\,0.80}$
Final-round WPV	$78.33{\scriptstyle\,\pm\,1.10}$	$76.87{\scriptstyle\,\pm\,0.50}$	$68.00{\scriptstyle\,\pm\,1.74}$	$38.33{\scriptstyle\,\pm\,0.92}$	$54.47{\scriptstyle\,\pm\,1.75}$	$29.12{\scriptstyle\,\pm\,3.29}$	$57.52{\scriptstyle\,\pm\,0.20}$
SAGE	$\mathbf{78.47}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.01}}$	$\mathbf{77.33}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.17}}$	$\mathbf{69.07}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.64}}$	$\mathbf{39.00}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.06}}$	$54.33{\scriptstyle\,\pm\,1.63}$	$\mathbf{31.65}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{3.04}}$	$\mathbf{58.31}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.20}}$
Ministral-3-3B

Appendix

Table 6: Contribution of earlier-stage answers. SAGE pools answers across stages; Final-round WPV uses only the last completed round. Accuracy (%) is mean

\pm

sample standard deviation over three runs; AVG is the mean across benchmarks within each run. Single and CoT are the results in Table 1 . Bold marks the highest mean within each backbone, including ties.

Figure 6: Donor rate by role. Percentage of problems in which a role is selected as donor when it is on the team (chance $=25\%$ ), pooled over three runs. Problems decided by the agent-index tie-break are excluded. ∗ marks benchmarks with a significant role preference (permutation test, $p<0.05$ ); the other columns are faded.

Explore similar work

May 9, 2026cs.AI

AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimization

Multi-agent reasoning has shown promise for improving the problem-solving ability of large language models by allowing multiple agents to explore diverse reasoning paths. However, most existing multi-agent methods rely on inference-time debate or aggregation, which can be vulnerable to incorrect peer influence and biased consensus. Moreover, the agents themselves remain static, as their underlying reasoning skills do not evolve across tasks. In this paper, we introduce \textbf{AgentPSO}, a particle-swarm-inspired framework for evolving multi-agent reasoning skills. AgentPSO treats each agent as a particle-like reasoner whose state is a natural-language skill and whose velocity is a semantic update direction, iteratively guiding agents toward higher-performing skill configurations. Across training iterations, each agent updates its skill by combining its previous velocity, personal-best skill, global-best skill, and a self-reflective direction derived from peer reasoning trajectories. This enables agents to learn reusable reasoning behaviors by drawing on their own experience and on the strongest skills found by the population, without updating the parameters of the backbone language model. Experiments on mathematical and general reasoning benchmarks show that AgentPSO improves over static single-agent skills and test-time-only multi-agent reasoning baselines. The evolved skills further transfer across benchmarks and to another backbone model, suggesting that AgentPSO captures reusable reasoning procedures rather than merely optimizing benchmark-specific prompts. Code is publicly available at https://github.com/HYUNMIN-HWANG/AgentPSO/.

Hyunmin Hwang, Jaemin Kim, Choonghan Kim +2

Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology, Seoul, South Korea.

Multi-Agent Reasoning Reasoning Benchmark Reasoning Trajectory Recursive Reasoner Particle Swarm Optimization

Ask PDF

Jul 12, 2026cs.AI

Route, Communicate, and Reason: Gated Routing and Adaptive Depth for Efficient Multi-Agent Reasoning

Multi-agent ensembling multiplies active parameters and inference cost without answering three basic questions: which agents to consult, how deeply a query should traverse a hierarchy of agents, and when inter-agent communication is worth its cost. We present GRADE (Gated Routing and Adaptive Depth for Efficient Reasoning), a hierarchical multi-agent system in which four lightweight learned gates jointly govern agent selection, hierarchy depth, inter-agent communication, and branch pruning. Training uses CoGRPO (Collaborative Group-Relative Policy Optimization), a novel critic-free recipe that adapts GRPO to multi-agent hierarchies and assigns a shared advantage signal to every gate and agent that participated in a rollout. Agent models are drawn from a hot-swappable Expert Registry; per-agent calibration maps allow experts to be replaced at inference time without retraining. At

\sim

17B average active parameters, GRADE outperforms all baselines on GSM8K, MMLUPro, and GPQA, surpassing the strongest baseline by 4.8 points on MMLUPro at half the active compute. On AIME-2025, where model depth dominates, GRADE remains competitive to existing frameworks. Ablations isolate the hierarchy and masked cross-attention as the largest contributors to accuracy, and show that per-agent calibration is necessary for safe hot-swapping.

Sudipto Ghosh, Tanmoy Chakraborty

Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi

Multi-Agent Reasoning Reasoning Skills Inference Cost

Ask PDF

Oct 14, 2025cs.MA

Benefits and Limitations of Communication in Multi-Agent Reasoning

Chain-of-thought prompting has popularized step-by-step reasoning in large language models, yet model performance still degrades as problem complexity and context length grow. By decomposing difficult tasks with long contexts into shorter, manageable ones, recent multi-agent paradigms offer a promising near-term solution to this problem. However, the fundamental capacities of such systems are poorly understood. In this work, we propose a theoretical framework to analyze the expressivity of multi-agent systems. We apply our framework to three algorithmic families: state tracking, recall, and

k

-hop reasoning. We derive bounds on (i) the number of agents required to solve the task exactly, (ii) the quantity and structure of inter-agent communication, and (iii) the achievable speedups as problem size and context scale. Our results identify regimes where communication is provably beneficial, delineate tradeoffs between agent count and bandwidth, and expose intrinsic limitations when either resource is constrained. We complement our theoretical analysis with a set of experiments on pretrained LLMs using controlled synthetic benchmarks. Empirical outcomes confirm the tradeoffs between key quantities predicted by our theory. Collectively, our analysis offers principled guidance for designing scalable multi-agent reasoning systems.

Michael Rizvi-Martel, Satwik Bhattamishra, Neil Rathi +2

Mila & Universit´e de Montr´eal · University of Oxford · Stanford University +1

Multi-Agent Reasoning Multi-Hop Reasoning Chain-of-Thought Reasoning Expressivity

Ask PDF