Self-Adapting Group of Experts for Multi-Agent Reasoning

Mohammad Atif Quamar; Nurbek Tastan; Karthik Nandakumar; Junpei Komiyama

cs.AISep 28, 2026

Self-Adapting Group of Experts for Multi-Agent Reasoning

Authors: Mohammad Atif Quamar, Nurbek Tastan, Karthik Nandakumar, Junpei Komiyama

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Michigan State University · RIKEN AIP

Abstract

Multi-agent systems bring together language model agents with different roles to propose, review, and refine solutions. Each agent's response depends on its model's capabilities, the reasoning strategy defined by its system prompt, and the information in its input context. Existing frameworks often adapt communication by changing this context while leaving individual prompts fixed, even when a problem calls for different skills. We study whether agents' initial responses can identify a strategy better suited to the current problem and guide its transfer to other agents. To address this, we introduce SAGE (Self-Adapting Group of Experts), a training-free framework that uses answer agreement, prefix consistency, and reciprocal peer review to select a strategy donor. SAGE transfers the selected donor's reasoning strategy to the other agents while preserving their original roles. This transfer uses only the agents' original system prompts, without access to the problem or generated solutions. After strategy adaptation, agents exchange responses through a dynamic, sparse directed acyclic graph that routes information from higher-scoring agents to lower-scoring agents. Experiments across multiple agent backbones and reasoning benchmarks show that SAGE achieves higher average accuracy than the evaluated baselines. Our code is available at https://github.com/atifquamar07/sage.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of SAGE . (1) $N$ agents answer the query $x$ independently. Each response is scored by answer agreement plus prefix consistency, and reciprocal peer review between higher- and lower-scoring agents selects a strategy donor (here, agent A). (2) Every other agent’s role prompt is rewritten to include the donor’s reasoning guidance while keeping its role; the rewriter sees only the two prompts, not the problem or any response. (3) Agents revise their answers along a sparse directed acyclic graph (DAG), reading the updated answers of up to $K$ higher-scoring parents; scores and the DAG are rebuilt after every round. (4) A weighted vote over initial answers, retained reviews, and round answers, with higher weight for each stage’s leader (gold rings), selects the final answer $\widehat{a}$ .

Qwen2.5-1.5B-Instruct
Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$\underline{28.96{\scriptstyle\,\pm\,2.78}}$	$\underline{53.64}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23$
MOC	$70.20{\scriptstyle\,\pm\,1.25}$	$71.20{\scriptstyle\,\pm\,0.92}$	$44.67{\scriptstyle\,\pm\,0.83}$	$34.27{\scriptstyle\,\pm\,1.22}$	$50.20{\scriptstyle\,\pm\,1.73}$	$26.94{\scriptstyle\,\pm\,2.04}$	$49.58$
MAD-M 2	$72.00{\scriptstyle\,\pm\,2.42}$	$\underline{73.80{\scriptstyle\,\pm\,0.80}}$	$62.33{\scriptstyle\,\pm\,0.61}$	$\underline{34.80{\scriptstyle\,\pm\,0.53}}$	$52.20{\scriptstyle\,\pm\,1.04}$	$25.25{\scriptstyle\,\pm\,2.67}$	$53.40$
G-Designer	$71.93{\scriptstyle\,\pm\,0.12}$	$72.47{\scriptstyle\,\pm\,0.64}$	$60.13{\scriptstyle\,\pm\,1.80}$	$34.13{\scriptstyle\,\pm\,0.95}$	$52.67{\scriptstyle\,\pm\,1.22}$	$27.78{\scriptstyle\,\pm\,5.13}$	$53.19$

Table 1: Main results on Qwen2.5-1.5B and Ministral-3-3B. Comparison of SAGE with single-agent and multi-agent baselines across six reasoning benchmarks. We report accuracy (%) as mean

\pm

sample standard deviation over three runs; AVG denotes the macro-average across benchmarks. Bold and underlined values indicate the best and second-best results within each backbone, respectively, including ties.

Figure 2: Effect of scaling the number of agents. We evaluate SAGE across six reasoning benchmarks with Qwen2.5-1.5B-Instruct (top) and Ministral-3-3B-Instruct (bottom) as the number of agents increases from 4 to 9. Accuracy generally improves with more agents on both backbones.

Figure 3: Backbone scaling. We compare Single and SAGE on GSM-Hard and GPQA-Diamond using Qwen2.5-Instruct models ranging from 0.5B to 72B parameters. The table reports mean

\pm

sample SD over three runs, with the higher mean in bold; the plots show the same results with evenly spaced model sizes.

Figure 4: Accuracy by team composition. Accuracy (%) for nine-agent teams on GSM8K and MMLU (one run). Q/M/P counts Qwen/Ministral/Phi agents; H denotes the mixed team. AVG is the benchmark mean; bold/underlined values mark best/second best.

Figure 5: Message corruption. SAGE and SelfOrg across five benchmarks with three of nine agents corrupted. Bars show means; error bars show sample SD over three runs.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SAGE
Active agents / parent limit / rounds	$N=4$ , $K=2$ , $T=3$
Prefix fraction / score weight	$\tau=0.60$ , $\lambda=0.50$
Review budget per score stratum	$m=2$
Initial, revision, and review temperature	$0.50$
Prefix / rewrite temperature	$0.60$ / $0.20$
Leader / other occurrence vote weight	$1.5$ / $1.0$

Appendix

Table 2: Main-study implementation settings. Rounds are counted after initialization. Shared rows apply to both backbones; decoding parameters apply across stages with the listed temperature overrides. Routing degree

K

and decoding top-

k

are distinct.

Method	MMMU-Pro	MathVista
Single	$40.40{\scriptstyle\,\pm\,0.20}$	$\underline{57.80{\scriptstyle\,\pm\,0.69}}$
CoT	$37.07{\scriptstyle\,\pm\,0.70}$	$51.27{\scriptstyle\,\pm\,0.92}$
SelfOrg	$\underline{41.40{\scriptstyle\,\pm\,2.11}}$	$55.47{\scriptstyle\,\pm\,1.29}$
SAGE	$\mathbf{42.40}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.80}}$	$\mathbf{61.53}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$

Appendix

Table 3: Vision-language reasoning. We compare SAGE with single-agent and SelfOrg on MMMU-Pro and MathVista using Qwen2.5-VL-3B-Instruct. SAGE achieves the best overall results on both benchmarks (left) and leads on easy and hard MMMU-Pro questions (right). Values are percentages, reported as mean

\pm

sample SD over three runs. Bold and underlined values indicate the best and second-best results, respectively.

Qwen2.5-1.5B
Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$28.96{\scriptstyle\,\pm\,2.78}$	$53.64{\scriptstyle\,\pm\,0.14}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23{\scriptstyle\,\pm\,0.80}$
SAGE-NoRewrite	$76.80{\scriptstyle\,\pm\,0.53}$	$75.60{\scriptstyle\,\pm\,1.25}$	$67.80{\scriptstyle\,\pm\,1.51}$	$37.21{\scriptstyle\,\pm\,1.31}$	$52.37{\scriptstyle\,\pm\,0.83}$	$27.49{\scriptstyle\,\pm\,1.01}$	$56.21{\scriptstyle\,\pm\,1.07}$
SAGE	$\mathbf{78.47}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.01}}$	$\mathbf{77.33}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.17}}$	$\mathbf{69.07}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.64}}$	$\mathbf{39.00}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.06}}$	$54.33{\scriptstyle\,\pm\,1.63}$	$\mathbf{31.65}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{3.04}}$	$\mathbf{58.31}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.20}}$
Ministral-3-3B

Appendix

Table 4: Prompt-rewriting ablation on both main-study backbones. Accuracy (%) is reported as mean

\pm

standard deviation over three runs; AVG first averages the six benchmark accuracies within each run. SAGE includes rewriting, while SAGE-NoRewrite retains every agent’s original prompt. Single and CoT are the results in Table 1 . Bold marks the highest mean within each backbone, including ties.

Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Qwen2.5-1.5B
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$28.96{\scriptstyle\,\pm\,2.78}$	$53.64{\scriptstyle\,\pm\,0.14}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23{\scriptstyle\,\pm\,0.80}$
SAGE-RandomDonor	$77.80{\scriptstyle\,\pm\,0.48}$	$75.20{\scriptstyle\,\pm\,0.98}$	$68.20{\scriptstyle\,\pm\,0.65}$	$37.73{\scriptstyle\,\pm\,1.21}$	$52.27{\scriptstyle\,\pm\,1.89}$	$29.14{\scriptstyle\,\pm\,3.54}$	$56.72{\scriptstyle\,\pm\,1.02}$
SAGE	$\mathbf{78.47}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.01}}$	$\mathbf{77.33}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.17}}$	$\mathbf{69.07}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.64}}$	$\mathbf{39.00}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.06}}$	$54.33{\scriptstyle\,\pm\,1.63}$	$\mathbf{31.65}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{3.04}}$	$\mathbf{58.31}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.20}}$

Appendix

Table 5: Donor-selection ablation on Qwen2.5-1.5B-Instruct. Accuracy (%) is reported as mean

\pm

standard deviation over three runs; AVG first averages the six benchmark accuracies within each run. Single, CoT, and SAGE are the results in Table 1 . SAGE-RandomDonor transfers the strategy of a randomly chosen agent instead of the donor selected by peer review. Bold marks the highest mean in each column.

Qwen2.5-1.5B
Method	MATH	GSM8K	AQuA	GSM-H	MMLU	GPQA	AVG
Single	$72.07{\scriptstyle\,\pm\,2.00}$	$69.80{\scriptstyle\,\pm\,0.72}$	$61.80{\scriptstyle\,\pm\,1.73}$	$34.27{\scriptstyle\,\pm\,0.50}$	$\mathbf{54.93}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.12}}$	$28.96{\scriptstyle\,\pm\,2.78}$	$53.64{\scriptstyle\,\pm\,0.14}$
CoT	$70.00{\scriptstyle\,\pm\,0.72}$	$71.40{\scriptstyle\,\pm\,2.23}$	$59.20{\scriptstyle\,\pm\,0.92}$	$31.80{\scriptstyle\,\pm\,0.40}$	$52.87{\scriptstyle\,\pm\,1.10}$	$28.11{\scriptstyle\,\pm\,3.21}$	$52.23{\scriptstyle\,\pm\,0.80}$
Final-round WPV	$78.33{\scriptstyle\,\pm\,1.10}$	$76.87{\scriptstyle\,\pm\,0.50}$	$68.00{\scriptstyle\,\pm\,1.74}$	$38.33{\scriptstyle\,\pm\,0.92}$	$54.47{\scriptstyle\,\pm\,1.75}$	$29.12{\scriptstyle\,\pm\,3.29}$	$57.52{\scriptstyle\,\pm\,0.20}$
SAGE	$\mathbf{78.47}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.01}}$	$\mathbf{77.33}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.17}}$	$\mathbf{69.07}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.64}}$	$\mathbf{39.00}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{1.06}}$	$54.33{\scriptstyle\,\pm\,1.63}$	$\mathbf{31.65}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{3.04}}$	$\mathbf{58.31}{\scriptstyle\,\boldsymbol{\pm}\,\mathbf{0.20}}$
Ministral-3-3B

Appendix

Table 6: Contribution of earlier-stage answers. SAGE pools answers across stages; Final-round WPV uses only the last completed round. Accuracy (%) is mean

\pm

sample standard deviation over three runs; AVG is the mean across benchmarks within each run. Single and CoT are the results in Table 1 . Bold marks the highest mean within each backbone, including ties.

Figure 6: Donor rate by role. Percentage of problems in which a role is selected as donor when it is on the team (chance $=25\%$ ), pooled over three runs. Problems decided by the agent-index tie-break are excluded. ∗ marks benchmarks with a significant role preference (permutation test, $p<0.05$ ); the other columns are faded.

Explore similar work

AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimization
May 9, 2026Hyunmin Hwang, Jaemin Kim, Choonghan Kim +2Multi-Agent Reasoning Reasoning Benchmark
Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology, Seoul, South Korea.
Route, Communicate, and Reason: Gated Routing and Adaptive Depth for Efficient Multi-Agent Reasoning
Jul 12, 2026Sudipto Ghosh, Tanmoy ChakrabortyMulti-Agent Reasoning Reasoning Skills
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi
Benefits and Limitations of Communication in Multi-Agent Reasoning
Oct 14, 2025Michael Rizvi-Martel, Satwik Bhattamishra, Neil Rathi +2Multi-Agent Reasoning Multi-Hop Reasoning
Mila & Universit´e de Montr´eal · University of Oxford · Stanford University +1

May 9, 2026, cs.AIJ/K move · Enter open · S save

AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimization

Hyunmin Hwang, Jaemin Kim, Choonghan Kim, Hangeol Chang+1

Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology, Seoul, South Korea.

Multi-Agent Reasoning Reasoning Benchmark Reasoning Trajectory Recursive Reasoner Particle Swarm Optimization

Open paper AskPDF arXiv

Multi-agent reasoning has shown promise for improving the problem-solving ability of large language models by allowing multiple agents to explore diverse reasoning paths. However, most existing multi-agent methods rely on inference-time debate or aggregation, which can be vulnerable to incorrect peer influence and biased consensus. Moreover, the agents themselves remain static, as their underlying reasoning skills do not evolve across tasks. In this paper, we introduce \textbf{AgentPSO}, a particle-swarm-inspired framework for evolving multi-agent reasoning skills. AgentPSO treats each agent as a particle-like reasoner whose state is a natural-language skill and whose velocity is a semantic update direction, iteratively guiding agents toward higher-performing skill configurations. Across training iterations, each agent updates its skill by combining its previous velocity, personal-best skill, global-best skill, and a self-reflective direction derived from peer reasoning trajectories. This enables agents to learn reusable reasoning behaviors by drawing on their own experience and on the strongest skills found by the population, without updating the parameters of the backbone language model. Experiments on mathematical and general reasoning benchmarks show that AgentPSO improves over static single-agent skills and test-time-only multi-agent reasoning baselines. The evolved skills further transfer across benchmarks and to another backbone model, suggesting that AgentPSO captures reusable reasoning procedures rather than merely optimizing benchmark-specific prompts. Code is publicly available at https://github.com/HYUNMIN-HWANG/AgentPSO/.

Abstract

Explore similar work

AgentPSO: Evolving Agent Reasoning Skill via Multi-agent Particle Swarm Optimization

Route, Communicate, and Reason: Gated Routing and Adaptive Depth for Efficient Multi-Agent Reasoning

Benefits and Limitations of Communication in Multi-Agent Reasoning