Reward-Guided Autoregressive Graph Generation for Efficient Multi-Agent Communication Topology Design

Poomphob Suwannapichat; Boonyarit Changaival; Caesar Wu; Pascal Bouvry

cs.MAAug 20, 2026

Reward-Guided Autoregressive Graph Generation for Efficient Multi-Agent Communication Topology Design

Authors: Poomphob Suwannapichat, Boonyarit Changaival, Caesar Wu, Pascal Bouvry

Organizations: University of Luxembourg, Luxembourg · King Mongkut’s University of Technology Thonburi, Thailand

Abstract

LLM-based Multi-Agent Systems (MAS) achieve strong performance on complex reasoning tasks by coordinating multiple agents, but at the cost of substantial token consumption. Recent work on automatic topology design, ARG-Designer, has reframed this problem as autoregressive graph generation. However, its training objective provides no explicit incentive for the model to generate sparse and efficient topologies. We address this limitation by introducing a Reward-Guided Autoregressive Graph Generation (RGA-Designer) inspired by Reinforcement Learning from Human Feedback (RLHF). We train a reward model that jointly captures task correctness and structural compactness, and then fine-tune the pretrained graph generator using the reward model as feedback. Our method preserves task accuracy at the level of ARG-Designer while reducing token consumption by an average of 20.5%.

Figures & tables

Figure 1: Overview of our reward-guided pipeline for MAS topology generation.

Category	Dataset	#Test	Metric
Mathematical	GSM8K	500	Accuracy
reasoning	AQuA	214	Accuracy
	MultiArith	500	Accuracy
	SVAMP	500	Accuracy
General reasoning	MMLU	500	Accuracy
Code generation	HumanEval	121	Pass@1

Table 1: Summary of the six evaluation benchmarks used in our experiments. #Test denotes the number of test instances sampled from each dataset for evaluation.

Benchmark	Vanilla	G-Designer	AgentPrune	AgentDropout	ARG	RGA (Our)
GSM8K	82.68 $\pm$ 0.72	86.98 $\pm$ 0.65	88.00 $\pm$ 1.59	89.08 $\pm$ 1.26	88.36 $\pm$ 1.04	88.76 $\pm$ 0.66
AQuA	76.87 $\pm$ 1.21	83.35 $\pm$ 1.86	82.90 $\pm$ 1.08	83.41 $\pm$ 1.86	81.64 $\pm$ 1.56	82.10 $\pm$ 3.89
MultiArith	97.56 $\pm$ 0.23	97.96 $\pm$ 0.08	98.02 $\pm$ 0.35	97.88 $\pm$ 0.33	98.46 $\pm$ 0.13	98.44 $\pm$ 0.23
SVAMP	90.20 $\pm$ 0.27	94.28 $\pm$ 0.21	94.80 $\pm$ 0.84	94.54 $\pm$ 0.38	94.98 $\pm$ 0.85	94.52 $\pm$ 0.49
HumanEval	73.22 $\pm$ 2.14	80.08 $\pm$ 1.29	82.23 $\pm$ 2.90	81.65 $\pm$ 3.67	82.56 $\pm$ 1.76	83.22 $\pm$ 1.95
MMLU	71.28 $\pm$ 0.78	60.46 $\pm$ 1.06	79.74 $\pm$ 1.49	79.86 $\pm$ 1.33	79.32 $\pm$ 0.91	79.78 $\pm$ 1.27

Table 2: Task accuracy (%) across six benchmarks, reported as mean

\pm

standard deviation over 10 runs.

Benchmark	Vanilla	G-Designer	AgentPrune	AgentDropout	ARG	RGA (Our)
GSM8K	224 $\pm$ 10	6104 $\pm$ 12	5694 $\pm$ 118	6447 $\pm$ 441	4546 $\pm$ 275	3863 $\pm$ 195
AQuA	509 $\pm$ 97	9134 $\pm$ 67	5710 $\pm$ 263	5535 $\pm$ 266	3914 $\pm$ 162	2633 $\pm$ 191
MultiArith	135 $\pm$ 0	5347 $\pm$ 10	4997 $\pm$ 147	5644 $\pm$ 454	2824 $\pm$ 11	2754 $\pm$ 507
SVAMP	142 $\pm$ 10	5242 $\pm$ 12	4921 $\pm$ 84	5748 $\pm$ 471	3816 $\pm$ 290	3349 $\pm$ 446
HumanEval	262 $\pm$ 1	2851 $\pm$ 23	3043 $\pm$ 133	2659 $\pm$ 122	2238 $\pm$ 119	1715 $\pm$ 88
MMLU	208 $\pm$ 16	10498 $\pm$ 47	11030 $\pm$ 1388	8353 $\pm$ 1353	5554 $\pm$ 798	3875 $\pm$ 538

Table 3: Average tokens usage per task across six benchmarks, reported as mean

\pm

standard deviation over 10 runs (rounded to integer). Best results among MAS methods (excluding the Vanilla baseline) are in bold .

Accuracy
Comparison	GSM8K	AQuA	MultiArith	SVAMP	HumanEval	MMLU
G-Designer	< 0.001 $\uparrow$	0.376 $\downarrow$	< 0.001 $\uparrow$	0.180 $\uparrow$	< 0.001 $\uparrow$	< 0.001 $\uparrow$
AgentPrune	0.188 $\uparrow$	0.544 $\downarrow$	0.006 $\uparrow$	0.377 $\downarrow$	0.384 $\uparrow$	0.949 $\uparrow$
AgentDropout	0.489 $\downarrow$	0.354 $\downarrow$	< 0.001 $\uparrow$	0.920 $\downarrow$	0.252 $\uparrow$	0.892 $\downarrow$
ARG-Designer	0.320 $\uparrow$	0.735 $\uparrow$	0.814 $\downarrow$	0.160 $\downarrow$	0.437 $\uparrow$	0.365 $\uparrow$
Token usage

Table 4: Welch

t

-test

p

-values of RGA-Designer against each benchmark. Bold marks

p<0.05

. Arrows give the direction of the difference: for accuracy

\uparrow

favours RGA-Designer, for token usage

\downarrow

favours RGA-Designer.

	GSM8K		HumanEval		MMLU
	Acc. $\uparrow$	Tok. $\downarrow$	Acc. $\uparrow$	Tok. $\downarrow$	Acc. $\uparrow$	Tok. $\downarrow$
ARG-Designer	88.36 $\pm$ 1.04	4546 $\pm$ 275	82.56 $\pm$ 1.76	2238 $\pm$ 119	79.32 $\pm$ 0.91	5554 $\pm$ 798
Full	88.76 $\pm$ 0.66	3863 $\pm$ 195	83.22 $\pm$ 1.95	1715 $\pm$ 88	79.78 $\pm$ 1.27	3875 $\pm$ 538
w/o RM	88.72 $\pm$ 0.99	4166 $\pm$ 522	82.40 $\pm$ 2.17	1839 $\pm$ 116	79.42 $\pm$ 1.18	5288 $\pm$ 889
w/o BoN	88.28 $\pm$ 0.76	4281 $\pm$ 253	81.49 $\pm$ 2.45	1805 $\pm$ 119	79.84 $\pm$ 0.94	5056 $\pm$ 527
w/o both	88.72 $\pm$ 1.14	4207 $\pm$ 451	82.64 $\pm$ 2.02	1908 $\pm$ 95	78.36 $\pm$ 0.96	5425 $\pm$ 459

Table 5: Ablation study on three settings. Full is our complete method; w/o RM removes the global reward model; w/o BoN removes Best-of-

N

selection. w/o both removes both components. Results are reported as mean

\pm

standard deviation over 10 runs.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description	Value
$\lambda_{c}$	Task-completion weight	0.6
$\lambda_{\mathcal{V}}$	Agent-count weight	0.3
$\lambda_{\mathcal{E}}$	Edge-count weight	0.1
$\mathcal{V}_{\mathrm{max}}$	Maximum agents allowed	4 / 5 / 6
$\mathcal{E}_{\mathrm{min}}$	Minimum edges	$\|\mathcal{V}\|-1$
$\mathcal{E}_{\mathrm{max}}$	Maximum edges	$\tfrac{\|\mathcal{V}\|(\|\mathcal{V}\|-1)}{2}$

Appendix

Table 6: Hyperparameter settings. Values are shared across all six benchmarks unless noted otherwise.

Multi-Agent Communication Topology

See all 34 papers

RADAR: Redundancy-Aware Diffusion for Multi-Agent Communication Structure Generation
May 11, 2026Zhen Zhang, Wanjing Zhou, Juncheng Li +3Multi-Agent LLM Systems Multi-Agent System Optimization
National Key Laboratory for Novel Software Technology, Nanjing University, China · Zhejiang University · University of Oxford +1
GoAgent: Group-of-Agents Communication Topology Generation for LLM-based Multi-Agent Systems
Mar 20, 2026Hongjiang Chen, Xin Zheng, Yixin Liu +7Multi-Agent Communication Topology Multi-Agent Communication
School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China · School of Computing Technologies, RMIT University, Melbourne, Australia · School of Information and Communication Technology, Griffith University, Goldcoast, Australia
Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization
Mar 3, 2026Yueyang Cang, Xiaoteng Zhang, Erlu Zhao +7Communication Topology Optimization Multi-Agent Communication Topology
Tsinghua University · Donghua University

May 11, 2026J/K move · Enter open · S save

RADAR: Redundancy-Aware Diffusion for Multi-Agent Communication Structure Generation

Zhen Zhang, Wanjing Zhou, Juncheng Li, Hao Fei+2

National Key Laboratory for Novel Software Technology, Nanjing University, China · Zhejiang University · University of Oxford+1

Multi-Agent LLM Systems Multi-Agent System Optimization Multi-Agent Communication Multi-Agent Communication Topology

Open paper AskPDF arXiv

Compared with individual agents, large language model based multi-agent systems have shown great capabilities consistently across diverse tasks, including code generation, mathematical reasoning, and planning, etc. Despite their impressive performance, the effectiveness and robustness of these systems heavily rely on their communication topology, which is often fixed or generated in a single step. This restricts fine-grained structural exploration and flexible composition, resulting in excessive token utilization on simple tasks while limiting capability on complicated tasks. To mitigate this challenge, we introduce RADAR, a redundancy-aware and query-adaptive generative framework that actively reduce communication overhead. Motivated by recent progress in conditional discrete graph diffusion models, we formulate communication topology design as a step-by-step generation process, guided by the effective size of the graph. Comprehensive experiments on six benchmarks demonstrate that RADAR consistently outperforms recent baselines, achieving higher accuracy, lower token consumption, and greater robustness across diverse scenarios. Our code and data are available at https://github.com/cszhangzhen/RADAR.

Abstract

Multi-Agent Communication Topology

Explore similar work

RADAR: Redundancy-Aware Diffusion for Multi-Agent Communication Structure Generation

GoAgent: Group-of-Agents Communication Topology Generation for LLM-based Multi-Agent Systems

Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization