MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation

Yuhan Sun; Jiaxin Zhao; Yongdong Luo; Yibo Hu; Meiguang Jin; Junfeng Ma; Weixun Wang; Weihang Pan

cs.CLOct 16, 2026

MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation

Authors: Yuhan Sun, Jiaxin Zhao, Yongdong Luo, Yibo Hu, Meiguang Jin, Junfeng Ma, Weixun Wang, Weihang Pan,

Organizations: Alibaba Group · Zhejiang University

Abstract

In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents. An LLM judge can score the performance of the subagents, but these scores depend on the judge's own assessment of the quality of subagents' responses. We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end. We first train a Teacher in a Searcher-Answerer workflow using answer correctness. During team training, the frozen Teacher scores student-generated search and summary tokens to provide local process supervision for the Searchers, while the Leader learns from team success and the Answerer from candidate correctness. Across seven datasets, the trained system reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL, the trained cascaded multiagent system, by 2.61 percentage points. The team also solves more questions within four attempts.

Figures & tables

Figure 1: MultiAgent-OPD: multi-wave collaboration with two training signals. The same Leader assigns successive waves of Searchers , collects their reports, and selects evidence for the Answerer . Each Searcher retrieves evidence for its assigned subtask and summarizes the retrieved passages into a report. Gray arrows show this workflow. Orange OPD feedback from the frozen Teacher trains the Searchers , and green final-answer reward feedback trains the Leader and the Answerer . All student roles share one model. Two waves are shown for illustration.

Figure 2: Role-selective process supervision in MultiAgent-OPD. (a) Multiple Agent Teams sample trajectories for the same question. Team outcomes yield GRPO advantages for the Leader , and candidate rewards yield GRPO advantages for the Answerer within each workflow. (b) The Leader spawns parallel Searchers over successive waves, collects their summaries, and delegates the final answer to the Answerer . The Searchers follow ReAct trajectories to search and summarize results. The Searchers receive per-token OPD guidance weights from the frozen Teacher through teacher–old-student log-probability differences under the same local history. Retrieved observations and summaries received by the Leader are masked from the corresponding role’s loss.

Cascaded and hierarchical systems
Method	Single-hop			Multi-hop				Macro
	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle
C-MAS base	30.0	50.9	25.6	23.6	12.9	6.3	26.4	25.09
C-MAS + RL	43.7	60.2	33.9	33.3	30.5	11.5	39.4	36.07
H-MAS base	30.7	49.8	23.2	27.1	22.6	9.7	37.2	28.60
Prompted Critic	37.4	56.9	28.5	33.1	33.9	11.5	41.0	34.62

Table 1: Main results: avg@4 and pass@4 (%). Macro weights the seven datasets equally. Missing and malformed answers score zero. C-MAS denotes the cascaded Searcher – Answerer system , and H-MAS denotes the hierarchical Leader – Searcher – Answerer system. Prompted Critic and MultiAgent-OPD both use H-MAS. The two base systems use the same general-purpose instruction model without task-specific training. The RL-based search agents use the same backbone and protocol. Bold marks the highest value for each metric and dataset.

Comparator	$\Delta$ avg@4 (pp)	95% CI
Prompted Critic	+4.06	[+2.98, +5.12]
C-MAS + RL	+2.61	[+1.49, +3.73]

Table 2: MultiAgent-OPD score minus each compared system (percentage points). In each bootstrap sample, we resample questions within datasets and keep all four attempts for each question together.

Method	Single-hop			Multi-hop				Macro
	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle
MultiAgent-OPD (full)	42.3	62.5	33.6	37.5	34.0	14.0	46.8	38.68
w/ outcome-based Searcher RL	12.5	30.8	14.7	16.8	27.6	3.9	18.2	17.78
w/ base-model teacher	27.8	49.7	26.4	29.5	25.2	11.2	36.6	29.50
w/o direct Searcher supervision	31.3	51.7	27.1	28.2	21.2	10.1	39.2	29.82

Table 3: Diagnostic comparisons of Searcher supervision and Teacher training: avg@4 and pass@4 (%). Relative to the full method, w/ outcome-based Searcher RL replaces Searcher OPD with RL using final-answer rewards. w/ base-model teacher uses the base model instead of the search-trained Teacher for OPD. w/o direct Searcher supervision removes the direct training losses of the Searchers . All variants keep the training of the Leader and the Answerer . Macro weights the seven datasets equally.

Figure 3: Training-time validation avg@1. Each curve shows one smoothed historical validation run, digitized from the recorded curves. The horizontal axis counts training steps, not compute. These trajectories describe the displayed runs and do not measure variation across training seeds.

Method	avg@4	pass@4	4/4 correct
C-MAS base	25.09	34.02	17.12
C-MAS + RL	36.07	45.97	27.05
H-MAS base	28.60	39.32	18.73
Prompted Critic	34.62	45.96	23.24
MultiAgent-OPD	38.68	48.73	29.27

Table 4: Success across four attempts (macro %). Pass@4 counts questions answered correctly at least once. The 4/4 score counts questions answered correctly every time. These scores do not include a method for selecting an answer.

Figure 4: Correctness and recorded inference workload. Recorded token totals include repeated histories and do not measure latency, unique context, service cost, or training compute. Error bars show pointwise question-bootstrap intervals.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	avg@4 CI	pass@4 CI
C-MAS base	[23.85, 26.37]	[32.41, 35.64]
C-MAS + RL	[34.54, 37.56]	[44.25, 47.65]
H-MAS base	[27.19, 29.95]	[37.63, 40.96]
Prompted Critic	[33.19, 35.99]	[44.25, 47.59]
MultiAgent-OPD	[37.15, 40.18]	[47.05, 50.39]

Appendix

Table 5: Macro scores with 95% question-bootstrap intervals (%) for the fixed models used in evaluation.

Method	No candidate	Empty candidate	Untagged	Failure (macro %)	Rescued	Relaxed EM (%)
C-MAS base	493	0	173	3.41	3	25.105
C-MAS + RL	22	0	0	0.10	0	36.069
H-MAS base	2,451	104	109	12.79	5	28.624
Prompted Critic	658	9	0	3.73	0	34.617
MultiAgent-OPD	62	66	0	0.68	0	38.677

Appendix

Table 6: Answer completion and flexible extraction. Candidate, untagged, and rescued are total counts out of 20,000 records per method. Failure rate and relaxed EM give each dataset equal weight. Rescued counts extra correct answers found by the flexible rules. All failures remain in the evaluation.

Comparator	Remaining gap (pp)	95% CI
Prompted Critic	+0.33	[-0.80, +1.46]
C-MAS + RL	+2.51	[+1.39, +3.65]

Appendix

Table 7: Upper bound from repairing format failures (pp). We assume every format failure in the compared system becomes correct and leave MultiAgent-OPD unchanged. These are assumed bounds, not results from actual repairs.

Step	EM (%)	No search (%)	Final answer (%)
100	27.0	1.2	64.8
200	30.9	0.4	78.8
300	26.0	0.5	49.9
400	21.8	100.0	99.3
500	19.2	100.0	99.5

Appendix

Table 8: Training progression of the outcome-based Searcher RL variant on 1,000 fixed development questions. EM, workflows with no recorded search, and workflows producing a final answer are percentages. Development EM uses one attempt per question and a pooled question average; it is distinct from test-set macro avg@4.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,032 papers

MAS-OPD: On-Policy Distillation for Multi-agent Systems
Sep 28, 2026Qiyong Zhong, Mao Zheng, Mingyang Song +5Multi-Agent Coordination Multi-Agent Collaboration
University of Science and Technology of China · Foundation Model Department, Tencent · Zhejiang University +1
Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems
Feb 9, 2026Lang Feng, Longtao Zheng, Shuo He +2Multi-Agent LLM Systems Multi-Agent System Optimization
Nanyang Technological University, Singapore
MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems
May 7, 2026Zhexuan Wang, Xuebo Liu, Li Wang +4Multi-Agent LLM Systems Multi-Agent System Optimization
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China.

Sep 28, 2026J/K move · Enter open · S save

MAS-OPD: On-Policy Distillation for Multi-agent Systems

Qiyong Zhong, Mao Zheng, Mingyang Song, Houcheng Jiang+4

University of Science and Technology of China · Foundation Model Department, Tencent · Zhejiang University+1

Multi-Agent Coordination Multi-Agent Collaboration Multi-Agent Systems On-Policy Distillation

Open paper AskPDF arXiv

Multi-agent systems (MAS) split a task across specialized roles and are promising on complex tasks, yet a prevailing approach relies on inference-time orchestration alone. General-purpose APIs are costly and hard to customize, while small models with role prompts rarely develop stable role competence or reliable collaboration, so post-training a MAS jointly is central. Most attempts use reinforcement learning, whose team-level reward leaves undetermined which step of which agent brought about the outcome, while local rewards need redesigning per task. On-policy distillation (OPD) gives token-level teacher supervision on trajectories the student samples, a denser signal needing no local reward, yet is underexplored for the interdependent agents of a MAS. Two difficulties arise: building complementary specialization from a judgement of which role a behavior belongs to while preserving the knowledge all roles need, and turning cross-agent collaborative information into supervision OPD can exploit. We present MAS-OPD, where Role-Advantage Specialization defines the role advantage as the difference between the teacher signals under target and non-target role conditions, and Privileged Attribution for Coordination attributes an interaction conflict to its source and supplies it to the teacher alone as privileged information. Extensive experiments on code and mathematics benchmarks show that MAS-OPD attains the highest mean score at both student scales and leads the agents to develop clearer role specialization and more effective collaborative behavior.

Abstract

Multi-Agent LLM Systems

Explore similar work

MAS-OPD: On-Policy Distillation for Multi-agent Systems

Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems