Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

Lang Feng; Longtao Zheng; Shuo He; Fuxiang Zhang; Bo An

cs.LGFeb 9, 2026

Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

Authors: Lang Feng, Longtao Zheng, Shuo He, Fuxiang Zhang, Bo An

Organizations: Nanyang Technological University, Singapore

Abstract

Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6% avg@16 and +4.6% pass@16 on math, and +15.2% avg@16 and +13.1% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.

Figures & tables

Figure 1: Left (algorithm): GRPO with global baseline can cause unstable gradient norm; Dr. MAS with per-agent normalization stabilizes the training of MAS. Right (system): Existing RL frameworks mainly support single LLM actors, while Dr. MAS enables co-training of multiple LLM actors.

Figure 2: Overview of multi-agent LLM RL framework. A multi-agent orchestrator manages distributed rollouts, agents are mapped to LLM worker groups with optional LLM sharing, and a shared resource pool schedules actor backends for efficient inference and per-model optimization.

Benchmark	Single-Agent		Multi-Agent & LLM Sharing				Multi-Agent & LLM Non-Sharing
	GRPO		GRPO		Dr. MAS		GRPO		Dr. MAS
	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16
Qwen3-4B
AIME’24	38.8	63.3	39.3	64.0	$\text{39.3}_{\smash{\scriptsize{\color[rgb]{0.5898,0.5898,0.5898}0.0}\phantom{00}}}$	$\text{63.3}_{\smash{\scriptsize{\color[rgb]{0.8516,0.5508,0.2734}\downarrow\!0.7}\phantom{0}}}$	42.7	73.3	$\text{46.9}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!4.2}}}$	$\text{80.0}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!6.7}}}$
AIME’25	33.1	56.7	31.4	53.3	$\text{38.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!6.7}\phantom{0}}}$	$\text{63.3}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!10.0}}}$	35.6	63.3	$\text{38.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.5}}}$	$\text{66.7}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!3.4}}}$
AMC’23	83.5	95.0	85.6	95.0	$\text{87.3}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!1.7}\phantom{0}}}$	$\text{95.0}_{\smash{\scriptsize{\color[rgb]{0.5898,0.5898,0.5898}0.0}\phantom{00}}}$	83.5	95.0	$\text{89.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!6.0}}}$	$\text{97.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.5}}}$

Table 1: Math results on Qwen3-4B/8B. We report the avg@16 and pass@16 of single-agent training with GRPO and multi-agent training under LLM sharing/non-sharing, using vanilla GRPO and Dr. MAS. Subscripts for Dr. MAS denote

\Delta

over the vanilla GRPO under the same multi-agent setting.

Benchmark	Single-Agent		Multi-Agent & LLM Sharing				Multi-Agent & LLM Non-Sharing
	GRPO		GRPO		Dr. MAS		GRPO		Dr. MAS
	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16
Qwen2.5-3B
NQ	40.6	54.7	41.0	59.0	$\text{43.8}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.8}}}$	$\text{58.5}_{\smash{\scriptsize{\color[rgb]{0.8516,0.5508,0.2734}\downarrow\!0.5}}}$	43.8	54.5	$\text{44.6}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.8}\phantom{0}}}$	$\text{58.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!3.6}\phantom{0}}}$
TriviaQA	58.1	68.8	57.9	68.4	$\text{61.7}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!3.8}}}$	$\text{70.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!1.7}}}$	60.6	70.8	$\text{61.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.5}\phantom{0}}}$	$\text{71.7}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.9}\phantom{0}}}$
PopQA	44.2	49.6	43.2	58.0	$\text{45.0}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!1.8}}}$	$\text{57.6}_{\smash{\scriptsize{\color[rgb]{0.8516,0.5508,0.2734}\downarrow\!0.4}}}$	45.6	54.5	$\text{46.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.9}\phantom{0}}}$	$\text{57.4}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.9}\phantom{0}}}$

Table 2: Multi-turn search QA results on Qwen2.5-3B/7B. We report the avg@16 and pass@16 of single-agent training with GRPO and multi-agent training under LLM sharing/non-sharing, using vanilla GRPO and Dr. MAS. Subscripts for Dr. MAS denote

\Delta

over the vanilla GRPO under the same multi-agent setting.

Figure 3: Comparison of training accuracy and gradient norm between GRPO and Dr. MAS. The results are recorded during multi-agent RL post-training for three-agent search orchestration under LLM non-sharing (Qwen2.5-3B).

Metric	Normalization Configuration
Metric	$(\mu,\sigma)$	$(\mu_{k},\sigma)$	$(\mu,\sigma_{k})$	$(\mu_{k},\sigma_{k})$
avg@16	28.0	$\text{39.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!11.1}}}$	$\text{42.9}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!14.9}}}$	$\text{43.8}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!15.8}}}$
pass@16	40.5	$\text{53.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!13.0}}}$	$\text{57.6}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!17.1}}}$	$\text{58.3}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!17.8}}}$

Table 3: Ablation study of different advantage normalization configurations on the search task. We report the avg@16 and pass@16 across all datasets. Subscripts denote

\Delta

over the vanilla GRPO.

Figure 4: Performance and efficiency comparison between homogeneous (all 7B models) and heterogeneous (7B for Verifier, 3B for Search/Answer) model assignment on search tasks. Token counts are the average tokens per trajectory for each agent. Cost (

) is estimated using OpenRouter market prices (7B:

0.30/M tokens, 3B: $0.06/M tokens) and reported as the total inference cost over the full test set (51.7k samples).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Illustration of the orchestrations. Left : Math orchestration uses a two-agent loop, where a solver proposes candidate solutions and a verifier evaluates and either approves or requests refinement. Right : Multi-turn search orchestration uses a hierarchical three-agent pipeline, where a top-level verifier selectively invokes either a search agent to retrieve external information or an answer agent to produce the final result.

Figure 6: Comparison of training accuracy and gradient norm between GRPO and Dr. MAS. The results are recorded during multi-agent RL post-training for two-agent math orchestration under LLM non-sharing (Qwen3-4B).

Figure 7: Training dynamics of the three-agent search task (Qwen2.5-7B, non-sharing). Vanilla GRPO suffers from serious gradient spikes that lead to “NaN” on the search agent, whereas Dr. MAS maintains stable gradients and converges effectively.

Figure 8: Training curves for different advantage normalization variants in the ablation study.

Method	Setting	AIME’24	AIME’25	OlympiadBench
Stronger-MAS	LLM sharing	50.0	35.2	56.8
Dr. MAS	LLM sharing	54.8	39.4	59.3
Stronger-MAS	LLM non-sharing	57.0	40.0	56.6
Dr. MAS	LLM non-sharing	44.6	41.5	59.0

Appendix

Table 4: Comparison with Stronger-MAS on Qwen3-8B math reasoning benchmarks. Stronger-MAS results are taken from its official paper [ 67 ] .

Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Average
GSPO	42.6	58.6	46.2	31.2	28.9	8.3	22.4	34.0
GSPO $+$ Dr. MAS	44.2	60.8	47.0	35.8	35.1	10.2	26.4	37.1

Appendix

Table 5: Orthogonality of Dr. MAS to GSPO on Qwen2.5-3B under the LLM non-sharing setting for multi-turn search. The results show the avg@16 scores.

Strategy	Model Configuration	GPU Consumption	Est. Time
LLM sharing	$1\times$ Qwen2.5-3B	$4\times$ H100 (95GB)	$\sim$ 12h
LLM non-sharing	$3\times$ Qwen2.5-3B	$4\times$ H100 (95GB)	$\sim$ 13h
LLM sharing	$1\times$ Qwen2.5-7B	$8\times$ H100 (95GB)	$\sim$ 25h
LLM non-sharing	$3\times$ Qwen2.5-7B	$8\times$ H100 (95GB)	$\sim$ 26h
Heterogeneous	$2\times$ Llama-3.2-3B $+1\times$ Qwen2.5-7B	$8\times$ H100 (95GB)	$\sim$ 16h

Appendix

Table 6: Resource usage on the search task. LLM non-sharing uses the same GPU budget as LLM sharing.

Strategy	Model Configuration	GPU Consumption	Est. Time
LLM sharing	$1\times$ Qwen3-4B	$4\times$ H100 (95GB)	$\sim$ 35h
LLM non-sharing	$2\times$ Qwen3-4B	$4\times$ H100 (95GB)	$\sim$ 38h
LLM sharing	$1\times$ Qwen3-8B	$8\times$ H100 (95GB)	$\sim$ 37h
LLM non-sharing	$2\times$ Qwen3-8B	$8\times$ H100 (95GB)	$\sim$ 42h

Appendix

Table 7: Resource usage on the math task. LLM non-sharing does not require additional GPUs.

Explore similar work

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs
May 22, 2026Yifan Zeng, Yiran Wu, Yaolun Zhang +4Large Language Model Workflows Multi-Agent Reinforcement Learning
1Oregon State University · 2Pennsylvania State University · 3Adobe Inc. +1
UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems
May 26, 2026Yiqun Chen, Wei Yang, Erhan Zhang +14Multi-Agent Reinforcement Learning Offline Reinforcement Learning
Renmin University of China · Xiaohongshu Inc.
TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination
May 1, 2026Yi Xie, Siao Liu, Falong Fan +3Multi-Agent Large Language Model Systems Teaming
Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1

May 22, 2026, cs.AIJ/K move · Enter open · S save

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

Yifan Zeng, Yiran Wu, Yaolun Zhang, Wentian Zhao+3

1Oregon State University · 2Pennsylvania State University · 3Adobe Inc.+1

Large Language Model Workflows Multi-Agent Reinforcement Learning Agentic Workflow Design Trade-Off

Open paper AskPDF arXiv

Multi-agent LLM workflows route inference through specialized roles to lift end-task accuracy, but jointly training those roles with reinforcement learning is unstable in ways that are poorly understood. We study when end-to-end RL training of multi-agent LLM workflows improves over their base models, comparing Shared-Policy training, where all roles update one policy, with Isolated-Policy training, where each role has its own parameters. Our experimental matrix spans Eval-Opt, Voting, and Orch-Workers workflows, math and code tasks, and three model scales (0.6B, 1.7B, 4B). We find that multi-agent RL usually improves over base models, but gains depend jointly on workflow, task, and scale, not on policy sharing alone. Isolated-Policy tends to reach higher peak accuracy yet more often falls off a terminal accuracy cliff, while Shared-Policy training does not eliminate failure; it redistributes failure into qualitatively different patterns. We then explain the strongest of these patterns through role-level gradient dynamics induced by workflow topology and policy routing: under Isolated-Policy, parallel same-role agents on shared prompts amplify per-role gradients and drive terminal degradation in Voting and Orch-Workers workflows; under Shared-Policy, asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role, producing different failure signatures by task and workflow. Together, the empirical map and its underlying mechanisms show that policy sharing routes training pressure through different channels rather than offering uniform stability, making it a design choice with workflow- and task-conditional tradeoffs.

Abstract

Explore similar work

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination