Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

Lang Feng; Longtao Zheng; Shuo He; Fuxiang Zhang; Bo An

cs.LGFeb 9, 2026

Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

Authors: Lang Feng, Longtao Zheng, Shuo He, Fuxiang Zhang, Bo An

Organizations: Nanyang Technological University, Singapore

Abstract

Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6% avg@16 and +4.6% pass@16 on math, and +15.2% avg@16 and +13.1% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.

Figures & tables

Figure 1: Left (algorithm): GRPO with global baseline can cause unstable gradient norm; Dr. MAS with per-agent normalization stabilizes the training of MAS. Right (system): Existing RL frameworks mainly support single LLM actors, while Dr. MAS enables co-training of multiple LLM actors.

Figure 2: Overview of multi-agent LLM RL framework. A multi-agent orchestrator manages distributed rollouts, agents are mapped to LLM worker groups with optional LLM sharing, and a shared resource pool schedules actor backends for efficient inference and per-model optimization.

Benchmark	Single-Agent		Multi-Agent & LLM Sharing				Multi-Agent & LLM Non-Sharing
	GRPO		GRPO		Dr. MAS		GRPO		Dr. MAS
	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16
Qwen3-4B
AIME’24	38.8	63.3	39.3	64.0	$\text{39.3}_{\smash{\scriptsize{\color[rgb]{0.5898,0.5898,0.5898}0.0}\phantom{00}}}$	$\text{63.3}_{\smash{\scriptsize{\color[rgb]{0.8516,0.5508,0.2734}\downarrow\!0.7}\phantom{0}}}$	42.7	73.3	$\text{46.9}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!4.2}}}$	$\text{80.0}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!6.7}}}$
AIME’25	33.1	56.7	31.4	53.3	$\text{38.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!6.7}\phantom{0}}}$	$\text{63.3}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!10.0}}}$	35.6	63.3	$\text{38.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.5}}}$	$\text{66.7}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!3.4}}}$
AMC’23	83.5	95.0	85.6	95.0	$\text{87.3}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!1.7}\phantom{0}}}$	$\text{95.0}_{\smash{\scriptsize{\color[rgb]{0.5898,0.5898,0.5898}0.0}\phantom{00}}}$	83.5	95.0	$\text{89.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!6.0}}}$	$\text{97.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.5}}}$

Table 1: Math results on Qwen3-4B/8B. We report the avg@16 and pass@16 of single-agent training with GRPO and multi-agent training under LLM sharing/non-sharing, using vanilla GRPO and Dr. MAS. Subscripts for Dr. MAS denote

\Delta

over the vanilla GRPO under the same multi-agent setting.

Benchmark	Single-Agent		Multi-Agent & LLM Sharing				Multi-Agent & LLM Non-Sharing
	GRPO		GRPO		Dr. MAS		GRPO		Dr. MAS
	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16	avg@16	pass@16
Qwen2.5-3B
NQ	40.6	54.7	41.0	59.0	$\text{43.8}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.8}}}$	$\text{58.5}_{\smash{\scriptsize{\color[rgb]{0.8516,0.5508,0.2734}\downarrow\!0.5}}}$	43.8	54.5	$\text{44.6}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.8}\phantom{0}}}$	$\text{58.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!3.6}\phantom{0}}}$
TriviaQA	58.1	68.8	57.9	68.4	$\text{61.7}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!3.8}}}$	$\text{70.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!1.7}}}$	60.6	70.8	$\text{61.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.5}\phantom{0}}}$	$\text{71.7}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.9}\phantom{0}}}$
PopQA	44.2	49.6	43.2	58.0	$\text{45.0}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!1.8}}}$	$\text{57.6}_{\smash{\scriptsize{\color[rgb]{0.8516,0.5508,0.2734}\downarrow\!0.4}}}$	45.6	54.5	$\text{46.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!0.9}\phantom{0}}}$	$\text{57.4}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!2.9}\phantom{0}}}$

Table 2: Multi-turn search QA results on Qwen2.5-3B/7B. We report the avg@16 and pass@16 of single-agent training with GRPO and multi-agent training under LLM sharing/non-sharing, using vanilla GRPO and Dr. MAS. Subscripts for Dr. MAS denote

\Delta

over the vanilla GRPO under the same multi-agent setting.

Figure 3: Comparison of training accuracy and gradient norm between GRPO and Dr. MAS. The results are recorded during multi-agent RL post-training for three-agent search orchestration under LLM non-sharing (Qwen2.5-3B).

Metric	Normalization Configuration
Metric	$(\mu,\sigma)$	$(\mu_{k},\sigma)$	$(\mu,\sigma_{k})$	$(\mu_{k},\sigma_{k})$
avg@16	28.0	$\text{39.1}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!11.1}}}$	$\text{42.9}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!14.9}}}$	$\text{43.8}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!15.8}}}$
pass@16	40.5	$\text{53.5}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!13.0}}}$	$\text{57.6}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!17.1}}}$	$\text{58.3}_{\smash{\scriptsize{\color[rgb]{0.457,0.2734,0.4922}\uparrow\!17.8}}}$

Table 3: Ablation study of different advantage normalization configurations on the search task. We report the avg@16 and pass@16 across all datasets. Subscripts denote

\Delta

over the vanilla GRPO.

Figure 4: Performance and efficiency comparison between homogeneous (all 7B models) and heterogeneous (7B for Verifier, 3B for Search/Answer) model assignment on search tasks. Token counts are the average tokens per trajectory for each agent. Cost (

) is estimated using OpenRouter market prices (7B:

0.30/M tokens, 3B: $0.06/M tokens) and reported as the total inference cost over the full test set (51.7k samples).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Illustration of the orchestrations. Left : Math orchestration uses a two-agent loop, where a solver proposes candidate solutions and a verifier evaluates and either approves or requests refinement. Right : Multi-turn search orchestration uses a hierarchical three-agent pipeline, where a top-level verifier selectively invokes either a search agent to retrieve external information or an answer agent to produce the final result.

Figure 6: Comparison of training accuracy and gradient norm between GRPO and Dr. MAS. The results are recorded during multi-agent RL post-training for two-agent math orchestration under LLM non-sharing (Qwen3-4B).

Figure 7: Training dynamics of the three-agent search task (Qwen2.5-7B, non-sharing). Vanilla GRPO suffers from serious gradient spikes that lead to “NaN” on the search agent, whereas Dr. MAS maintains stable gradients and converges effectively.

Figure 8: Training curves for different advantage normalization variants in the ablation study.

Method	Setting	AIME’24	AIME’25	OlympiadBench
Stronger-MAS	LLM sharing	50.0	35.2	56.8
Dr. MAS	LLM sharing	54.8	39.4	59.3
Stronger-MAS	LLM non-sharing	57.0	40.0	56.6
Dr. MAS	LLM non-sharing	44.6	41.5	59.0

Appendix

Table 4: Comparison with Stronger-MAS on Qwen3-8B math reasoning benchmarks. Stronger-MAS results are taken from its official paper [ 67 ] .

Method	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	Average
GSPO	42.6	58.6	46.2	31.2	28.9	8.3	22.4	34.0
GSPO $+$ Dr. MAS	44.2	60.8	47.0	35.8	35.1	10.2	26.4	37.1

Appendix

Table 5: Orthogonality of Dr. MAS to GSPO on Qwen2.5-3B under the LLM non-sharing setting for multi-turn search. The results show the avg@16 scores.

Strategy	Model Configuration	GPU Consumption	Est. Time
LLM sharing	$1\times$ Qwen2.5-3B	$4\times$ H100 (95GB)	$\sim$ 12h
LLM non-sharing	$3\times$ Qwen2.5-3B	$4\times$ H100 (95GB)	$\sim$ 13h
LLM sharing	$1\times$ Qwen2.5-7B	$8\times$ H100 (95GB)	$\sim$ 25h
LLM non-sharing	$3\times$ Qwen2.5-7B	$8\times$ H100 (95GB)	$\sim$ 26h
Heterogeneous	$2\times$ Llama-3.2-3B $+1\times$ Qwen2.5-7B	$8\times$ H100 (95GB)	$\sim$ 16h

Appendix

Table 6: Resource usage on the search task. LLM non-sharing uses the same GPU budget as LLM sharing.

Strategy	Model Configuration	GPU Consumption	Est. Time
LLM sharing	$1\times$ Qwen3-4B	$4\times$ H100 (95GB)	$\sim$ 35h
LLM non-sharing	$2\times$ Qwen3-4B	$4\times$ H100 (95GB)	$\sim$ 38h
LLM sharing	$1\times$ Qwen3-8B	$8\times$ H100 (95GB)	$\sim$ 37h
LLM non-sharing	$2\times$ Qwen3-8B	$8\times$ H100 (95GB)	$\sim$ 42h

Appendix

Table 7: Resource usage on the math task. LLM non-sharing does not require additional GPUs.

Explore similar work

May 22, 2026cs.AI

When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs

Multi-agent LLM workflows route inference through specialized roles to lift end-task accuracy, but jointly training those roles with reinforcement learning is unstable in ways that are poorly understood. We study when end-to-end RL training of multi-agent LLM workflows improves over their base models, comparing Shared-Policy training, where all roles update one policy, with Isolated-Policy training, where each role has its own parameters. Our experimental matrix spans Eval-Opt, Voting, and Orch-Workers workflows, math and code tasks, and three model scales (0.6B, 1.7B, 4B). We find that multi-agent RL usually improves over base models, but gains depend jointly on workflow, task, and scale, not on policy sharing alone. Isolated-Policy tends to reach higher peak accuracy yet more often falls off a terminal accuracy cliff, while Shared-Policy training does not eliminate failure; it redistributes failure into qualitatively different patterns. We then explain the strongest of these patterns through role-level gradient dynamics induced by workflow topology and policy routing: under Isolated-Policy, parallel same-role agents on shared prompts amplify per-role gradients and drive terminal degradation in Voting and Orch-Workers workflows; under Shared-Policy, asymmetric per-step gradient mass causes the shared policy to be captured by the dominant role, producing different failure signatures by task and workflow. Together, the empirical map and its underlying mechanisms show that policy sharing routes training pressure through different channels rather than offering uniform stability, making it a design choice with workflow- and task-conditional tradeoffs.

Yifan Zeng, Yiran Wu, Yaolun Zhang +4

1Oregon State University · 2Pennsylvania State University · 3Adobe Inc. +1

Large Language Model Workflows Multi-Agent Reinforcement Learning Agentic Workflow Design Trade-Off

Ask PDF

May 26, 2026cs.AI

UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systems

LLM-based multi-agent systems decompose complex tasks into interacting roles, but most remain manually orchestrated by prompts, tools, and control rules, while agents are rarely optimized through a unified reinforcement learning interface. Existing RL post-training frameworks mainly target single-policy optimization and lack abstractions for user-defined multi-agent workflows, structured interaction, role-specific credit assignment, and configurable parameter sharing. We present UnityMAS-O, a general RL optimization framework for LLM-based multi-agent systems. UnityMAS-O treats the complete workflow as the optimization unit, rather than a single response or policy trajectory. It represents workflows through four first-class objects: logical agent roles, graph trajectories, user-defined rewards, and agent--model mappings. This decouples logical agents from physical model parameters, supporting full sharing, full separation, and partial sharing, with rewards assigned at role, turn, and trajectory levels. UnityMAS-O extends verl with a Ray-based star-topology runtime. A central controller executes workflows, invokes tools, records structured trajectories, and assembles rewards; model-local worker groups handle rollout, buffering, advantage computation, and distributed PPO-style updates. Users can define agents, workflows, model mappings, and rewards without rewriting the optimization infrastructure. We instantiate UnityMAS-O on retrieval-augmented QA, iterative agentic search, and reflective code generation. Across Natural Questions, HotpotQA, and held-out code tasks, multi-agent RL improves manually specified workflows after optimization, with especially large gains for smaller models and strict code all-passed metrics. These results show that UnityMAS-O can serve as a reusable substrate for converting diverse LLM-based multi-agent workflows into trainable multi-agent RL systems.

Yiqun Chen, Wei Yang, Erhan Zhang +14

Renmin University of China · Xiaohongshu Inc.

Multi-Agent Reinforcement Learning Offline Reinforcement Learning Credit Assignment

Ask PDF

May 1, 2026cs.LG

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

Multi-agent LLM systems have shown promise for complex reasoning, yet recent evaluations reveal they often underperform single-model baselines. We identify a structural failure mode in sequential fine-tuning of shared-context teams: updating one agent shifts the team's context distribution, and when subsequent updates are evaluated on cached rollouts, this mismatch compounds. We formalize this as the compounding occupancy shift and prove that stale-occupancy evaluation incurs a penalty that scales quadratically with the number of agents. In contrast, intermediate-occupancy evaluation reduces this to linear scaling. We propose TeamTR, a trust-region framework that resamples trajectories after each component update and enforces per-agent divergence control, yielding rigorous per-update and per-stage improvement lower bounds. Experiments show that TeamTR outperforms single-agent and sequential baselines with 7.1% on average, mitigates coordination regressions, and supports plug-and-play component replacement. Code is available at https://github.com/Yydc/TeamTR.

Yi Xie, Siao Liu, Falong Fan +3

Department of Electrical & Computer Engineering, University of Arizona · Future Science and Engineering College, Soochow University · INSAIT, Sofia University ”St. Kliment Ohridski” +1

Multi-Agent Large Language Model Systems Teaming Model Fine-Tuning Occupancy

Ask PDF