Heterogeneous Agent Collaborative Reinforcement Learning

Zhixia Zhang; Zixuan Huang; Gongxun Li; Huaiyang Wang; Chengyi Yuan; Xin Xia; Deqing Wang; Fuzhen Zhuang

cs.LGMar 3, 2026

Heterogeneous Agent Collaborative Reinforcement Learning

Authors: Zhixia Zhang, Zixuan Huang, Gongxun Li, Huaiyang Wang, Chengyi Yuan, Xin Xia, Deqing Wang, Fuzhen Zhuang, +5 more

Organizations: Beihang University · Bytedance China · Apple · Tsinghua University · Peking University

Abstract

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.

Figures & tables

Figure 1: In HACPO, shared rollouts from multiple heterogeneous agents are leveraged for collaborative training. Built upon vanilla RL Optimization, HACPO introduces four algorithmic innovations to mitigate capability and policy distribution discrepancy.

Qwen3-4B and Qwen3-4B-Instruct ( Heterogeneous State )
Model	MATH-500	MATH	GSM8K	AIME2025	AMC23	Minerva	Olympiad	AVG
4B	0.802	0.836	0.907	0.335	0.65	0.39	0.524	0.635
4B (GRPO)	0.88	0.889	0.918	0.582	0.775	0.386	0.592	0.717
4B (GSPO)	0.854	0.87	0.925	0.485	0.675	0.412	0.564	0.684
4B (GSPO $\times$ 2)	0.876	0.875	0.923	0.522	0.675	0.39	0.579	0.691
4B (Naive)	0.728	0.737	0.891	0.378	0.6	0.353	0.394	0.583

Table 1: Main results across three heterogeneity settings (state, size and model). We compare our method against Standard Single-Agent Baselines (GRPO, GSPO), a Resource-Equivalent Baseline (GSPO

\times

2) and a Naive multi-agent rollout share baseline(Naive). The content in the brackets represents a comparison with GSPO

\times

Panel of Figure 2 — Figure 2: Training Curves of HACPO, GSPO and GSPO $\times$ 2

Model	MATH-500	MATH	GSM8K	AIME2025	AMC23	Minerva	Olympiad	AVG
1.7B(HACPO - Adv)	0.696	0.659	0.825	0.126	0.375	0.261	0.313	0.465
1.7B(HACPO)	0.69	0.674	0.822	0.225	0.45	0.279	0.314	0.493
4B(HACPO - Adv)	0.774	0.771	0.912	0.308	0.55	0.348	0.442	0.586
4B(HACPO)	0.808	0.801	0.903	0.267	0.575	0.386	0.467	0.601

Table 2: Ablation of Advantage Estimator

Model	MATH-500	math	gsm8k	aime2025	ACM23	minerva	olympiad	AVG
1.7B(HACPO - $\omega$ )	0.666	0.657	0.806	0.105	0.425	0.25	0.324	0.462
1.7B(HACPO)	0.69	0.674	0.822	0.225	0.45	0.279	0.314	0.493
4B(HACPO - $\omega$ )	0.803	0.797	0.902	0.261	0.55	0.401	0.475	0.6
4B(HACPO)	0.808	0.801	0.903	0.267	0.575	0.386	0.467	0.601

Table 3: Ablation of Model Capabilities Discrepancy Coefficient

Panel of Figure 3 — Figure 3: Ablation of Stepwise Clipping (MATH500)

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model Combination	$\alpha$	1- $\delta$	$\delta_{step}$
Qwen3-4B and Qwen3-4B-Instruct	3.0	0.8	0.01
Qwen3-1.7B-Base and Qwen3-4B-Base	1.0	0.8	0.025
Qwen3-4B-Base and Qwen3-8B-Base	3.0	0.8	0.025
Llama3.2-1B-Instruct and Llama3.2-3B-Instruct	1.0	0.9	0.01
Qwen3-1.7B-Base and Llama3.2-1B-Instruct	1.0	0.8	0.025
Qwen3-4B-Base and Llama3.2-3B-Instruct	1.0	0.8	0.025

Appendix

Table 4: The Details of Hyperparameter

Model	mean	max	min	range
$s^{homo}$	1.00002	1.00020	0.99960	0.00060
$s^{hete}$	0.89550	0.93615	0.86198	0.07417

Appendix

Table 5:

s_{t,i}^{(1,1)}

and

s_{t,i}^{(1,2)}

of Qwen3-1.7B-Base in all steps

Method	0	1	42	1337	3407
Qwen3-1.7B-Base
HACPO	0.656	0.656	0.664	0.656	0.644
GSPO	0.622	0.614	0.622	0.622	0.618
GSPO $\times$ 2	0.624	0.628	0.624	0.626	0.62
Qwen3-4B-Base
HACPO	0.762	0.78	0.792	0.772	0.774

Appendix

Table 6: Seed Experiments

Method	Qwen3-1.7B-Base	Qwen3-4B-Base	Total Time
GSPO	1h 31m	2h 38m	4h 9m
GSPO updates $\times$ 2	2h 6m	2h 59m	5h 5m
GSPO $\times$ 2	2h 43m	4h 1m	6h 44m
HACPO	Overall 5h 31m

Appendix

Table 7: Runtime Comparison

Method	Qwen3-1.7B-Base	Qwen3-4B-Base
GSPO	76.8%	82.7%
GSPO updates $\times$ 2	76.6%	80.3%
GSPO $\times$ 2	82.2%	84.2%
HACPO	Overall 86.0%

Appendix

Table 8: GPU Memory Usage

Model	MATH-500	math	gsm8k	aime2025	AMC23	minerva	olympiad	AVG
Qwen3-4B-Base and Qwen3-8B-Base
4B-Base	0.61	0.676	0.445	0.1	0.4	0.308	0.347	0.412
4B-Base(GRPO)	0.796	0.788	0.885	0.307	0.475	0.349	0.454	0.579
4B-Base(GSPO)	0.782	0.787	0.877	0.25	0.525	0.368	0.46	0.578
4B-Base(GSPO $\times$ 2)	0.756	0.794	0.873	0.208	0.55	0.382	0.463	0.575
4B-Base(Naive)	0.734	0.712	0.895	0.143	0.55	0.342	0.354	0.526

Appendix

Table 9: Additional Experimental Results

Explore similar work

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning
Jun 29, 2026Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara +4Multi-Agent System Optimization Multi-Agent Coordination
1KAIST · 2KRAFTON · University of Toronto +2
HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization
Mar 4, 2026Hao Zhang, Yaru Niu, Yikai Wang +2Decentralized MARL Heterogeneous MARL
ETAIC Lab, Department of Electrical Engineering, University of Texas at Arlington, USA · Safe AI Lab, Department of Mechanical Engineering, Carnegie Mellon University, USA.
Hierarchical Lead Critic based Multi-Agent Reinforcement Learning
Feb 25, 2026David Eckel, Henri MeeßMulti-Agent Reinforcement Learning Cooperative MARL
Fraunhofer Institute IVI, Ingolstadt, Germany

Jun 29, 2026J/K move · Enter open · S save

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning

Daiki E. Matsunaga, Junho Na, Tri Wahyu Guntara, Scott Sanner+3

1KAIST · 2KRAFTON · University of Toronto+2

Multi-Agent System Optimization Multi-Agent Coordination Decentralized MARL Policy Learning Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions. The belief acts as the coordination mechanism which ties the independent per-agent updates into a joint gradient step. We evaluate ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows.

Abstract

Explore similar work

ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning

HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning