HA-DiPO: Diffusion Policy Optimization for Heterogeneous Cooperative Transport

Xuebin Li; Jinpeng Xiao; Zong Chen; Pierluigi Arpenti; Bruno Siciliano; Jiebao Sun; Yiqun Li

cs.ROOct 24, 2026

HA-DiPO: Diffusion Policy Optimization for Heterogeneous Cooperative Transport

Authors: Xuebin Li, Jinpeng Xiao, Zong Chen, Pierluigi Arpenti, Bruno Siciliano, Jiebao Sun, Yiqun Li

Organizations: School of Mathematics, Harbin Institute of Technology, Harbin, China · State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China · Department of Electrical Engineering and Information Technology, University of Naples Federico II, Naples, Italy

Abstract

Cooperative transport with heterogeneous legged robots requires coordinating different motion capabilities while supporting an unfixed shared payload. We present HA-DiPO (Heterogeneous-Agent Diffusion Policy Optimization), a hierarchical multi-agent reinforcement learning framework under centralized training with decentralized execution (CTDE). The framework decouples team-level coordination from embodiment-specific motion control: at the high level, a centralized critic uses team-value estimates to jointly train independent coordination policies, aligning robot-specific action responses with a shared transport objective; at the low level, embodiment-specific robust motion policies track the coordination commands. The high-level diffusion policies offer more flexible action distributions than single-Gaussian parameterizations, allowing potentially multimodal coordination responses to be represented. The coordination policies learn from scratch through reward-driven simulator interaction without demonstrations. At execution, each robot generates coordination commands using only local robot and payload observations, without partner-state inputs or inter-robot messages. Simulation comparisons show that HA-DiPO achieves higher final mean returns and velocity-tracking success rates, as well as lower velocity errors, than the evaluated baselines. Real-world experiments further demonstrate quadrupedal-wheeled-legged and human-robot cooperative transport.

Figures & tables

Fig. 1: Overview of HA-DiPO. (a) Heterogeneous robots support an unfixed shared payload. (b) Centralized twin critics guide sequential policy updates through a diffusion-path variational objective, using simulator interaction without demonstrations. (c) Local histories and the team command $\mathbf{c}$ form $o_{i}$ for each independent diffusion actor. An embodiment-specific, PPO-pretrained robust policy tracks $a_{i}$ and remains frozen during high-level coordination learning. (d) The conditional network $f_{\theta_{i}}$ drives iterative sampling from Gaussian noise, and $T_{i}$ maps terminal samples to bounded actions. Distribution drawings illustrate representational flexibility. Orange dashed arrows denote training operations.

Panel of Figure 2 — Fig. 2: Geometry-constrained onboard payload perception. (a), (b) Example three-band layout and geometry, with the estimation pipeline below. (c)–(e) Real-robot LiDAR measurements from one static scan: raw points, filtered returns, and cluster centers (white circles) with the constrained rod-axis estimate (magenta).

Panel of Figure 3 — Fig. 3: Cooperative transport in simulation and hardware. (a)–(f) Direct Isaac Lab-to-MuJoCo policy transfer for B2–B2W translation, turning, and disturbance tests; cyan/orange arrows mark motion commands/disturbances. (g), (h) Real-world straight travel without/with an additional payload. (i) Turning without an additional payload. (j), (k) Two configurations during one loaded turn. (l) Human–robot transport with a B2W.

Fig. 4: Learning curves for heterogeneous B2–B2W transport: (a) episode return and (b) mean planar-velocity L2 error. Curves show the mean and shading indicates

\pm

one standard deviation across three seeds, each trained for approximately 50 million environment transitions.

Panel of Figure 5 — Fig. 5: Policy representation and transport value structure. (a) Three-seed action densities on the bimodal bandit; gray denotes the Boltzmann target. (b) Forward-velocity slice of the learned clipped double-Q function at one visited transport state, with other actions fixed. Translation and pivoting form distinct maxima separated by lower values along the indicated straight segment.

Method	Survival $\uparrow$ (0–1)	Tracking success $\uparrow$ (0–1)	$v$ RMSE $\downarrow$ (m/s)
MAPPO	0.966 ${}\pm{}$ 0.041	0.484 ${}\pm{}$ 0.049	0.0720 ${}\pm{}$ 0.0024
HAPPO	0.867 ${}\pm{}$ 0.084	0.532 ${}\pm{}$ 0.059	0.1037 ${}\pm{}$ 0.0142
HASAC	0.857 ${}\pm{}$ 0.146	0.510 ${}\pm{}$ 0.114	0.0777 ${}\pm{}$ 0.0111
HALO	0.599 ${}\pm{}$ 0.167	0.066 ${}\pm{}$ 0.068	0.2188 ${}\pm{}$ 0.0101
HA-DiPO (ours)	1.000 ${}\pm{}$ 0.000	0.935 ${}\pm{}$ 0.074	0.0476 ${}\pm{}$ 0.0036

TABLE I: B2–B2W best-checkpoint evaluation: seven nonzero commands, 448 episodes per seed. Mean

\pm

sample SD across three training seeds; bold marks the best mean.

Fig. 6: Real-world heterogeneous transport: high-level velocity commands and filtered odometry estimates for (a) quadrupedal B2 and (b) wheeled-legged B2W. B2 exhibits a persistent command–velocity offset, whereas B2W follows its varying commands more closely. Longitudinal velocities are expressed in each robot’s body frame.

Multi-Robot Cargo Transportation

See all 16 papers

Heterogeneous Agent Collaborative Reinforcement Learning
Mar 3, 2026Zhixia Zhang, Zixuan Huang, Gongxun Li +10Multi-Agent Collaboration Heterogeneous MARL
Beihang University · Bytedance China · Apple +2
Residual Denoising Enables Sample-Efficient Multi-Agent Coordination on Demand
Sep 26, 2026Dayi Dong, Maulik Bhatt, Aayushi Shrivastava +2Multi-Agent Coordination Robot Policy Adaptation
Department of Mechanical Engineering, University of California Berkeley, Berkeley, CA 94709, USA
CoDiMAD: Diffusion-Based Privileged Distillation for Communication-Free Multi-Robot Coordination
Jul 10, 2026Jiyue Tao, Shunheng Xin, Tongsheng Shen +2Multi-Agent Coordination Multi-Robot Coordination
Robotics and Control Laboratory, School of Advanced Manufacturing and Robotics, and the State Key Laboratory of Turbulence and Complex Systems, Peking University, Beijing, 100871, China · National Innovation Institute of Defense Technology, Beijing 100071, China

Mar 3, 2026J/K move · Enter open · S save

Heterogeneous Agent Collaborative Reinforcement Learning

Zhixia Zhang, Zixuan Huang, Gongxun Li, Huaiyang Wang+9

Beihang University · Bytedance China · Apple+2

Multi-Agent Collaboration Heterogeneous MARL Cooperative MARL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.

Abstract

Multi-Robot Cargo Transportation

Explore similar work

Heterogeneous Agent Collaborative Reinforcement Learning

Residual Denoising Enables Sample-Efficient Multi-Agent Coordination on Demand

CoDiMAD: Diffusion-Based Privileged Distillation for Communication-Free Multi-Robot Coordination