HA-DiPO: Diffusion Policy Optimization for Heterogeneous Cooperative Transport
Authors: Xuebin Li, Jinpeng Xiao, Zong Chen, Pierluigi Arpenti, Bruno Siciliano, Jiebao Sun, Yiqun Li
Organizations: School of Mathematics, Harbin Institute of Technology, Harbin, China · State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China · Department of Electrical Engineering and Information Technology, University of Naples Federico II, Naples, Italy
Cooperative transport with heterogeneous legged robots requires coordinating different motion capabilities while supporting an unfixed shared payload. We present HA-DiPO (Heterogeneous-Agent Diffusion Policy Optimization), a hierarchical multi-agent reinforcement learning framework under centralized training with decentralized execution (CTDE). The framework decouples team-level coordination from embodiment-specific motion control: at the high level, a centralized critic uses team-value estimates to jointly train independent coordination policies, aligning robot-specific action responses with a shared transport objective; at the low level, embodiment-specific robust motion policies track the coordination commands. The high-level diffusion policies offer more flexible action distributions than single-Gaussian parameterizations, allowing potentially multimodal coordination responses to be represented. The coordination policies learn from scratch through reward-driven simulator interaction without demonstrations. At execution, each robot generates coordination commands using only local robot and payload observations, without partner-state inputs or inter-robot messages. Simulation comparisons show that HA-DiPO achieves higher final mean returns and velocity-tracking success rates, as well as lower velocity errors, than the evaluated baselines. Real-world experiments further demonstrate quadrupedal-wheeled-legged and human-robot cooperative transport.
Figures & tables
Fig. 1: Overview of HA-DiPO. (a) Heterogeneous robots support an unfixed shared payload. (b) Centralized twin critics guide sequential policy updates through a diffusion-path variational objective, using simulator interaction without demonstrations. (c) Local histories and the team command c form oi for each independent diffusion actor. An embodiment-specific, PPO-pretrained robust policy tracks ai and remains frozen during high-level coordination learning. (d) The conditional network fθi drives iterative sampling from Gaussian noise, and Ti maps terminal samples to bounded actions. Distribution drawings illustrate representational flexibility. Orange dashed arrows denote training operations.
Fig. 2: Geometry-constrained onboard payload perception. (a), (b) Example three-band layout and geometry, with the estimation pipeline below. (c)–(e) Real-robot LiDAR measurements from one static scan: raw points, filtered returns, and cluster centers (white circles) with the constrained rod-axis estimate (magenta).
Fig. 3: Cooperative transport in simulation and hardware. (a)–(f) Direct Isaac Lab-to-MuJoCo policy transfer for B2–B2W translation, turning, and disturbance tests; cyan/orange arrows mark motion commands/disturbances. (g), (h) Real-world straight travel without/with an additional payload. (i) Turning without an additional payload. (j), (k) Two configurations during one loaded turn. (l) Human–robot transport with a B2W.
Fig. 4: Learning curves for heterogeneous B2–B2W transport: (a) episode return and (b) mean planar-velocity L2 error. Curves show the mean and shading indicates ± one standard deviation across three seeds, each trained for approximately 50 million environment transitions.
Fig. 5: Policy representation and transport value structure. (a) Three-seed action densities on the bimodal bandit; gray denotes the Boltzmann target. (b) Forward-velocity slice of the learned clipped double-Q function at one visited transport state, with other actions fixed. Translation and pivoting form distinct maxima separated by lower values along the indicated straight segment.
Method
Survival ↑ (0–1)
Tracking success ↑ (0–1)
v RMSE ↓ (m/s)
MAPPO
0.966 ± 0.041
0.484 ± 0.049
0.0720 ± 0.0024
HAPPO
0.867 ± 0.084
0.532 ± 0.059
0.1037 ± 0.0142
HASAC
0.857 ± 0.146
0.510 ± 0.114
0.0777 ± 0.0111
HALO
0.599 ± 0.167
0.066 ± 0.068
0.2188 ± 0.0101
HA-DiPO (ours)
1.000 ± 0.000
0.935 ± 0.074
0.0476 ± 0.0036
TABLE I: B2–B2W best-checkpoint evaluation: seven nonzero commands, 448 episodes per seed. Mean ± sample SD across three training seeds; bold marks the best mean.
Fig. 6: Real-world heterogeneous transport: high-level velocity commands and filtered odometry estimates for (a) quadrupedal B2 and (b) wheeled-legged B2W. B2 exhibits a persistent command–velocity offset, whereas B2W follows its varying commands more closely. Longitudinal velocities are expressed in each robot’s body frame.
We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.
Pretrained robot policies offer strong manipulation skills but are typically limited to single-agent settings, where a robot acts in isolation. In this work, we study how to adapt pretrained single-agent diffusion policies to multi-agent settings using minimal collaborative data, co-optimizing for two key objectives: high coordination performance and single-agent skill retention. To this end, we introduce ALTER, an adaptation method for coordination on demand: the adapted policy coordinates with other robots when deployed in a team while remaining capable of acting independently when operating alone. Execution is decentralized: each robot acts only on its own visual observations, without explicit inter-agent communication. Our method trains a coordination head that predicts a residual denoiser to transform single-agent behavior into coordinated multi-agent behavior when necessary while also preserving single-agent capabilities. To preserve single-agent capabilities, we augment a small number of collaborative demonstrations with self-distilled data generated by the base policy during training of the residual denoiser. In simulation, ALTER achieves higher coordination success over our baselines while retaining much higher source-skill retention. In our hardware experiments, we find similar trends where ALTER better co-optimizes for coordination success and single-agent skill retention than the baselines.
Dayi Dong, Maulik Bhatt, Aayushi Shrivastava +2
Department of Mechanical Engineering, University of California Berkeley, Berkeley, CA 94709, USA
Decentralized multi-robot coordination under partial observability remains challenging, especially in communication-free settings where agents must act solely from local sensor observations. Privileged policy distillation provides a promising approach by transferring knowledge from a globally informed oracle to sensor-constrained students. However, in multi-agent systems, the same local observation may correspond to multiple global configurations requiring qualitatively different cooperative actions, making the conditional action distribution inherently multi-modal. Standard deterministic distillation collapses these modes to their mean, often yielding invalid or hesitant actions. To address this issue, we propose CoDiMAD, a three-stage framework that trains a privileged oracle with MAPPO, constructs an offline dataset of local-observation-oracle-action pairs, and distills the oracle into decentralized students parameterized as conditional denoising diffusion probabilistic models. By approximating the conditional oracle-action distribution through the diffusion reverse process, CoDiMAD samples decisive actions from coherent coordination modes rather than averaging across them. Theoretical analysis characterizes the mode-averaging failure of deterministic distillation and the distributional recovery property of diffusion-based distillation. Experiments on three cooperative tasks show that CoDiMAD consistently outperforms direct local MARL and deterministic distillation baselines. The source code will be made publicly available upon acceptance.
Jiyue Tao, Shunheng Xin, Tongsheng Shen +2
Robotics and Control Laboratory, School of Advanced Manufacturing and Robotics, and the State Key Laboratory of Turbulence and Complex Systems, Peking University, Beijing, 100871, China · National Innovation Institute of Defense Technology, Beijing 100071, China