Dual-Quadruped Collaborative Transportation in Narrow Environments via Safe Reinforcement Learning
Authors: Zhezhi Lei, Zhihai Bi, Wenxin Wang, Jun Ma
Organizations: Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Advanced Remanufacturing & Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR), Singapore 637143 · Cheng Kar-Shun Robotics Institute, The Hong Kong University of Science and Technology, Hong Kong SAR, China
Collaborative transportation, where multiple robots collaboratively transport a payload, has garnered significant attention in recent years. While ensuring safe and high-performance inter-robot collaboration is critical for effective task execution, it is difficult to pursue in narrow environments where the feasible region is extremely limited. To address this challenge, we propose a novel approach for dual-quadruped collaborative transportation via safe reinforcement learning (RL). Specifically, we model the task as a fully cooperative constrained Markov game, where collision avoidance is formulated as constraints. We introduce a cost-advantage decomposition method that enforces the sum of team constraints to remain below an upper bound, thereby guaranteeing task safety within an RL framework. Furthermore, we propose a constraint allocation method that assigns shared constraints to individual robots to maximize the overall task reward, encouraging autonomous task-assignment among robots, thereby improving collaborative task performance. Simulation and real-time experimental results demonstrate that the proposed approach achieves superior performance and a higher success rate in dual-quadruped collaborative transportation compared to existing methods.
Figures & tables
Fig. 1 : An overview of our method. 1) Task Environment: The robot team is required to collaboratively transport an object from the initial region to the target region while avoiding collisions. 2) Proposed Framework: We formulate the task as a constrained Markov decision process. For the total reward of the team, we adopt a safe RL approach to sequentially estimate the surrogate returns of R1 and R2, thereby determining the contribution of each member to the overall reward. For the total cost of the team, we employ a constraint allocation method to assign individual cost budgets to each team member. Robots R1 and R2 execute independent policies to accomplish the task in a fully distributed manner. 3) Training Strategy: We use two separate critics to estimate the expected reward and cost, respectively. We adopt a Lagrangian approach in which the total advantage is computed by combining the reward and cost advantages. Then, the residual between the cost and the budget is used to update the Lagrange multiplier. Lastly, the updated policy is subsequently deployed in the simulation environment to collect data for the next iteration.
Method
Explicit Constraint
Constraint Allocation
Independent Policies
Lagrangian
MAPPO
✗
–
✗
–
HAPPO
✗
–
✓
–
UCA
✓
✗
✓
✓
MACPO
✓
✓
✗
✓
Ours
✓
✓
✓
✓
TABLE I : Comparison of Baseline Methods
Fig. 2 : Average episode reward (left) and cost (right) during training. Our method (red curve) achieves high reward while maintaining a low cost, suggesting that it effectively balances safety and efficiency.
Method
Collision Rate [%]
Arrival Rate [%]
Trajectory Straightness
Time Consumption [s]
MAPPO
90.0
10.0
–
–
HAPPO
83.3
16.7
–
–
UCA
6.7
53.3
0.68
26.8
MACPO
0.0
6.7
0.74
32.0
Ours
0.0
100.0
0.95
18.3
TABLE II : Safety and Collaboration Efficiency of Different Methods in the Gate Scenario over 30 Trials ( Trajectory Straightness=Trajectory LengthDisplacement Along Goal Direction )
Fig. 3 : (a): Parameters of the gate scenario. The robot team is required to traverse the environment from the initial region to the target region without collisions. (b): The robot team successfully reaches the target region within the specified time horizon. The red line denotes the trajectory of the payload. (c) & (d): The robot team fails to reach the target region within the specified time horizon. The payload trajectory is distorted. (e) & (f): The robots collide with obstacles during navigation.
Fig. 4 : Robot team collaboration for navigating the gate scenario. From t1 to t3 , the team collaboratively adjusts its formation, and then passes through the narrow region in a front–rear formation.
Fig. 5 : Robot team collaboration in a long and narrow corridor scenario. Robot R1 continuously follows Robot R2 to maintain the formation. From t2 to t4 , the team changes its formation to pass through the corner.
Fig. 6 : Robot team collaboration in a forest scenario. From t1 to t2 , when encountering a junction, the two robots make identical action decisions to avoid conflicts.
Robotic Systems Lab, ETH Zurich. · Centro di Ricerca “Enrico Piaggio”, and Dipartimento di Ingegneria dell’Informazione, Universit`a di Pisa, Pisa, Italy.
Center for Artificial Intelligence and Robotics, Indian Institute of Technology Mandi, India · Department of Mechanical Engineering, Clemson University, Clemson, SC 29630, USA
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · Department of Electronic Engineering, and the Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing, China.