cs.AIOct 23, 2026

PPO vs. SAC: A Comparative Study for Cooperative Motion Control of Dual-Arm Space Robots

Authors: Nasim Fallahiarezoodar, Zheng H. Zhu

Organizations: Department of Earth and Space Science and Engineering, York University, M3J1P3 · Mechanical Engineering, York University, M3J1P3

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The growing population of orbital debris has intensified interest in active debris removal and on-orbit servicing, for which dual-arm space manipulators offer improved grasping stability and load distribution relative to single-arm systems. However, coordinating two manipulators under inter-arm collision avoidance, orientation alignment, and sensing uncertainty remains challenging. This paper develops a coordinated reinforcement learning framework for cooperative motion planning of a dual-arm space manipulator in a target-reaching task with inter-arm collision-avoidance constraints. A centralized state representation combines joint-space, end-effector, and target-relative information, while a multi-objective reward function promotes target-reaching accuracy, orientation alignment, motion smoothness, synchronized task completion, and collision avoidance. Within this framework, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are compared as representative on-policy and off-policy algorithms, and the learned policies are further evaluated for generalization to previously unseen target configurations. Across five random seeds, SAC substantially outperforms PPO within the considered fixed-base simulation scenarios: it reaches an 80% success threshold in every seed (median 1.6M environment steps) whereas PPO does not reach it within a 4M-step budget, and it attains a higher mean success rate (78.6% versus 7.4%) with lower position and orientation errors and tighter bimanual synchronization. Both algorithms degrade on unseen boundary targets, but SAC retains most of its performance while PPO's collapses. The results also expose a failure mode of PPO: under the two-phase shaped reward, it tends to converge to a reward-loitering local optimum rather than committing to the precision task. This behavior highlights the sensitivity of on-policy learning to reward shaping.

Figures & tables

StudyReachCollOrientSmoothSyncNoiseGenAlgorithm
Wu et al. [ 19 ]✓––––––DDPG
Li et al. [ 20 ]✓✓–✓–––DDPG
DA-VIL [ 21 ]✓–––✓––PPO
Al Ali & Zhu [ 18 ]✓✓✓––✓–DDPG
This work✓✓✓✓✓–✓PPO, SAC
Table 1: Objective coverage of representative deep reinforcement learning studies for space and dual-arm manipulators. ✓ = addressed; – = not addressed. Reach: target reaching; Coll: collision avoidance; Orient: orientation alignment; Smooth: motion smoothness; Sync: synchronized bimanual completion; Noise: observation-noise robustness; Gen: generalization to unseen targets. To our knowledge, no prior work jointly addresses this objective set within a single dual-arm space-manipulator formulation.
Figure 1: Dual-arm space manipulator and simulation environment used in this study: (a) the simulation environment, showing the spacecraft base, the two four-joint arms with two-finger grippers, and the right (red) and left (blue) targets; (b) the kinematic schematic of the eight joint frames and the desired target headings within the common reachable region.
Figure 2: Bimanual reachability map of the dual-arm space manipulator.
ParameterPPOSAC
Learning rate1×10−41\times 10^{-4} (decayed)3×10−43\times 10^{-4}
Discount factor γ\gamma0.990.97
Network architecture[512,512][512,512][512,512,256][512,512,256]
Batch size512256
Rollout length nstepsn_{\text{steps}}2048N/A
Clip range0.15 (decayed)N/A
Table 2: Final training hyperparameters for PPO and SAC. GAE: generalized advantage estimation.
Figure 3: Training convergence of PPO and SAC: deterministic success rate versus environment steps (median with bootstrap 95% CI over five seeds).
Figure 4: Per-seed deterministic success rate versus environment steps for all five seeds of PPO and SAC (faint lines: raw checkpoints; bold lines: exponential moving average, weight 0.8).
MetricSACPPO
Success rate (%)78.6 [72.6, 84.2]7.4 [3.2, 11.4]
Final position error (cm)2.24 [1.80, 3.54]4.68 [4.25, 5.95]
Final heading error (deg)11.9 [11.3, 16.3]24.0 [21.4, 42.2]
Collision rate (%)4.2 [3.0, 5.0]6.4 [3.8, 9.0]
Episode length (steps)123 [101, 135]569 [500, 591]
Inter-arm completion gap (steps)7.9 [5.8, 11.2]77.3 [30.0, 127.7]
Table 3: In-distribution task performance over five seeds (100 episodes per seed) [bootstrap 95% CI]; success and collision rates are across-seed means, the remaining metrics across-seed medians. Bold marks the better value.
Figure 5: Per-objective task performance for PPO and SAC, in-distribution and on unseen boundary targets (bootstrap 95% CI over five seeds; the collision rate is an across-seed mean, the other metrics across-seed medians): (a) reaching accuracy, (b) orientation alignment, (c) collision avoidance, and (d) bimanual synchronization.
MetricSACPPO
Success rate (%)68.6 [60.0, 76.8]2.6 [0.2, 6.0]
Final position error (cm)3.83 [2.72, 5.33]7.32 [6.06, 9.43]
Final heading error (deg)18.5 [12.5, 22.4]31.2 [27.1, 49.3]
Collision rate (%)7.4 [4.2, 12.4]11.2 [6.8, 16.0]
Table 4: Generalization to unseen boundary targets over five seeds [bootstrap 95% CI]; success and collision rates are across-seed means, the remaining metrics across-seed medians. Bold marks the better value.
Figure 6: Success rate of PPO and SAC on in-distribution and unseen boundary targets (across-seed mean with bootstrap 95% CI over five seeds).
More in this topic

Space Robotics

Space robotics develops robots and autonomy for exploring, assembling, and maintaining spacecraft and planetary environments.

See all 71 papers

Explore similar work

CardsList
May 25, 2026

Prior Policy Guided Dual-Agent Coordinated Manipulation Planning of Spacecraft-Manipulator System

The strong dynamic coupling between the manipulator and the base poses a significant challenge to maintaining spacecraft attitude stability, potentially compromising mission safety. In this paper, we propose a Dual-Agent Coordinated Manipulation Planning (DACMP) framework that simultaneously achieves high-precision end-effector pose reaching for a 6-DoF space manipulator and attitude stabilization of the base spacecraft. To enhance learning efficiency, we present a prior policy-guided Deep Reinforcement Learning algorithm incorporating the Timestep-level Expert Switching Guidance (TESG) mechanism, thereby promoting global convergence and improving task success rates. Extensive experiments demonstrate that DACMP significantly outperforms baseline DRL algorithms in terms of task success rate and control precision. Furthermore, the robustness of DACMP is validated under various challenging scenarios, including system constraints, environmental disturbances, and perception uncertainties. The code and simulation configurations are available on GitHub: https://github.com/HIT-YuhuiHu/DACMP.

Yuhui Hu, Dong Zhou, Kaihong Ouyang +3

School of Astronautics, Harbin Institute of Technology, Harbin, 150001, China · School of Automation, Chongqing University, Chongqing, 400044, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, 730000, China

Robotic ControlSpacecraft Attitude DeterminationReinforcement LearningMulti-Agent ControlRobotic Manipulation
AskPDF
Oct 13, 2026

CALM: Decentralized Control of Cooperative Aerial Manipulation with Multi-Agent Reinforcement Learning

Cooperative aerial manipulation is difficult under wind, payload motion, and noisy goal broadcasts because multiple Unmanned Aerial Vehicles (UAVs) are coupled through a shared payload. We present CALM (Cooperative Aerial Learned Manipulation), a decentralized controller for aerial manipulators carrying a basket while tracking a moving ground robot. CALM combines a parameter-shared Multi-Agent Reinforcement Learning (MARL) policy, trained using Multi-Agent Proximal Policy Optimization (MAPPO), with goal estimation, per-UAV Extended State Observers (ESOs), basket-mass estimation, and anticipatory formation-yaw feedforward. The anticipatory formation-yaw feedforward uses estimated goal motion to rotate the formation before position error develops, reducing basket-yaw lag without adding net commanded force. A disturbance-conditioned critic supports centralized training while execution remains decentralized. Across held-out trajectories, CALM achieves lower mean position and orientation errors than all evaluated baselines, including the matched arm baseline (7.68 versus 10.95 cm and 3.3 versus 4.7∘^\circ), the cable-suspended realization, and two model-based controllers. Under unseen full-information-path degradation, CALM remains statistically comparable or superior to the arm baseline. Videos are available at https://petreric.github.io/CALM/.

Jiaxi Huo, Petre Ricioppo, Fausto Francesco Lizzio

Department of Mechanical and Aerospace Engineering, Politecnico di Torino, Torino, Italy

Aerial ManipulationDecentralized MARLMulti-Robot Formation ControlMulti-Agent Reinforcement Learning
AskPDF
Jul 24, 2026

Embodying Multi-Hand Manipulation Policies by Searching the Assignment and Null Spaces

Learned manipulation policies increasingly predict motions for abstract "hands" and are attractive in practice because they rely on easily collected demonstrations and transfer across robot platforms. Executing these trajectories on multi-arm robots, however, is not trivial. Multi-hand policy outputs must be assigned to physical arms, each arm must realize a configuration-space motion that tracks its prescribed end-effector trajectory, and all arms must respect kinematic limits and avoid collisions. In the absence of algorithms that directly address this problem, practitioners typically extend single-arm inverse-kinematics (IK) pipelines in an ad hoc way, with no guarantees of feasibility or safety. In this work, we close this execution gap with a search-based framework that is theoretically complete for grounding policy-generated multi-hand trajectories onto physical multi-arm systems. Building on Conflict-Based Search, our method explicitly searches over both the discrete assignment of trajectories to arms and the continuous Jacobian null spaces of redundant manipulators, using redundancy to avoid inter-arm collisions while tracking the prescribed motions. This unified treatment of assignment and null-space motion yields a practically efficient planner that safely realizes coordinated manipulation-policy outputs on multi-arm robots. See omcbsa.github.io for more.

Yorai Shaoul, Jiaoyang Li, Maxim Likhachev

The Robotics Institute, Carnegie Mellon University

Multi-Robot Motion PlanningConstrained Motion PlanningBimanual Robotic ManipulationConflict-Based SearchMotion PlanningInverse Kinematics
AskPDF