CALM: Decentralized Control of Cooperative Aerial Manipulation with Multi-Agent Reinforcement Learning

Jiaxi Huo; Petre Ricioppo; Fausto Francesco Lizzio

cs.ROOct 13, 2026

CALM: Decentralized Control of Cooperative Aerial Manipulation with Multi-Agent Reinforcement Learning

Authors: Jiaxi Huo, Petre Ricioppo, Fausto Francesco Lizzio

Organizations: Department of Mechanical and Aerospace Engineering, Politecnico di Torino, Torino, Italy

Abstract

Cooperative aerial manipulation is difficult under wind, payload motion, and noisy goal broadcasts because multiple Unmanned Aerial Vehicles (UAVs) are coupled through a shared payload. We present CALM (Cooperative Aerial Learned Manipulation), a decentralized controller for aerial manipulators carrying a basket while tracking a moving ground robot. CALM combines a parameter-shared Multi-Agent Reinforcement Learning (MARL) policy, trained using Multi-Agent Proximal Policy Optimization (MAPPO), with goal estimation, per-UAV Extended State Observers (ESOs), basket-mass estimation, and anticipatory formation-yaw feedforward. The anticipatory formation-yaw feedforward uses estimated goal motion to rotate the formation before position error develops, reducing basket-yaw lag without adding net commanded force. A disturbance-conditioned critic supports centralized training while execution remains decentralized. Across held-out trajectories, CALM achieves lower mean position and orientation errors than all evaluated baselines, including the matched arm baseline (7.68 versus 10.95 cm and 3.3 versus 4.7 $^\circ$ ), the cable-suspended realization, and two model-based controllers. Under unseen full-information-path degradation, CALM remains statistically comparable or superior to the arm baseline. Videos are available at https://petreric.github.io/CALM/.

Figures & tables

Fig. 1: Illustration of three aerial manipulators carrying a shared basket while tracking a ground robot from noisy goal broadcasts under wind and a moving CoM. The lower-right inset shows the hardware design under development.

Panel of Figure 2 — Fig. 2: CALM architecture with per-UAV execution (solid), team-bus signals (blue dashed), training-only parts (amber), and unmeasured disturbances (dash-dot). Orange marks evaluation-only degradation of the full information path (Sec. IV-E ). Symbols are defined in Secs. III – IV .

Policy / low-level rate	100/300 Hz; simulation step 3.33 ms; episode 80 s
Arm joints (actuated)	3-axis shoulder ( $\pm$ 90 ∘ ), elbow (0–180 ∘ ), 3-axis wrist ( $\pm$ 90 ∘ ); two 0.125-m links, 50 g each
Joint effort limits	20 N m shoulder/elbow, 5 N m wrist, 20 N telescoping
Spring / bracket	setpoint PD; spring 200 N m/rad, damping 20; bracket 250 N/m, damping 25, travel $\pm$ 0.10 m
Airframe / total mass	0.602 kg per airframe, $\approx$ 6.57 kg total
Wind mean force	$\leq$ 1.15 N per aerial manipulator and basket (drag-equivalent wind speed $\approx$ 7.1 m/s, curriculum-ramped)
Wind turbulence	OU noise, 0.575-N standard deviation, scaled in sweeps; rate 12 s -1 ; weight 0.9 on a part shared by all bodies

TABLE I: Parameter values.

\mathcal{U}

is a uniform draw, resampled per episode.

Fig. 3: Tracking under nominal disturbance. The top panel shows a representative CALM path colored by position error, with the dashed goal trajectory and commanded (gray) and realized (black) basket yaw. The lower panels show median planar position and yaw errors over 32 trace environments recorded separately from the statistical pool, with interquartile shading. Right markers give scoring-window medians of 3-D position and full orientation error.

Method	Boustrophedon	Circle	Figure-8	Perimeter	Contour	Chevron
CALM (ours)	8.5 $\pm$ 0.4/3.0 $\pm$ 0.1	8.3 $\pm$ 0.4/4.4 $\pm$ 0.2	8.3 $\pm$ 0.4/3.8 $\pm$ 0.2	6.9 $\pm$ 0.3/3.4 $\pm$ 0.2	6.5 $\pm$ 0.3/2.4 $\pm$ 0.1	7.6 $\pm$ 0.4/2.8 $\pm$ 0.1
Dec-MARL (arm)	8.9 $\pm$ 0.5/4.0 $\pm$ 0.2	13.4 $\pm$ 0.8/6.1 $\pm$ 0.4	10.9 $\pm$ 0.7/4.4 $\pm$ 0.3	12.0 $\pm$ 0.8/5.2 $\pm$ 0.3	11.3 $\pm$ 0.6/4.6 $\pm$ 0.3	9.2 $\pm$ 0.5/3.7 $\pm$ 0.2
Dec-MARL (cable)	19.2 $\pm$ 0.7/19.5 $\pm$ 0.4	18.1 $\pm$ 0.6/123.2 $\pm$ 1.2	19.6 $\pm$ 0.6/26.5 $\pm$ 0.7	18.7 $\pm$ 0.6/19.4 $\pm$ 0.3	18.8 $\pm$ 0.6/19.5 $\pm$ 0.3	18.7 $\pm$ 0.6/19.4 $\pm$ 0.4
Grasp allocation	57.0 $\pm$ 1.7/75.3 $\pm$ 3.3	51.5 $\pm$ 2.6/75.2 $\pm$ 4.2	52.5 $\pm$ 2.5/76.9 $\pm$ 5.0	52.9 $\pm$ 2.1/75.3 $\pm$ 4.8	52.2 $\pm$ 1.8/73.3 $\pm$ 3.9	51.7 $\pm$ 2.9/74.1 $\pm$ 5.4
RotorTM	76.8 $\pm$ 1.0/91.1 $\pm$ 2.9	72.6 $\pm$ 1.2/92.1 $\pm$ 5.3	73.9 $\pm$ 0.9/91.9 $\pm$ 4.0	72.9 $\pm$ 1.6/90.3 $\pm$ 4.2	72.6 $\pm$ 1.2/92.2 $\pm$ 3.4	72.8 $\pm$ 1.2/92.1 $\pm$ 4.6

TABLE II: Held-out position [cm] / orientation [deg] errors, given as mean

\pm

SD across 30 trials of the per-trial environment-time medians.

Fig. 4: Position (a,b) and orientation (c,d) sweeps under wind (left) and CoM deviation (right). Equivalent wind speed matches total mean applied force to system drag, with per-body force on the top axis. Curves show across-trial medians of the per-trial p50 and p90, with

n=30

trials per method.

Panel of Figure 7 — Fig. 5: Isaac Sim snapshots at (a) the training wind bound (1.15 N, 7.1 m/s equivalent), (b) twice that bound (2.30 N, 10.0 m/s equivalent), and (c) 0.20-m CoM deviation. Markers give goal and basket pose. In (d), SolidWorks verifies that the hardware design follows a CALM flight under wind, gusts, and the moving CoM within its joint travel and without part interference.

Fig. 6: Position and orientation with undegraded (filled), goal-degraded (tick), and goal-and-bus-degraded (open) information paths in 30 paired trials. Narrow panels show bus-only changes with 95% CIs. Filled ones exclude zero.

Unpowered group	Position	Orientation
all seven joints (passive arm)	2.93 [2.29, 3.49]	5.52 [4.28, 6.80]
wrist roll and pitch	2.64 [2.33, 3.44]	5.35 [4.29, 7.31]
shoulder and elbow (wrist active)	1.49 [1.28, 1.75]	1.97 [1.58, 2.46]
shoulder roll and pitch	1.33 [1.20, 1.55]	1.55 [1.36, 1.90]
wrist yaw	1.17 [1.05, 1.32]	1.15 [0.98, 1.31]
shoulder yaw	1.07 [0.91, 1.31]	1.19 [0.99, 1.41]

TABLE III: Median-error ratio of each joint-group ablation to the matched unmodified controller, with bootstrap 95% CIs.

Explore similar work

Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty
Jun 15, 2026Shivansh Pratap Singh, Samaksh Ujjwal, Ishita Chaudhary +3Robot Dynamics Identification RL for Robotics
International Institute of Information Technology Hyderabad, India. · Department of Computer Science, University of Manchester, UK.
AIR-VLA+: Decoupling Movement and Manipulation via Cascaded Dual-Action Decoders with Asymmetric MoE for Aerial Robots
Jun 11, 2026Jianli Sun, Bin Tian, Qiyao Zhang +6VLMs for Robotics Aerial Manipulation
The Institute of Automation, Chinese Academy of Sciences · School of Automation, Beijing Institute of Technology · College of Automotive and Energy Engineering, Tongji University +2
Decentralized End-to-End Multi-AAV Pursuit Using Predictive Spatio-Temporal Observation via Deep Reinforcement Learning
Mar 25, 2026Yude Li, Zhexuan Zhou, Huizhe Li +6Multi-Agent Systems Multi-Robot Coordination
The authors are with the School of Intelligence Science and Engineering, the Guangdong Key Laboratory of Intelligent Morphing Mechanisms and Adaptive Robotics, and the Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments, Harbin Institute of Technology, Shenzhen, Guangdong 518055, China

Jun 15, 2026J/K move · Enter open · S save

Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty

Shivansh Pratap Singh, Samaksh Ujjwal, Ishita Chaudhary, V R Vasudevan+2

International Institute of Information Technology Hyderabad, India. · Department of Computer Science, University of Manchester, UK.

Robot Dynamics Identification RL for Robotics Quadrotor Control Aerial Manipulation

Open paper AskPDF arXiv

Aerial manipulators enable physical interaction in hard-to-reach environments; however, the combined problem of direct whole-body aerial manipulation under rapid arm motion, payload changes, and related unknown dynamic uncertainty remains a largely unsolved problem. We present a hierarchical control framework that combines Reinforcement Learning (RL) with an inner-loop dynamics estimator to address this problem. The RL outer loop maps desired 6-degrees-of-freedom (DOF) end-effector targets to coordinated whole-body commands, enabling direct task-driven control without relying on a fully accurate coupled dynamic model in the policy layer. An inner loop then tracks these commands while compensating for transient inertial shifts and uncertainty during execution via a dynamics estimator scheme without requiring system model knowledge. We validate the proposed approach on a custom quadrotor equipped with a 3-DoF manipulator through hardware experiments under varying payload conditions. Compared with RL+PID and RL+INDI+PID baselines, the proposed method reduces end-effector tracking error and improves task success rate across the tested hardware conditions. These results show that combining learned whole-body coordination with estimator-based low-level compensation improves the precision and robustness of aerial manipulation under changing operating conditions.

Abstract

Explore similar work

Reinforcement Learning with Inner-loop Dynamics Estimator for Aerial Manipulation under Uncertainty

AIR-VLA+: Decoupling Movement and Manipulation via Cascaded Dual-Action Decoders with Asymmetric MoE for Aerial Robots

Decentralized End-to-End Multi-AAV Pursuit Using Predictive Spatio-Temporal Observation via Deep Reinforcement Learning