Learning Robust Control Policies for Inverted Pose on Miniature Blimp Robots

Yuanlin Yang; Lin Hong; Fumin Zhang

cs.ROFeb 27, 2026

Learning Robust Control Policies for Inverted Pose on Miniature Blimp Robots

Authors: Yuanlin Yang, Lin Hong, Fumin Zhang

Organizations: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China

Abstract

The ability to achieve and maintain inverted poses is essential for unlocking the full agility of miniature blimp robots (MBRs). However, developing reliable inverted control strategies for MBRs remains challenging due to their complex and underactuated dynamics. To address this challenge, we propose a novel framework that enables robust control policy learning for inverted pose on MBRs. The proposed framework consists of three core stages. First, a high-fidelity three-dimensional (3D) simulation environment is constructed and calibrated using real-world MBR motion data. Second, a robust inverted control policy is trained in simulation using a modified Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm combined with a domain randomization strategy. Third, a mapping layer is designed to bridge the sim-to-real gap and facilitate real-world deployment of the learned policy. Comprehensive evaluations in the simulation environment demonstrate that the learned policy achieves a higher success rate compared to the energy-shaping controller. Furthermore, experimental results confirm that the learned policy with a mapping layer enables an MBR to achieve and maintain a fully inverted pose in real-world settings.

Figures & tables

Fig. 1: Overview of the proposed method for inverted pose on MBRs. The MBR with an upright pose can reach and maintain the inverted pose with the learned robust policy.

Fig. 2: Dynamic model analysis of the MBR: (a) Upright pose; (b) Inverted pose.

Fig. 3: Pipeline of developing and deploying robust policy for inverted control of MBRs.

Fig. 4: 3D simulation environment designed for robust policy learning for inverted control of MBRs.

Fig. 5: The MBR’s structure in the simulation environment.

$g_{\omega_{x}}$	$g_{\omega_{y}}$	$g_{\omega_{z}}$	$g_{a_{x}}$	$g_{a_{y}}$	$g_{a_{z}}$	$g_{\phi}$	$g_{\theta}$	$g_{\psi}$	$\zeta$	$g_{n}$
0.01	0.01	0.01	0.001	0.001	0.001	5.0	5.0	0.5	0.1	10

TABLE I: Parameters in the reward function

$N$	$t_{e}$	$N_{e}$	$\sigma$	$\xi$	$n_{\sigma}$	$d_{p}$	$\gamma$	$\varpi$	$c_{a}$	$c_{b}$	$\lambda$	$\psi_{0}$
10	30s	500	0.15	0.95	100	2	0.98	0.01	0.1	0.1	[0.6, 1]	0.5

TABLE II: Parameters in policy training

Fig. 6: Roll angle variations with changes in $m_{w}$ . Solid lines represent the learned policy, while the dotted line represents the baseline controller.

Method	$m_{w}=5g$	$m_{w}=10g$	$m_{w}=15g$	$m_{w}=20g$	$m_{w}=25g$
Baseline	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Our policy	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE III: Comparison of the baseline controller and the learned policy with varied

m_{w}

Method	$\lambda=0.6$	$\lambda=0.7$	$\lambda=0.8$	$\lambda=0.9$	$\lambda=1.0$
Baseline	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Our policy	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE IV: Comparison of the baseline controller and the learned policy with varied

\lambda

Fig. 7: Roll angle variations with changes in $\lambda$ .

Method	$g_{m}=0.5$	$g_{m}=1.0$	$g_{m}=1.5$	$g_{m}=2.0$	$g_{m}=2.5$
Baseline	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Our policy	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE V: Comparison of the baseline controller and the learned policy with varied

g_{m}

Fig. 8: Roll angle variations with changes in $g_{m}$ .

Test Case	C1	C2	C3	C4	C5
$m_{w}$ ( $\mathrm{g}$ )	15	15	20	25	25
$\lambda$	0.8	1.0	0.9	0.7	0.8
$g_{m}$	1.7	1.0	1.6	1.5	1.4
Baseline	$\times$	$\times$	$\times$	$\times$	$\times$
Our Policy	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$

TABLE VI: Comparison of the baseline controller and the learned policy with varied

m_{w}

\lambda

, and

g_{m}

Fig. 9: Roll angle variations with changes in $m_{w}$ , $\lambda$ and $g_{m}$ .

Fig. 10: Comparison of the average reward across three cases, computed using a moving average with a window size of 19.

Fig. 11: Roll angle variations of the MBR with different $m_{\phi}$ .

Fig. 12: (a) Experimental setup; (b) Visualization of the action sequence of an MBR for achieving an inverted pose.

Weights	$MBR_{1}$	$MBR_{2}$	$MBR_{3}$	$MBR_{4}$	$MBR_{5}$
$m_{w1}(g)$	$25$	$26.07$	$25$	$27.59$	$25$
$m_{w2}(g)$	$0$	$0$	$1.07$	$0$	$2.59$

TABLE VII: Configuration of the extra weights in the physical experiment

Fig. 13: Roll angle variation of the MBR with different $m_{w}$ .

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 584 papers

May 23, 2026

AcroRL: Learning Aggressive Quadrotor Inversion using Bidirectional Thrust

Bidirectional thrust grants quadrotors a second equilibrium condition and increased control authority, expanding the envelope of possible aggressive maneuvers and enabling inverted flight, perching, and sensing. Prior geometric control approaches extend differential flatness through Hopf fibration-based attitude representations to support bidirectional thrust, but struggle with actuator saturation and motor reversal delay during inversions, requiring heuristic thrust posture scheduling and waypoint tuning. We propose a learning-based framework that modulates a constant reference trajectory to perform compact, position-constrained quadrotor inversions while remaining compatible with traditional trajectory generation and tracking across flight regimes. Separate policies are trained via reinforcement learning for nominal-to-inverted and inverted-to-nominal transitions. In JAX-based simulation, the proposed method achieves the lowest position deviation and settling time across all evaluated baselines, reducing position root mean square error (RMSE) by 32% and settling time by 57% relative to the strongest optimization-based baseline. Hardware experiments demonstrate successful inversion across multiple yaw configurations with position RMSE below 0.35m, and compatibility with downstream trajectory generation and control through circular flight in both regimes. Additionally, we provide an open-source implementation of the proposed framework.

Gabriel Rodriguez, Henri Sayag, Abhishek Rathod +4

Carnegie Mellon University · Shield AI

RL Control Aerial Robotics Quadrotor Control

Ask PDF

Mar 11, 2026

ASTER: Attitude-aware Suspended-payload Quadrotor Traversal via Efficient Reinforcement Learning

Agile maneuvering of the quadrotor cable-suspended system is significantly hindered by its non-smooth hybrid dynamics. While model-free Reinforcement Learning (RL) circumvents explicit differentiation of complex models, achieving attitude-constrained or inverted flight remains an open challenge due to the extreme reward sparsity under strict orientation requirements. This paper presents ASTER, a robust RL framework that achieves, to our knowledge, the first successful autonomous inverted flight for the cable-suspended system. We propose hybrid-dynamics-informed state seeding (HDSS), an initialization strategy that back-propagates target configurations through physics-consistent kinematic inversions across both taut and slack cable phases. HDSS enables the policy to discover aggressive maneuvers that are unreachable via standard exploration. Extensive simulations and real-world experiments demonstrate remarkable agility, precise attitude alignment, and robust zero-shot sim-to-real transfer across complex trajectories.

Dongcheng Cao, Jin Zhou, Shuo Li

College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China.

Quadrotor Control Robotic RL Robust RL

Ask PDF

May 2, 2026

Bi-Level Reinforcement Learning Control for an Underactuated Blimp via Center-of-Mass Reconfiguration

This paper investigates goal-directed tracking control of underactuated blimps with center-of-mass (CoM) reconfiguration. Unlike conventional overactuated blimp designs that rely on redundant actuation for simplified control, this paper focuses on a compact architecture consisting of two thrusters and a movable internal slider, aiming to improve energy efficiency and payload capacity. This hardware-efficient configuration introduces significant underactuation and strong nonlinear coupling between CoM dynamics and vehicle motion. To address these challenges, this paper proposes a bi-level reinforcement learning framework that explicitly decouples task-level CoM planning from continuous thrust control. The outer policy determines a target-dependent CoM configuration prior to flight, while the inner policy generates thrust commands to track straight-line references. To ensure stable learning, this paper introduces a two-stage learning strategy, supported by a convergence analysis of the resulting bi-level process. Extensive simulations and real-world experiments on a 27-goal evaluation set demonstrate that the proposed method consistently outperforms fixed-CoM baselines and PID-based controllers, achieving higher tracking accuracy, enhanced robustness, and reliable sim-to-real transfer.

Xiaorui Wang, Hongwu Wang, Yue Fan +2

RL Control Aerial Robotics Hierarchical RL Continuous Control

Ask PDF

Abstract

Robotic Control

Explore similar work

AcroRL: Learning Aggressive Quadrotor Inversion using Bidirectional Thrust

ASTER: Attitude-aware Suspended-payload Quadrotor Traversal via Efficient Reinforcement Learning

Bi-Level Reinforcement Learning Control for an Underactuated Blimp via Center-of-Mass Reconfiguration