Learning End-to-End Control for Omnidirectional Aerial Motion on Overactuated Tilt-rotor Quadrotors

Wentao Zhang; Zhaoqi Ma; Jinjie Li; Huayi Wang; Haokun Liu; Junichiro Sugihara; Chen Chen; Yicheng Chen

cs.ROFeb 25, 2026

Learning End-to-End Control for Omnidirectional Aerial Motion on Overactuated Tilt-rotor Quadrotors

Authors: Wentao Zhang, Zhaoqi Ma, Jinjie Li, Huayi Wang, Haokun Liu, Junichiro Sugihara, Chen Chen, Yicheng Chen, +3 more

Organizations: DRAGON Lab, Department of Mechanical Engineering, The University of Tokyo, Tokyo, 113-8656, Japan · School of Computing and Data Science, The University of Hong Kong, 999077, HKSAR · Autonomous Systems Lab(ASL), ETH Zurich, 8092 Zurich, Switzerland

Abstract

While reinforcement learning (RL) has been successfully applied to conventional quadrotors for agile and robust flight, whether actuator-level RL can be reliably deployed on tilt-rotor aerial robots remains an open question, as the hybrid actuation coupling brushless rotors with rotational joints introduces a substantially harder sim-to-real gap. In this work, we propose an end-to-end RL framework for omnidirectional motion control on overactuated tilt-rotor quadrotors, directly mapping target poses to joint and rotor commands. The learning framework combines actuator-level simulation with an asymmetric actor-critic architecture for 6D pose-reaching. For reliable sim-to-real transfer on the hybrid actuation, we integrate system identification with minimal yet physically grounded domain randomization. The trained policy is deployed zero-shot on real hardware and evaluated across waypoint hovering, external disturbances, payload variation and trajectory tracking, together with simulated traversal of allocation-singular configurations. The policy is compared with a state-of-the-art NMPC baseline: NMPC attains lower steady-state position error, whereas the RL policy offers a more uniform orientation error across evaluations, transitions between poses faster, and requires less onboard computation.

Figures & tables

Fig. 2: Model-based control framework for tiltable aerial robots.

Fig. 3: Data-driven control frameworks for quadrotors. LV: rely on a control stack to process velocity commands. CTBR: require a feedback regulator to process rotor thrusts. SRT: directly command rotor thrusts.

Fig. 4: Overall framework for simulation, RL training, and deployment. Robot : Rotor modules (blue, target thrust ${\bm{f}}_{\mathrm{r}}^{*}\rightarrow$ real thrust $\bm{f}_{\mathrm{r}}$ and torque $\bm{\tau}_{\mathrm{r}}$ ), joint modules (red, joint position commands $\bm{q}_{\mathrm{j}}^{*}\rightarrow$ position feedback $\bm{q}_{\mathrm{j}}$ ), and system latency (green arrows) are modeled to construct physically consistent simulation environments. Training : The policy is trained under an asymmetric actor-critic framework and dynamics randomization (green) is integrated according to models and deployment scheme. Deployment : Trained policy is deployed on real robot platform with a consistent system architecture. The robot’s state is estimated by fusing data from a motion capture system (MoCap) and an onboard IMU using an Extended Kalman filter (EKF).

Pose Reaching
Reward	Formulation	Weight
Towards Position	$\tanh\!\left(\frac{\parallel\bm{p}_{e}\parallel}{0.6}\right)$	$-1.0$
Towards Pose	$\left(1-\tanh\!\left(\frac{\parallel\bm{\theta}_{e}\parallel}{2}\right)\right)\cdot\left(1-\tanh\!\left(\frac{\parallel\bm{p}_{e}\parallel}{2}\right)\right)$	$2.0$
Reach Position	$\left[\hskip-5.0pt\begin{array}[]{c}\mathbb{I}\left[\parallel\bm{p}_{e}\parallel<p_{\mathrm{th}}\right]\\ \land\mathbb{I}\left[\parallel\bm{v}_{\mathrm{b}}\parallel<v_{\mathrm{th}}\right]\\ \end{array}\hskip-5.0pt\right]\cdot\left[e^{-\frac{\parallel\bm{p}_{e}\parallel}{p_{\mathrm{th}}}}\hskip-2.0pt+e^{-\frac{\parallel\bm{v}_{\mathrm{b}}\parallel}{v_{\mathrm{th}}}}\right]$	$1.0$
Reach Pose	$\begin{array}[]{c}\left[\hskip-5.0pt\begin{array}[]{c}\mathbb{I}\left[\parallel\bm{p}_{e}\parallel<0.02\right]\land\mathbb{I}\left[\parallel\bm{\theta}_{e}\parallel<0.05\right]\\ \land\mathbb{I}\left[\parallel\bm{v}_{\mathrm{b}}\parallel<0.02\right]\land\mathbb{I}\left[\parallel\bm{\omega}_{\mathrm{b}}\parallel<0.02\right]\\ \end{array}\hskip-5.0pt\right]\\[10.0pt] \cdot\left[\hskip-5.0pt\begin{array}[]{c}e^{-\frac{\parallel\bm{p}_{e}\parallel}{0.02}}+e^{-\frac{\parallel\bm{v}_{\mathrm{b}}\parallel}{0.02}}\\ +e^{-2\frac{\parallel\bm{\theta}_{e}\parallel}{0.05}}+e^{-2\frac{\parallel\bm{\omega}_{\mathrm{b}}\parallel}{0.02}}\\ \end{array}\hskip-5.0pt\right]\\ \end{array}$	$0.75$
Regularization

TABLE I: Reward Description

Dynamics Parameters
Variables	Operation	Latency $(\mathrm{d}t=$ 10\text{~}\mathrm{ms} $)$
mass [ $\mathrm{kg}$ ]	scale [0.9,1.1]	-
inertia [ $\mathrm{kg}\text{$ {}\cdot{} $}{\mathrm{m}}^{2}$ ]	scale [0.9,1.1]	-
COM [ $\mathrm{m}$ ]	bias $\pm 0.01$	-
State Observations
$\bm{p}_{\mathrm{b}}$ [ $\mathrm{m}$ ]	bias $\pm 0.01$ , noise $\pm 0.01$	0

TABLE II: Domain Randomization, Noise and Time Delay

	$\mathcal{U}\!\parallel\bm{p}_{e}\!\parallel$ $\left(\pm std,\ \begin{subarray}{c}\scriptscriptstyle min\\ \scriptscriptstyle max\end{subarray}\right)$ [ $\mathrm{m}$ ] $\downarrow$	$\mathcal{U}\!\parallel\bm{\theta}_{e}\!\parallel$ $\left(\pm std,\ \begin{subarray}{c}\scriptscriptstyle min\\ \scriptscriptstyle max\end{subarray}\right)$ [ $\deg$ ] $\downarrow$	$\mathcal{L}\!\parallel\bm{p}_{e}\!\parallel$ $\left(\pm std,\ \begin{subarray}{c}\scriptscriptstyle min\\ \scriptscriptstyle max\end{subarray}\right)$ [ $\mathrm{m}$ ] $\downarrow$	$\mathcal{L}\!\parallel\bm{\theta}_{e}\!\parallel$ $\left(\pm std,\ \begin{subarray}{c}\scriptscriptstyle min\\ \scriptscriptstyle max\end{subarray}\right)$ [ $\deg$ ] $\downarrow$	$\left\|\bm{q}_{\mathrm{j}}^{*}-\bm{q}_{\mathrm{j}}\right\|$ ( $\pm std$ ) [ $\mathrm{rad}$ ] $\downarrow$	$\bm{f}_{\mathrm{r}}^{*}$ ( $\pm std$ ) [ $\mathrm{N}$ ] $\downarrow$
RL	0.077 $\left(\pm 0.029,\ \begin{subarray}{c}\scriptscriptstyle 0.019\\ \scriptscriptstyle 0.304\end{subarray}\right)$	3.88 $\left(\pm 5.70,\ \begin{subarray}{c}\scriptscriptstyle 0.25\\ \scriptscriptstyle 116.73\end{subarray}\right)$	0.127 $\left(\pm 0.071,\ \begin{subarray}{c}\scriptscriptstyle 0.030\\ \scriptscriptstyle 0.895\end{subarray}\right)$	7.85 $\left(\pm 10.60,\ \begin{subarray}{c}\scriptscriptstyle 0.41\\ \scriptscriptstyle 145.77\end{subarray}\right)$	0.029 ( $\pm 0.083$ )	10.00 ( $\pm 3.27$ )
NMPC	0.024 $\left(\pm 0.011,\ \begin{subarray}{c}\scriptscriptstyle 0.002\\ \scriptscriptstyle 0.068\end{subarray}\right)$	25.12 $\left(\pm 35.06,\ \begin{subarray}{c}\scriptscriptstyle 0.16\\ \scriptscriptstyle 170.78\end{subarray}\right)$	0.422 $\left(\pm 0.147,\ \begin{subarray}{c}\scriptscriptstyle 0.096\\ \scriptscriptstyle 0.955\end{subarray}\right)$	40.93 $\left(\pm 31.03,\ \begin{subarray}{c}\scriptscriptstyle 6.49\\ \scriptscriptstyle 171.01\end{subarray}\right)$	0.004 ( $\pm 0.015$ )	9.31 ( $\pm 2.97$ )

TABLE III: Continuous Pose Reaching in Simulation

Fig. 5: Cumulative distributions of steady-state position and orientation errors across the sampled target set.

	$\scriptstyle\mathcal{U}\!\parallel\bm{p}_{e}\!\parallel\,\downarrow$	$\scriptstyle\mathcal{U}\!\parallel\bm{\theta}_{e}\!\parallel\,\downarrow$	$\scriptstyle\mathcal{L}\!\parallel\bm{p}_{e}\!\parallel\,\downarrow$	$\scriptstyle\mathcal{L}\!\parallel\bm{\theta}_{e}\!\parallel\,\downarrow$	$\scriptstyle\mathcal{U}\!\parallel\bm{v}_{\mathrm{b}}\!\parallel\,\downarrow$	$\scriptstyle\mathcal{U}\!\parallel\bm{\omega}_{\mathrm{b}}\!\parallel\,\downarrow$	$\scriptstyle\left\|\bm{q}_{\mathrm{j}}^{*}-\bm{q}_{\mathrm{j}}\right\|\,\downarrow$	$\scriptstyle\bm{f}_{\mathrm{r}}^{*}\,\downarrow$	$\scriptstyle\left\|\dot{\bm{q}}_{\mathrm{j},t}^{*}\right\|\,\downarrow$	$\scriptstyle\left\|\dot{\bm{f}}_{\mathrm{r},t}^{*}\right\|\,\downarrow$
	[ $\mathrm{m}$ ]	[ $\deg$ ]	[ $\mathrm{m}$ ]	[ $\deg$ ]	[ $\mathrm{m}\text{/}\mathrm{s}$ ]	[ $\mathrm{rad}\text{/}\mathrm{s}$ ]	[ $\mathrm{rad}$ ]	[ $\mathrm{N}$ ]	[ $\mathrm{rad}$ /step]	[ $\mathrm{N}$ /step]
Base	0.077	3.88	0.127	7.85	0.009	0.007	0.029	10.00	0.69	7.35
(a)	0.081	3.79	0.135	7.26	0.010	0.010	0.032	10.22	0.79	7.77
(b)	0.081	3.83	0.134	7.96	0.011	0.009	0.029	9.93	0.69	7.62
(c)	0.079	3.74	0.140	8.27	0.012	0.012	0.031	10.29	0.69	7.32

TABLE IV: Rewards Ablation Study Results

$\bm{p}_{\mathrm{w}}^{*}$ [ $\mathrm{m}$ ]	$\bm{\theta}_{\mathrm{b}}^{*}$ [ $\deg$ ]	RL		NMPC
$\bm{p}_{\mathrm{w}}^{*}$ [ $\mathrm{m}$ ]	$\bm{\theta}_{\mathrm{b}}^{*}$ [ $\deg$ ]	$\mathcal{U}\!\parallel\bm{p}_{e}\!\parallel$ [ $\mathrm{m}$ ] $\downarrow$	$\mathcal{U}\!\parallel\bm{\theta}_{e}\!\parallel$ [ $\deg$ ] $\downarrow$	$\mathcal{U}\!\parallel\bm{p}_{e}\!\parallel$ [ $\mathrm{m}$ ] $\downarrow$	$\mathcal{U}\!\parallel\bm{\theta}_{e}\!\parallel$ [ $\deg$ ] $\downarrow$
[0.0, 0.0, 0.8]	[0, 0, 0]	0.114	2.589	0.026	2.644
[1.0, 0.0, 1.0]	[45, 0, 0]	0.060	4.350	0.074	4.372
[0.0, 1.0, 1.0]	[0, -25, 0]	0.046	4.214	0.054	8.225
[-1.0, 0.0, 1.0]	[-25, 0, 0]	0.074	5.561	0.028	4.742
[0.0, 0.0, 0.8]	[0, 0, 90]	0.091	4.590	0.028	1.768

TABLE V: Continuous Pose Reaching in Real World

Panel of Figure 10 — Fig. 6: State trajectories from real-world waypoint hovering experiments. Solid lines show the mean values across trials, and the shaded regions represent the corresponding minimum and maximum ranges.

Fig. 7: Real-world disturbance rejection evaluation. Left: response under wind. Right: response to impulsive stick pushes.

Fig. 8: Steady-state position and orientation errors under varying payloads. Lines denote the mean over all trials, shaded regions the min–max range, and markers the individual samples. Each evaluation trial contains four samples of nominal pose and three samples of tilted pose.

Fig. 9: Velocity profile of the

90\text{~}\deg

pitch maneuver.

Fig. 10: Real-world trajectory tracking of a 3D lemniscate with synchronized orientation modulation, repeated five times.

Fig. 11: State evolution under continuous rotation through

\bm{A}_{\alpha}

singularities.

Term	Value	Term	Value
Weight Parameters
mass [ $\mathrm{kg}$ ]	3.0386	inertia $I_{x},I_{y},I_{z}$ [ $\mathrm{g}\text{$ {}\cdot{} $}{\mathrm{m}}^{2}$ ]	63, 62, 95
Joint Parameters
$K_{p}$	0.3449	$K_{d}$	0.0094
position limit [ $\mathrm{rad}$ ]	3.96	velocity limit [ $\mathrm{rad}\text{/}\mathrm{s}$ ]	10.0
effort limit [ $\mathrm{N}\text{$ {}\cdot{} $}\mathrm{m}$ ]	3.0

TABLE VI: Physical Parameters

Term	Value	Term	Value
General
num of robots	4096	num of epochs	5
gradient clipping	1.0	adam epsilon	$1.0e-8$
num of steps per iteration	48	num of steps per episode	1,200
total training iterations		20,000
PPO

TABLE VII: Hyperparameters

Explore similar work

Agile Fall Recovery for Quadrotors with Bidirectional Thrust via Reinforcement Learning
Jun 15, 2026Anke Zhao, Yuhang Zhong, Kenghou Hoi +5Robot Failure Recovery Robotic RL
Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · Differential Robotics, Hangzhou 311121, China
Physics-Aware End-to-End Deep Reinforcement Learning for Quadcopter Control with Actuator Dynamics
Jul 28, 2026Ya-Chia Shen, Woei-Leong ChanRL Control Quadrotor Control
Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Aeronautics and Astronautics, National Cheng Kung University
Hybrid Neural Network and Conventional Controller Approach for Robust Control of Highly Unstable Systems: Application to Tilt-Rotor Control
Jun 7, 2026Ali Kafili Gavgani, Amin Talaeizadeh, Aria Alasty +1Robust Control Feedback Control
Advanced Research Lab for Control and Agricultural Robotics (Sharif AgRoLab) Department of Mechanical Engineering, Sharif University of Technology, Tehran, Iran

Jun 15, 2026J/K move · Enter open · S save

Agile Fall Recovery for Quadrotors with Bidirectional Thrust via Reinforcement Learning

Anke Zhao, Yuhang Zhong, Kenghou Hoi, Junyu Mou+4

Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · Differential Robotics, Hangzhou 311121, China

Robot Failure Recovery Robotic RL Quadrotor Control

Open paper AskPDF arXiv

Autonomous fall recovery is a critical capability for quadrotors operating in real-world environments, where collisions or failures may leave the vehicle resting on the ground in an arbitrary attitude. This problem is challenging because recovery must be achieved under limited onboard sensing, in constrained free space, with ground contact, and in the presence of unknown disturbances. In this letter, we present an RL-based framework for autonomous fall recovery of a quadrotor from arbitrary ground attitudes to stable hover using only lightweight onboard sensors. To address severe partial observability and intermittent sensor invalidity, we train a recurrent policy within an asymmetric actor--critic architecture, leveraging an Incremental Nonlinear Dynamic Inversion (INDI) controller to track the policy output. Combined with high-fidelity simulations of motor response and optical flow, the overall training framework significantly reduces the sim-to-real gap. Simulation ablation studies validate the importance of the main design choices, while real-world experiments demonstrate zero-shot transfer and robust recovery under different initial attitudes, wind disturbances, and additional payloads. These results demonstrate that agile quadrotor fall recovery can be achieved without explicit state estimation using only limited and unreliable onboard sensing.

Abstract

Explore similar work

Agile Fall Recovery for Quadrotors with Bidirectional Thrust via Reinforcement Learning

Physics-Aware End-to-End Deep Reinforcement Learning for Quadcopter Control with Actuator Dynamics

Hybrid Neural Network and Conventional Controller Approach for Robust Control of Highly Unstable Systems: Application to Tilt-Rotor Control