VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control

Fanxing Li; Fangyu Sun; Tianbao Zhang; Shuyu Wu; Dexin Zuo; yufei Yan; Wenxian Yu; Danping Zou

cs.ROMar 22, 2026

VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control

Authors: Fanxing Li, Fangyu Sun, Tianbao Zhang, Shuyu Wu, Dexin Zuo, yufei Yan, Wenxian Yu, Danping Zou

Abstract

First-order reinforcement learning with differentiable simulation is promising for quadrotor control, but practical progress remains fragmented across task-specific settings. To support more systematic development and evaluation, we present a unified differentiable framework for multi-task quadrotor control. The framework is wrapped, extensible, and equipped with deployment-oriented dynamics, providing a common interface across four representative tasks: hovering, tracking, landing, and racing. We also present the suite of first-order learning algorithms, where we identify two practical bottlenecks of standard first-order training: limited state coverage caused by horizon initialization and gradient bias caused by partially non-differentiable rewards. To address these issues, we propose Amended Backpropagation Through Time (ABPT), which combines differentiable rollout optimization, a value-based auxiliary objective, and visited-state initialization to improve training robustness. Experimental results show that ABPT yields the clearest gains in tasks with partially non-differentiable rewards, while remaining competitive in fully differentiable settings. We further provide proof-of-concept real-world deployments showing initial transferability of policies learned in the proposed framework beyond simulation.

Figures & tables

Fig. 1: Overview of the proposed unified differentiable framework for multi-task quadrotor control. The framework provides a wrapped and extensible architecture, supports differentiable simulation with deployment-oriented dynamics, and includes four representative tasks: hovering, tracking, landing, and racing.

Environments	Observation	Reward Function
Hovering	state & $\hat{p}$	$c-k_{1}\left\\|p-\hat{p}\right\\|-k_{2}\left\\|q-\hat{q}\right\\|-k_{3}\left\\|v\right\\|-k_{4}\left\\|\omega\right\\|$ (fully DIFF)
Tracking	state & next 10 $\hat{p}_{i=1\sim 10}$	$c-k_{1}\left\\|p-\hat{p}_{0}\right\\|-k_{2}\left\\|q-\hat{q}\right\\|-k_{3}\left\\|v\right\\|-k_{4}\left\\|\omega\right\\|$ (fully DIFF)
Landing	state & $\hat{p}$	$-k_{1}f^{+}\big(\left\\|p_{xy}-\hat{p}_{xy}\right\\|\big)+k_{2}f^{+}\big(\left\\|v_{z}-\hat{v}_{z}\right\\|\big)+k_{3}s$ (partially DIFF)
Racing	state & next 2 $\hat{p}_{i=1,2}$ of gates	$c-k_{1}\left\\|p-\hat{p}_{0}\right\\|-k_{2}\left\\|q-\hat{q}\right\\|-k_{3}\left\\|v\right\\|-k_{4}\left\\|\omega\right\\|+k_{5}s$ (partially DIFF)

TABLE I: Task Definitions, Observations, and Reward Structures in the Unified Multi-Task Framework

Fig. 2: Overview of ABPT. ABPT combines 0-step return and N-step return to compensate for gradient bias resulting from partially non-differentiable rewards. The red dashed lines indicate the direction of backpropagation. The replay buffer stores only visited states for episode initialization to improve sampling efficiency and is not used for off-policy training.

Fig. 3: Illustration of the limited state coverage in the current BPTT implementation. Assuming the agent’s minimal horizon length is five, it cannot end at points 4 and 5 after executing a horizon even with randomization. As a result, part of the observation space never serves as beginning of horizons, reducing sampling efficiency.

Fig. 4: Illustration of gradient bias caused by partially non-differentiable rewards. In the racing task introduced in Section V-A , the reward for passing a gate is a conditional constant and therefore does not directly contribute gradients through backpropagation.

Fig. 5: Training curves of PPO, SHAC, BPTT, and our ABPT in both time-step ( Top ) and wall-time ( Bottom ). Each curve is averaged over results from five random seeds, and the shaded area denotes the range of best and worst reward.

Fig. 6: Ablation study: the key components of ABPT are sequentially removed in turn to evaluate each one’s contribution.

Fig. 7: Training curves for the racing task using binary reward relaxations, which are trained with SHAC after replacing the binary reward with relaxed continuous reward.

Fig. 8: Training curves with different rewards: position-based rewards ( Left column ) and velocity-based rewards ( Right column ). The number of passed gates is visualized as the performance metric because of different rewards used for training.

Explore similar work

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation
Apr 12, 2026Fanxing Li, Shengyang Wang, Yuxiang Huang +5Quadrotor Control Vision-Based Robot Control
Shanghai Jiao Tong University · Kunshan Duke University
E2E-Fly: An Integrated Training-to-Deployment System for End-to-End Quadrotor Autonomy
Apr 14, 2026Fangyu Sun, Fanxing Li, Linzuo Zhang +5Quadrotor Control Sim-to-Real Transfer
School of Automation and Perception, Shanghai Jiao Tong University, China
Constrained Differentiable Policy Learning for Safe Quadrotor Navigation
Oct 12, 2026Yan Rui Tan, Praveen Krishnapur, Zeyu Feng +3Robot Policy Learning Constrained RL
National University of Singapore, Singapore. · Agency for Science, Technology and Research (A*STAR), Singapore.

Apr 12, 2026J/K move · Enter open · S save

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation

Fanxing Li, Shengyang Wang, Yuxiang Huang, Fangyu Sun+4

Shanghai Jiao Tong University · Kunshan Duke University

Quadrotor Control Vision-Based Robot Control Robot Collision Avoidance RL for Robotics

Open paper AskPDF arXiv

Obstacle avoidance is a fundamental vision-based task essential for enabling quadrotors to perform advanced applications. When planning the trajectory, existing approaches both on optimization and learning typically regard quadrotor as a point-mass model, giving path or velocity commands then tracking the commands by outer-loop controller. However, at high speeds, planned trajectories sometimes become dynamically infeasible in actual flight, which beyond the capacity of controller. In this paper, we propose a novel end-to-end policy that directly maps depth images to low-level bodyrate commands by reinforcement learning via differentiable simulation. The high-fidelity simulation in training after parameter identification significantly reduces all the gaps between training, simulation and real world. Analytical process by differentiable simulation provides accurate gradient to ensure efficiently training the low-level policy without expert guidance. The policy employs a lightweight and the most simple inference pipeline that runs without explicit mapping, backbone networks, primitives, recurrent structures, or backend controllers, nor curriculum or privileged guidance. By inferring low-level command directly to the hardware controller, the method enables full flight envelope control and avoids the dynamic-infeasible issue.Experimental results demonstrate that the proposed approach achieves the highest success rate and the lowest jerk among state-of-the-art baselines across multiple benchmarks. The policy also exhibits strong generalization, successfully deploying zero-shot in unseen, outdoor environments while reaching speeds of up to 7.5m/s as well as stably flying in the super-dense forest. This work is released at https://github.com/Fanxing-LI/avoidance.

Abstract

Explore similar work

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation

E2E-Fly: An Integrated Training-to-Deployment System for End-to-End Quadrotor Autonomy

Constrained Differentiable Policy Learning for Safe Quadrotor Navigation