Differentiable simulation enables efficient policy optimization through analytic gradients, yet safety is commonly incorporated as a weighted penalty, requiring manual tuning of the trade-off between task performance and constraint satisfaction. We instead formulate obstacle avoidance as an explicit constraint and train quadrotor policies using primal--dual gradient descent--ascent through a differentiable simulator. We introduce two key components to enable effective constrained training. First, we propose a bounded clearance-based constraint function that provides informative gradients around the safety boundary while preventing large violations from destabilizing primal--dual optimization. Second, we introduce log-mean-exp (LME) aggregation to smoothly emphasize critical per-step safety violations across a trajectory and training batch within a single constraint governed by one dual variable. Together, these components enable the desired safety level to be specified directly while balancing constraint satisfaction and task completion. We theoretically prove that the proposed optimization with differentiable simulation converges to feasible policy under mild assumptions. Experiments demonstrate improved safety--performance trade-offs over fixed-penalty and existing policy-learning baselines, as well as generalization to denser out-of-distribution environments. Finally, the learned policy transfers zero-shot to Gazebo/PX4 and a physical quadrotor without additional fine-tuning.
Figures & tables
Fig. 1 : Zero-shot deployment of our primal–dual policy on a physical quadrotor. Due to practical limitations in reproducing the exact obstacle geometry, the physical obstacles serve as proxies for the virtual cylindrical obstacles perceived by the policy. The inset shows the corresponding trajectory in the policy’s perceived environment.
Fig. 2 : A. Comparison of our proposed bounded constraint cost with a conventional unbounded signed-distance-based cost proposed in [ 4 ] . B. Our permutation-invariant policy architecture for mapping vehicle states and variable obstacle configurations to control actions.
Task and training
Symbol
Description
Value
Symbol
Description
Value
B
Parallel envs.
256
T
Horizon
300 steps
Δt
Time step
0.03 s
Nobs
Obstacles
4
Nepoch
Epochs
20 k
vmax
Horiz. vel.
1.0 m/s
vmaxz
Vert. vel.
0.3 m/s
Constraint
TABLE I : Training parameters.
Fig. 3 : Training convergence of B2N (BPTT) and B2N (PPO), with B2N (BPTT) converging to a better-performing policy.
Fig. 4 : A. Representative trajectories of the baseline policies and our proposed method. B. Representative OOD trajectories comparing our method with B2N(BPTT). B2N fails near obstacle-adjacent goals (green/red) and in the narrow passage (blue), while ours succeeds in all cases. C. Consistent trajectories across differentiable simulation, Gazebo, and real-world deployment demonstrate zero-shot sim-to-real transfer. D. Real-world quadrotor trajectory over a 12-leg continuous circuit, with waypoints distributed on a 3 m-radius circle centered on the obstacle configuration.
Method
Task ↓
LME ↓
Coll. (%) ↓
Clear. p5 (m) ↑
Success ↑
B2N (PPO)
3.48±2.73
0.027±0.054
0.18
+0.504
90.5%
B2N (BPTT)
1.70±0.77
0.019±0.006
0.00
+0.665
100.0%
CGPO ( α/5 )
6.40±5.56
0.129±0.214
4.12
+0.042
10.3%
CGPO ( α/40 )
0.52±0.11
1.098±0.217
92.75
−0.664
97.7%
λ=4
0.34±0.00
1.122±0.171
99.23
−0.662
100.0%
λ=16
6.76±4.99
0.450±0.202
8.66
−0.044
67.9%
TABLE II : Held-out comparison over 10,240 obstructed scenarios (mean ± std.). Shaded rows denote methods achieving 100% success with positive 5th-percentile clearance. Within this subset, bold indicates the lower task cost.
Fig. 5 : A. λ evolution with our proposed bounded constraint cost. B. λ evolution with the unbounded constraint cost used by B2N [ 4 ] with β2 parameter sweep. C. Analysis of constraint aggregation. LME closely matches the exact maximum in separating colliding from collision-free trajectories and remains consistent across rollout horizons, unlike cumulative aggregation.
Obstacle avoidance is a fundamental vision-based task essential for enabling quadrotors to perform advanced applications. When planning the trajectory, existing approaches both on optimization and learning typically regard quadrotor as a point-mass model, giving path or velocity commands then tracking the commands by outer-loop controller. However, at high speeds, planned trajectories sometimes become dynamically infeasible in actual flight, which beyond the capacity of controller. In this paper, we propose a novel end-to-end policy that directly maps depth images to low-level bodyrate commands by reinforcement learning via differentiable simulation. The high-fidelity simulation in training after parameter identification significantly reduces all the gaps between training, simulation and real world. Analytical process by differentiable simulation provides accurate gradient to ensure efficiently training the low-level policy without expert guidance. The policy employs a lightweight and the most simple inference pipeline that runs without explicit mapping, backbone networks, primitives, recurrent structures, or backend controllers, nor curriculum or privileged guidance. By inferring low-level command directly to the hardware controller, the method enables full flight envelope control and avoids the dynamic-infeasible issue.Experimental results demonstrate that the proposed approach achieves the highest success rate and the lowest jerk among state-of-the-art baselines across multiple benchmarks. The policy also exhibits strong generalization, successfully deploying zero-shot in unseen, outdoor environments while reaching speeds of up to 7.5m/s as well as stably flying in the super-dense forest. This work is released at https://github.com/Fanxing-LI/avoidance.
Fanxing Li, Shengyang Wang, Yuxiang Huang +5
Shanghai Jiao Tong University · Kunshan Duke University
Deep Reinforcement Learning has demonstrated remarkable capability in quadrotor control, yet learned policies offer no guarantee of respecting safety constraints during training or deployment. We present CALOS (Control-Affine Lyapunov On-manifold Safety), a runtime safety layer that enforces attitude constraints on a quadrotor without modifying the underlying learning algorithm. CALOS formulates four tilt-angle inequalities and a Lyapunov descent condition as a single quadratic program whose solution is the minimum-norm correction to the nominal torque output of the policy. The quadratic program is solved exactly via active-set enumeration over the three-dimensional torque space, with a computational cost low enough to enforce constraints in real time across thousands of parallel simulation environments, as required by modern massively parallel Deep Reinforcement Learning training. Evaluated on trajectory-tracking tasks in NVIDIA Isaac Lab, CALOS reduces lateral tracking error by 55-60% relative to an unconstrained Proximal Policy Optimization baseline while achieving zero attitude-constraint violations on the training trajectory. By restricting exploration to safe regions of the state space, the safety layer also accelerates training convergence and improves data efficiency without producing suboptimal policies.
First-order reinforcement learning with differentiable simulation is promising for quadrotor control, but practical progress remains fragmented across task-specific settings. To support more systematic development and evaluation, we present a unified differentiable framework for multi-task quadrotor control. The framework is wrapped, extensible, and equipped with deployment-oriented dynamics, providing a common interface across four representative tasks: hovering, tracking, landing, and racing. We also present the suite of first-order learning algorithms, where we identify two practical bottlenecks of standard first-order training: limited state coverage caused by horizon initialization and gradient bias caused by partially non-differentiable rewards. To address these issues, we propose Amended Backpropagation Through Time (ABPT), which combines differentiable rollout optimization, a value-based auxiliary objective, and visited-state initialization to improve training robustness. Experimental results show that ABPT yields the clearest gains in tasks with partially non-differentiable rewards, while remaining competitive in fully differentiable settings. We further provide proof-of-concept real-world deployments showing initial transferability of policies learned in the proposed framework beyond simulation.