Constrained Differentiable Policy Learning for Safe Quadrotor Navigation

Yan Rui Tan; Praveen Krishnapur; Zeyu Feng; Han Bao; Yew-Soon Ong; Fan Shi

cs.ROOct 12, 2026

Constrained Differentiable Policy Learning for Safe Quadrotor Navigation

Authors: Yan Rui Tan, Praveen Krishnapur, Zeyu Feng, Han Bao, Yew-Soon Ong, Fan Shi

Organizations: National University of Singapore, Singapore. · Agency for Science, Technology and Research (A*STAR), Singapore.

Abstract

Differentiable simulation enables efficient policy optimization through analytic gradients, yet safety is commonly incorporated as a weighted penalty, requiring manual tuning of the trade-off between task performance and constraint satisfaction. We instead formulate obstacle avoidance as an explicit constraint and train quadrotor policies using primal--dual gradient descent--ascent through a differentiable simulator. We introduce two key components to enable effective constrained training. First, we propose a bounded clearance-based constraint function that provides informative gradients around the safety boundary while preventing large violations from destabilizing primal--dual optimization. Second, we introduce log-mean-exp (LME) aggregation to smoothly emphasize critical per-step safety violations across a trajectory and training batch within a single constraint governed by one dual variable. Together, these components enable the desired safety level to be specified directly while balancing constraint satisfaction and task completion. We theoretically prove that the proposed optimization with differentiable simulation converges to feasible policy under mild assumptions. Experiments demonstrate improved safety--performance trade-offs over fixed-penalty and existing policy-learning baselines, as well as generalization to denser out-of-distribution environments. Finally, the learned policy transfers zero-shot to Gazebo/PX4 and a physical quadrotor without additional fine-tuning.

Figures & tables

Fig. 1 : Zero-shot deployment of our primal–dual policy on a physical quadrotor. Due to practical limitations in reproducing the exact obstacle geometry, the physical obstacles serve as proxies for the virtual cylindrical obstacles perceived by the policy. The inset shows the corresponding trajectory in the policy’s perceived environment.

Fig. 2 : A. Comparison of our proposed bounded constraint cost with a conventional unbounded signed-distance-based cost proposed in [ 4 ] . B. Our permutation-invariant policy architecture for mapping vehicle states and variable obstacle configurations to control actions.

Task and training
Symbol	Description	Value	Symbol	Description	Value
$B$	Parallel envs.	$256$	$T$	Horizon	$300$ steps
$\Delta t$	Time step	$0.03$ s	$N_{\mathrm{obs}}$	Obstacles	$4$
$N_{\mathrm{epoch}}$	Epochs	$20$ k	$v_{\max}$	Horiz. vel.	$1.0$ m/s
$v_{\max}^{z}$	Vert. vel.	$0.3$ m/s
Constraint

TABLE I : Training parameters.

Fig. 3 : Training convergence of B2N (BPTT) and B2N (PPO), with B2N (BPTT) converging to a better-performing policy.

Panel of Figure 5 — Fig. 4 : A. Representative trajectories of the baseline policies and our proposed method. B. Representative OOD trajectories comparing our method with B2N(BPTT). B2N fails near obstacle-adjacent goals (green/red) and in the narrow passage (blue), while ours succeeds in all cases. C. Consistent trajectories across differentiable simulation, Gazebo, and real-world deployment demonstrate zero-shot sim-to-real transfer. D. Real-world quadrotor trajectory over a 12-leg continuous circuit, with waypoints distributed on a $3$ m-radius circle centered on the obstacle configuration.

Method	Task $\downarrow$	LME $\downarrow$	Coll. (%) $\downarrow$	Clear. p5 (m) $\uparrow$	Success $\uparrow$
B2N (PPO)	$3.48{\pm}2.73$	$0.027{\pm}0.054$	$0.18$	$+0.504$	$90.5\%$
B2N (BPTT)	$1.70{\pm}0.77$	$0.019{\pm}0.006$	$0.00$	$+0.665$	$100.0\%$
CGPO ( $\alpha/5$ )	$6.40{\pm}5.56$	$0.129{\pm}0.214$	$4.12$	$+0.042$	$10.3\%$
CGPO ( $\alpha/40$ )	$0.52{\pm}0.11$	$1.098{\pm}0.217$	$92.75$	$-0.664$	$97.7\%$
$\lambda=4$	$0.34{\pm}0.00$	$1.122{\pm}0.171$	$99.23$	$-0.662$	$100.0\%$
$\lambda=16$	$6.76{\pm}4.99$	$0.450{\pm}0.202$	$8.66$	$-0.044$	$67.9\%$

TABLE II : Held-out comparison over 10,240 obstructed scenarios (mean

\pm

std.). Shaded rows denote methods achieving 100% success with positive 5th-percentile clearance. Within this subset, bold indicates the lower task cost.

Panel of Figure 7 — Fig. 5 : A. $\lambda$ evolution with our proposed bounded constraint cost. B. $\lambda$ evolution with the unbounded constraint cost used by B2N [ 4 ] with $\beta_{2}$ parameter sweep. C. Analysis of constraint aggregation. LME closely matches the exact maximum in separating colliding from collision-free trajectories and remains consistent across rollout horizons, unlike cumulative aggregation.

Metric	Diff Sim	Gazebo
Velocity tracking cost, $\ell_{t}^{\mathrm{v}}$	$0.068\pm 0.031$	$0.077\pm 0.034$
LME constraint	$0.669\pm 0.114$	$0.863\pm 0.134$
Deepest incursion ( $\mathrm{m}$ )	$0.0001\pm 0.0012$	$0.0188\pm 0.0217$

TABLE III : Sim-to-sim comparison (mean

\pm

std.).

Explore similar work

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation
Apr 12, 2026Fanxing Li, Shengyang Wang, Yuxiang Huang +5Quadrotor Control Vision-Based Robot Control
Shanghai Jiao Tong University · Kunshan Duke University
CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors
Sep 15, 2026Fabrizio Cesareo, Sebastiano Mengozzi, Nicola Mimmo +1RL Control Lyapunov-Based Control
Department of Electrical, Electronic, and Information Engineering “Guglielmo Marconi” — DEI University of Bologna, Italy
VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control
Mar 22, 2026Fanxing Li, Fangyu Sun, Tianbao Zhang +5Reinforcement Learning Quadrotor Control

Apr 12, 2026J/K move · Enter open · S save

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation

Fanxing Li, Shengyang Wang, Yuxiang Huang, Fangyu Sun+4

Shanghai Jiao Tong University · Kunshan Duke University

Quadrotor Control Vision-Based Robot Control Robot Collision Avoidance RL for Robotics

Open paper AskPDF arXiv

Obstacle avoidance is a fundamental vision-based task essential for enabling quadrotors to perform advanced applications. When planning the trajectory, existing approaches both on optimization and learning typically regard quadrotor as a point-mass model, giving path or velocity commands then tracking the commands by outer-loop controller. However, at high speeds, planned trajectories sometimes become dynamically infeasible in actual flight, which beyond the capacity of controller. In this paper, we propose a novel end-to-end policy that directly maps depth images to low-level bodyrate commands by reinforcement learning via differentiable simulation. The high-fidelity simulation in training after parameter identification significantly reduces all the gaps between training, simulation and real world. Analytical process by differentiable simulation provides accurate gradient to ensure efficiently training the low-level policy without expert guidance. The policy employs a lightweight and the most simple inference pipeline that runs without explicit mapping, backbone networks, primitives, recurrent structures, or backend controllers, nor curriculum or privileged guidance. By inferring low-level command directly to the hardware controller, the method enables full flight envelope control and avoids the dynamic-infeasible issue.Experimental results demonstrate that the proposed approach achieves the highest success rate and the lowest jerk among state-of-the-art baselines across multiple benchmarks. The policy also exhibits strong generalization, successfully deploying zero-shot in unseen, outdoor environments while reaching speeds of up to 7.5m/s as well as stably flying in the super-dense forest. This work is released at https://github.com/Fanxing-LI/avoidance.

Abstract

Explore similar work

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation

CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors

VisFly-Lab: Unified Differentiable Framework for First-Order Reinforcement Learning of Quadrotor Control