E2DRL: End-to-End Differentiable Reinforcement Learning for 3D Obstacle Avoidance in Quadruped Robots

Cheng Fan; Bingyang Zhou; Gang Yang; Zixuan Zhuang; Aoqian Zhang; Pengzhi Yang; Xin Liu; Chunzheng Wang

cs.ROOct 14, 2026

E2DRL: End-to-End Differentiable Reinforcement Learning for 3D Obstacle Avoidance in Quadruped Robots

Authors: Cheng Fan, Bingyang Zhou, Gang Yang, Zixuan Zhuang, Aoqian Zhang, Pengzhi Yang, Xin Liu, Chunzheng Wang,

Organizations: Department of Electrical and Computer Engineering, National University of Singapore, Singapore.

Abstract

Differentiable reinforcement learning shows strong promise for robot policy learning, but its application to legged locomotion has remained limited to blind walking or simulated 2D tasks. Extending it to end-to-end visual obstacle avoidance is challenging because hard collision constraints and early termination upon contact corrupt first-order gradients, trapping optimizers in local minima. We present E2DRL, which addresses this with a soft differentiable obstacle-interaction formulation: penetration penalties across multiple body keypoints with velocity-conditioned stop-gradient shaping to provide the first-order training signals for 3D obstacle avoidance. To realize a deployable perception-control system, a privileged teacher trained via Soft Analytic Policy Optimization (SAPO) in a differentiable simulator is distilled into a recurrent depth-based student via DAgger, then blended with a base locomotion policy using a terrain-aware gating module for deployment. In simulation, E2DRL achieves higher sample efficiency than PPO and demonstrates superior velocity-tracking performance and 3D obstacle-avoidance capability compared with both PPO and Duality-based DCBF NMPC. We present the first sim-to-real validation of end-to-end differentiable RL for 3D obstacle avoidance on a Unitree Go2 in static and dynamic environments.

Figures & tables

Fig. 1 : Overview of E2DRL. We first train a privileged teacher with SAPO using the proposed soft differentiable obstacle-interaction formulation. The teacher is then distilled into a recurrent depth-based student via DAgger. For deployment, a terrain-aware gating module blends the student policy with a nominal locomotion policy.

Symbol	Description	Formulation	Weight
Task tracking
$r^{\text{lin}}$	Linear velocity tracking	$-\sqrt{(v_{x}^{\text{cmd}}-v_{x})^{2}+(v_{y}^{\text{cmd}}-v_{y})^{2}}$	$2.0$
$r^{\text{ang}}$	Yaw rate tracking	$-\sqrt{(\omega_{z}^{\text{cmd}}-\omega_{z})^{2}}$	$1.0$
Stability
$r^{\text{ang}_{xy}}$	Roll/pitch angular velocity	$-\sum_{i\in\{x,y\}}\omega_{i}^{2}$	$0.05$
$r^{\text{height}}$	Base height tracking	$-\|h_{\text{base}}-0.34\|$	$4.0$

TABLE I : Reward terms used during training.

Panel of Figure 3 — Fig. 2 : 3D obstacle avoidance performance of the robot in a simulated room environment.

Fig. 3 : Average discounted reward over training for PPO, SAPO, and SHAC across five random seeds. The solid line denotes the mean, the darker shaded region denotes one standard deviation, and the lighter shaded region spans the minimum and maximum across seeds.

Panel of Figure 5 — Fig. 4 : Average velocity-tracking performance at a commanded forward speed of $1.2\,$ m/s in two obstacle scenarios: an overhanging obstacle with a lower-surface height of $0.28\,$ m and a ground obstacle with a width of $0.8\,$ m.

Panel of Figure 6 — Fig. 5 : Real-world static obstacle avoidance on the Unitree Go2. The quadruped robot successfully avoids overhanging and ground obstacles across a variety of static indoor and outdoor scenarios.

Panel of Figure 7 — Fig. 6 : Collision-free success rates computed over multiple experiments under varying obstacle difficulty for overhanging and ground obstacle avoidance.

Panel of Figure 8 — Fig. 7 : Real-world dynamic obstacle avoidance. The robot reacts to sequentially introduced overhanging and ground obstacles. The images are labeled (a)–(d) from top to bottom.

Panel of Figure 9 — Fig. 8 : Actor-gradient magnitude during SAPO training with the differentiable friction model in ( 4 )–( 5 ) and the default Warp friction model: (a) total actor gradient $|\nabla\mathcal{L}|$ during the walking stage, and (b) obstacle-term gradient $|\nabla_{\mathrm{obs}}\mathcal{L}|$ across the transition into the obstacle-avoidance stage.

Panel of Figure 10 — Fig. S1 : Visualization of the selected body keypoints and the soft penetration penalty from the side (left) and top (right) views. Red markers denote the selected keypoints, and blue lines indicate their penetration depths.

Panel of Figure 11 — Fig. S2 : Obstacle-related gradients from the SAPO actor loss along a representative crawling rollout. From left to right, the panels show $\partial L_{\mathrm{obs}}/\partial x$ , $\partial L_{\mathrm{obs}}/\partial y$ , and $\partial L_{\mathrm{obs}}/\partial z$ as functions of longitudinal position. Purple denotes SAPO-Soft with velocity stop-gradient, and green denotes the otherwise identical formulation without it.

Panel of Figure 12 — Fig. S3 : Obstacle-related gradients from the SAPO actor loss along a representative bypassing rollout. From left to right, the panels show $\partial L_{\mathrm{obs}}/\partial x$ , $\partial L_{\mathrm{obs}}/\partial y$ , and $\partial L_{\mathrm{obs}}/\partial z$ . Removing the velocity stop-gradient introduces a dominant negative longitudinal component, in addition to the geometric gradients that drive lateral and vertical obstacle avoidance.

Panel of Figure 13 — Fig. S4 : Forward-velocity tracking with and without the velocity stop-gradient for an overhanging obstacle with a clearance height of $0.26\,$ m (left) and a ground obstacle with a width of $0.90\,$ m (right). The commanded forward velocity is $1.2\,$ m/s in both scenarios.

Panel of Figure 14 — Fig. S5 : Comparison of obstacle-avoidance success rate between the teacher and the student for overhanging obstacles (left) and ground obstacles (right).

Convergence criterion	Steps (million)		Time (min)
	SAPO	PPO	SAPO	PPO
Reward plateau	17.0	53.0	366	130
Reward stably above 90	10.5	60.0	226	147
Avoidance success-rate plateau	16.0	67.0	344	166

TABLE S1 : Environment steps and wall-clock time required by SAPO and PPO to reach convergence under three convergence criteria. Wall-clock times were measured on the same hardware and training configuration.

Parameter	Tested value	Min. overhanging-obstacle height (m)	Max. ground-obstacle width (m)
$\sigma$	0.25	0.29	0.90
	0.50 (base)	0.24	1.00
	0.75	0.25	1.00
$K_{\mathrm{overhanging}}$	20	0.25	n/a
	40 (base)	0.24	n/a
	60	0.24	n/a

TABLE S2 : Sensitivity to

\sigma

and the task-specific penetration weights

K

. Each obstacle limit is the most challenging geometry at which success is at least 90%. Bold entries indicate base settings; n/a denotes a metric not applicable to the task-specific sweep.

Parameter	Unit	Robot geometry	Ground	Effective (pair average)
$k_{n}$ ( contact_ke )	N/m	$4.0\times 10^{3}$	$1.0\times 10^{5}$	$5.2\times 10^{4}$
$k_{d}$ ( contact_kd )	N $\cdot$ s/m 2	$1.0\times 10^{3}$	$1.0\times 10^{3}$	$1.0\times 10^{3}$
$k_{t}$ ( contact_kf )	N $\cdot$ s/m	$3.0\times 10^{2}$	$1.0\times 10^{3}$	$6.5\times 10^{2}$
$\mu$ ( contact_mu )	dimensionless	$1.0$	$1.0$	$1.0$

TABLE S3 : Contact and friction parameters used in all simulation experiments.

Item	Value
Tuning scheme	Automatic (target-entropy based)
Initial value $\alpha_{0}$	$0.01$
Target entropy	$-0.6$
Optimizer for $\alpha$	AdamW, learning rate $2\times 10^{-3}$
Observed range during training	$0.006$ to $0.02$
Observed median during training	$\approx 0.01$

TABLE S4 : Configuration and observed values of the entropy coefficient

\alpha

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,704 papers

Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition
Oct 26, 2026Yuanzhi He, Victor Romero-Cano, Jiayu Xu +2Reinforcement Learning Robotic RL
School of Computational and Mathematical Sciences, Cardiff University, United Kingdom
Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation
Apr 12, 2026Fanxing Li, Shengyang Wang, Yuxiang Huang +5Quadrotor Control Vision-Based Robot Control
Shanghai Jiao Tong University · Kunshan Duke University
Constrained Differentiable Policy Learning for Safe Quadrotor Navigation
Oct 12, 2026Yan Rui Tan, Praveen Krishnapur, Zeyu Feng +3Robot Policy Learning Constrained RL
National University of Singapore, Singapore. · Agency for Science, Technology and Research (A*STAR), Singapore.

Oct 26, 2026J/K move · Enter open · S save

Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition

Yuanzhi He, Victor Romero-Cano, Jiayu Xu, Juan David Hernández+1

School of Computational and Mathematical Sciences, Cardiff University, United Kingdom

Reinforcement Learning Robotic RL Robot Learning Robot Motion Planning Reward Shaping

Open paper AskPDF arXiv

Training robotic manipulation policies in simulation is widely adopted due to safety and cost efficiency; however, tasks involving obstacles remain under-performing and often converge slowly under Deep Reinforcement Learning (DRL). To address this challenge, we propose a task decomposition framework for DRL that integrates Transfer Learning (TL) with Reward Redesign (RR) to improve learning efficiency and robustness in obstacle environments. The TL strategy selectively transfers only the first-layer critic parameters across structurally related tasks, since that layer captures general state--action representations while deeper layers encode reward- and obstacle-specific value geometry; RR decomposes tasks into staged sub-objectives to provide denser optimization signals. Experiments on the PyBullet Panda platform show consistent gains in obstacle settings. On Push with obstacle, our method achieves

0.83 \pm 0.05

, outperforming the strongest baseline (Truncated Quantile Critics (TQC),

0.77 \pm 0.06

) by approximately

7.8\%

. On Slide with obstacle, performance reaches

0.95 \pm 0.02

, exceeding Soft Actor-Critic (SAC) (

0.84 \pm 0.06

) by about

11\%

success rates. In addition to higher final performance, forward transfer reduces convergence time by up to

70\%

in obstacle tasks, and RR shortens sparse-reward Pick-and-Place training from roughly

0.75

M to

0.20

M steps (

\approx73\%

reduction). The best policy is deployed on a real Franka Emika FR3 robot as a qualitative feasibility demonstration.

Abstract

Reinforcement Learning

Explore similar work

Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition

Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation

Constrained Differentiable Policy Learning for Safe Quadrotor Navigation