PanoDP: Learning Collision-Free Navigation with Panoramic Depth and Differentiable Physics

Hao Zhong; Pei Chi; Jiang Zhao; Shenghai Yuan; Xuyang Gao; Thien-Minh Nguyen; Lihua Xie

cs.ROMar 8, 2026

PanoDP: Learning Collision-Free Navigation with Panoramic Depth and Differentiable Physics

Authors: Hao Zhong, Pei Chi, Jiang Zhao, Shenghai Yuan, Xuyang Gao, Thien-Minh Nguyen, Lihua Xie

Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore. · Beihang University, Beijing, China. · The University of Queensland, Australia.

Abstract

Autonomous collision-free navigation in cluttered environments requires safe decision-making under partial observability with both static structure and dynamic obstacles. We present PanoDP, a communication-free learning framework that combines four-view panoramic depth perception with differentiable-physics-based training signals. PanoDP encodes panoramic depth using a lightweight CNN and optimizes policies with dense differentiable collision and motion-feasibility terms, improving training stability beyond sparse terminal collisions. We evaluate PanoDP on a controlled ring-to-center benchmark with systematic sweeps over agent count, obstacle density/layout, and dynamic behaviors, and further test out-of-distribution generalization in an external simulator (e.g., AirSim). Across settings, PanoDP increases collision-free and completion rates over single-view and non-physics-guided baselines under matched training budgets, and ablations (view masking, rotation augmentation) confirm the policy leverages 360-degree information. Code will be open source upon acceptance.

Figures & tables

Fig. 1 : Panoramic depth construction. (a) Four cameras with $100^{\circ}$ FOV cover the full azimuth with overlapping regions. (b) Cosine-squared blend weights $w_{k}(\theta)$ ensure smooth transitions. (c) The resulting equirectangular panorama $\mathbf{P}^{i}_{t}$ wraps continuously at $0^{\circ}/360^{\circ}$ .

Fig. 2 : PanoDP training pipeline. At each time step the four onboard depth images are stitched into a

360^{\circ}

panorama, encoded by the circular CNN, fused with the body-frame state vector in a GRU cell, and mapped to an acceleration command. Because the dynamics integrator ( 13 )–( 14 ) is fully differentiable, the trajectory loss

\mathcal{L}

can be back-propagated through the entire

T

-step unrolled graph (dashed arrows), yielding exact gradients

\partial\mathcal{L}/\partial\theta

that update all learnable parameters in a single pass.

Fig. 3 : Panorama and forward-depth training comparison over 50 K iterations (6 metrics). Insets zoom into the converged tail (40–50 K steps). PanoDP (blue, solid) consistently outperforms the forward-only baseline DPD † (grey, dashed) across all metrics.

Fig. 4 : Ablation training curves over 50 K iterations (6 metrics), with tail zoom (40–50 K). Full PanoDP converges most stably with lowest variance. Removing the GRU raises jerk and snap; removing circular convolutions or using a flat MLP yields higher plateaus.

Fig. 5 : Ablation bar chart (tail-mean at 50 K iterations) across eight training metrics. PanoDP (Ours, red border) achieves the best or near-best value on every metric. Stars (

\star

) mark the best method per subplot.

Method	Loss $\downarrow$	Succ. $\uparrow$	AR $\uparrow$	Col. $\downarrow$	$\mathcal{L}_{v}\downarrow$	$\mathcal{L}_{obj}\downarrow$	Jerk $\downarrow$	Snap $\downarrow$	Acc. $\downarrow$	G.Aff. $\downarrow$	Speed $\uparrow$
DPD † (Forward)	0.394	0.935	1.479	0.009	0.118	0.093	15.9110	3.6412	2.5618	0.1256	1.579
PanoDP (Ours)	0.289	0.966	1.558	0.003	0.091	0.071	15.0810	3.4216	2.5628	0.0912	1.617
w/o GRU	0.359	0.938	1.491	0.005	0.120	0.078	26.3286	5.3502	3.0066	0.1145	1.597
w/o Circular Conv	0.309	0.959	1.535	0.004	0.099	0.075	15.7935	3.5793	2.5514	0.0851	1.605
Concat MLP	0.313	0.961	1.533	0.003	0.102	0.075	16.2036	3.6878	2.5590	0.0892	1.601

TABLE I : Training ablation at 50 K iterations (tail-mean). Succ.: training rollout success rate

\in[0,1]

; AR: success-weighted mean speed (Succ.

\times

Speed); G.Aff.: mean squared positive altitude (ground affinity).

(a) Scale Robustness ( $\rho_{\mathrm{obs}}\!=\!1.0$ , $v_{\mathrm{drone}}\!=\!\mathrm{rand}$ )
Method	$N\!=\!64$			$N\!=\!128$			$N\!=\!256$			$N\!=\!512$
	SR( $\%$ ) $\uparrow$	CR $\downarrow$	MFCT(s) $\uparrow$	SR( $\%$ ) $\uparrow$	CR $\downarrow$	MFCT(s) $\uparrow$	SR( $\%$ ) $\uparrow$	CR $\downarrow$	MFCT(s) $\uparrow$	SR( $\%$ ) $\uparrow$	CR $\downarrow$	MFCT(s) $\uparrow$
PanoDP (Ours)	84.4	0.27	16.8	80.7	0.38	32.5	80.6	0.80	37.5	87.2	0.88	48.6
D-CBF [ 8 ]	82.8	0.30	9.1	78.1	0.43	19.9	78.6	0.76	25.0	84.4	1.03	40.4
RAMFT [ 10 ]	86.5	0.24	10.3	74.7	0.46	27.4	74.2	0.83	31.3	79.4	1.32	33.6
CBF-AF [ 9 ]	67.7	0.60	7.7	66.9	0.61	17.0	67.7	1.04	24.9	74.0	1.81	40.8

TABLE II : Stress-test benchmark (circle-swap): three-panel comparison across 12 configurations (3 maps each). Bold = best; underline = second. Non-learning baselines receive privileged neighbor state (

r_{s}\!=\!24

m); PanoDP uses only onboard depth.

Fig. 6 : Single-camera occlusion robustness. (a)–(d) 360° panoramic depth with one camera blocked (yellow: max-range fill). (e) In multi-drone tests, blocking the right camera drops SR by 24 pp, while blocking the left has negligible effect, revealing a directional convention. (f) In single-drone control, any non-front camera is expendable ( $<$ 1 pp drop), showing the asymmetry stems from coordination. Shaded bands: $\pm$ 1 std over 10 seeds.

Fig. 7 : Circle-swap with $N\!=\!64$ . Left: Panoramic (93.8% success, 4 collisions); Right: Forward-only (59.4%, 26 collisions). The swarm self-organizes a counterclockwise rotation through the congested center. Bottom: per-drone depth tiles; red X = frozen after collision.

Fig. 8 : Sim-to-sim transfer in AirSim: a multi-drone swarm navigates a dense bamboo forest with photorealistic depth and stochastic wind. A policy trained in a lightweight differentiable simulator with spherical obstacles transfers to AirSim without retraining, showing robustness to visual domain shift and aerodynamic perturbations.

Panoramic Depth Estimation

See all 12 papers

Learning Vision-Based Omnidirectional Navigation: A Teacher-Student Approach Using Monocular Depth Estimation
Mar 2, 2026Jan Finke, Wayne Paul Martis, Adrian Schmelter +3Robot Navigation Vision-Based Robot Control
SEA-Nav: Efficient Policy Learning for Safe and Agile Quadruped Navigation in Cluttered Environments
Mar 10, 2026Shiyi Chen, Mingye Yang, Haiyi Liu +6Quadruped Robot Navigation Robot Navigation
Tsinghua University, Beijing, China. · Imperial College London, London, UK.
DevGRU: Depth-guided Visual Navigation using a Collision-aware Recurrent Model
Aug 19, 2026Kyung Min Han, Eunsom Kim, Young J. KimGated Recurrent Unit Robot Navigation

Mar 2, 2026J/K move · Enter open · S save

Learning Vision-Based Omnidirectional Navigation: A Teacher-Student Approach Using Monocular Depth Estimation

Jan Finke, Wayne Paul Martis, Adrian Schmelter, Lars Erbach+2

Robot Navigation Vision-Based Robot Control

Open paper AskPDF arXiv

Reliable obstacle avoidance in industrial settings demands 3D scene understanding, but widely used 2D LiDAR sensors perceive only a single horizontal slice of the environment, missing critical obstacles above or below the scan plane. We present a teacher-student framework for vision-based mobile robot navigation that eliminates the need for LiDAR sensors. A teacher policy trained via Proximal Policy Optimization (PPO) in NVIDIA Isaac Lab leverages privileged 2D LiDAR observations that account for the full robot footprint to learn robust navigation. The learned behavior is distilled into a student policy that relies solely on monocular depth maps predicted by a fine-tuned Depth Anything V2 model from four RGB cameras. The complete inference pipeline, comprising monocular depth estimation (MDE), policy execution, and motor control, runs entirely onboard an NVIDIA Jetson Orin AGX mounted on a DJI RoboMaster platform, requiring no external computation for inference. In simulation, the student achieves success rates of 82-96.5%, consistently outperforming the standard 2D LiDAR teacher (50-89%). In real-world experiments, the MDE-based student outperforms the 2D LiDAR teacher when navigating around obstacles with complex 3D geometries, such as overhanging structures and low-profile objects, that fall outside the single scan plane of a 2D LiDAR.

Abstract

Panoramic Depth Estimation

Explore similar work

Learning Vision-Based Omnidirectional Navigation: A Teacher-Student Approach Using Monocular Depth Estimation

SEA-Nav: Efficient Policy Learning for Safe and Agile Quadruped Navigation in Cluttered Environments

DevGRU: Depth-guided Visual Navigation using a Collision-aware Recurrent Model