Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore. · Beihang University, Beijing, China. · The University of Queensland, Australia.
Autonomous collision-free navigation in cluttered environments requires safe decision-making under partial observability with both static structure and dynamic obstacles. We present PanoDP, a communication-free learning framework that combines four-view panoramic depth perception with differentiable-physics-based training signals. PanoDP encodes panoramic depth using a lightweight CNN and optimizes policies with dense differentiable collision and motion-feasibility terms, improving training stability beyond sparse terminal collisions. We evaluate PanoDP on a controlled ring-to-center benchmark with systematic sweeps over agent count, obstacle density/layout, and dynamic behaviors, and further test out-of-distribution generalization in an external simulator (e.g., AirSim). Across settings, PanoDP increases collision-free and completion rates over single-view and non-physics-guided baselines under matched training budgets, and ablations (view masking, rotation augmentation) confirm the policy leverages 360-degree information. Code will be open source upon acceptance.
Figures & tables
Fig. 1 : Panoramic depth construction. (a) Four cameras with 100∘ FOV cover the full azimuth with overlapping regions. (b) Cosine-squared blend weights wk(θ) ensure smooth transitions. (c) The resulting equirectangular panorama Pti wraps continuously at 0∘/360∘ .
Fig. 2 : PanoDP training pipeline. At each time step the four onboard depth images are stitched into a 360∘ panorama, encoded by the circular CNN, fused with the body-frame state vector in a GRU cell, and mapped to an acceleration command. Because the dynamics integrator ( 13 )–( 14 ) is fully differentiable, the trajectory loss L can be back-propagated through the entire T -step unrolled graph (dashed arrows), yielding exact gradients ∂L/∂θ that update all learnable parameters in a single pass.
Fig. 3 : Panorama and forward-depth training comparison over 50 K iterations (6 metrics). Insets zoom into the converged tail (40–50 K steps). PanoDP (blue, solid) consistently outperforms the forward-only baseline DPD † (grey, dashed) across all metrics.
Fig. 4 : Ablation training curves over 50 K iterations (6 metrics), with tail zoom (40–50 K). Full PanoDP converges most stably with lowest variance. Removing the GRU raises jerk and snap; removing circular convolutions or using a flat MLP yields higher plateaus.
Fig. 5 : Ablation bar chart (tail-mean at 50 K iterations) across eight training metrics. PanoDP (Ours, red border) achieves the best or near-best value on every metric. Stars ( ⋆ ) mark the best method per subplot.
Method
Loss ↓
Succ. ↑
AR ↑
Col. ↓
Lv↓
Lobj↓
Jerk ↓
Snap ↓
Acc. ↓
G.Aff. ↓
Speed ↑
DPD † (Forward)
0.394
0.935
1.479
0.009
0.118
0.093
15.9110
3.6412
2.5618
0.1256
1.579
PanoDP (Ours)
0.289
0.966
1.558
0.003
0.091
0.071
15.0810
3.4216
2.5628
0.0912
1.617
w/o GRU
0.359
0.938
1.491
0.005
0.120
0.078
26.3286
5.3502
3.0066
0.1145
1.597
w/o Circular Conv
0.309
0.959
1.535
0.004
0.099
0.075
15.7935
3.5793
2.5514
0.0851
1.605
Concat MLP
0.313
0.961
1.533
0.003
0.102
0.075
16.2036
3.6878
2.5590
0.0892
1.601
TABLE I : Training ablation at 50 K iterations (tail-mean). Succ.: training rollout success rate ∈[0,1] ; AR: success-weighted mean speed (Succ. × Speed); G.Aff.: mean squared positive altitude (ground affinity).
(a) Scale Robustness ( ρobs=1.0 , vdrone=rand )
Method
N=64
N=128
N=256
N=512
SR( % ) ↑
CR ↓
MFCT(s) ↑
SR( % ) ↑
CR ↓
MFCT(s) ↑
SR( % ) ↑
CR ↓
MFCT(s) ↑
SR( % ) ↑
CR ↓
MFCT(s) ↑
PanoDP (Ours)
84.4
0.27
16.8
80.7
0.38
32.5
80.6
0.80
37.5
87.2
0.88
48.6
D-CBF [ 8 ]
82.8
0.30
9.1
78.1
0.43
19.9
78.6
0.76
25.0
84.4
1.03
40.4
RAMFT [ 10 ]
86.5
0.24
10.3
74.7
0.46
27.4
74.2
0.83
31.3
79.4
1.32
33.6
CBF-AF [ 9 ]
67.7
0.60
7.7
66.9
0.61
17.0
67.7
1.04
24.9
74.0
1.81
40.8
TABLE II : Stress-test benchmark (circle-swap): three-panel comparison across 12 configurations (3 maps each). Bold = best; underline = second. Non-learning baselines receive privileged neighbor state ( rs=24 m); PanoDP uses only onboard depth.
Fig. 6 : Single-camera occlusion robustness. (a)–(d) 360° panoramic depth with one camera blocked (yellow: max-range fill). (e) In multi-drone tests, blocking the right camera drops SR by 24 pp, while blocking the left has negligible effect, revealing a directional convention. (f) In single-drone control, any non-front camera is expendable ( < 1 pp drop), showing the asymmetry stems from coordination. Shaded bands: ± 1 std over 10 seeds.
Fig. 7 : Circle-swap with N=64 . Left: Panoramic (93.8% success, 4 collisions); Right: Forward-only (59.4%, 26 collisions). The swarm self-organizes a counterclockwise rotation through the congested center. Bottom: per-drone depth tiles; red X = frozen after collision.
Fig. 8 : Sim-to-sim transfer in AirSim: a multi-drone swarm navigates a dense bamboo forest with photorealistic depth and stochastic wind. A policy trained in a lightweight differentiable simulator with spherical obstacles transfers to AirSim without retraining, showing robustness to visual domain shift and aerodynamic perturbations.
Reliable obstacle avoidance in industrial settings demands 3D scene understanding, but widely used 2D LiDAR sensors perceive only a single horizontal slice of the environment, missing critical obstacles above or below the scan plane. We present a teacher-student framework for vision-based mobile robot navigation that eliminates the need for LiDAR sensors. A teacher policy trained via Proximal Policy Optimization (PPO) in NVIDIA Isaac Lab leverages privileged 2D LiDAR observations that account for the full robot footprint to learn robust navigation. The learned behavior is distilled into a student policy that relies solely on monocular depth maps predicted by a fine-tuned Depth Anything V2 model from four RGB cameras. The complete inference pipeline, comprising monocular depth estimation (MDE), policy execution, and motor control, runs entirely onboard an NVIDIA Jetson Orin AGX mounted on a DJI RoboMaster platform, requiring no external computation for inference. In simulation, the student achieves success rates of 82-96.5%, consistently outperforming the standard 2D LiDAR teacher (50-89%). In real-world experiments, the MDE-based student outperforms the 2D LiDAR teacher when navigating around obstacles with complex 3D geometries, such as overhanging structures and low-profile objects, that fall outside the single scan plane of a 2D LiDAR.
Efficiently learning safe and agile quadruped navigation in densely cluttered environments remains difficult: existing methods often lack safety and agility, or become conservative in complex scenes and require long training schedules. We propose SEA-Nav (Safe, Efficient, and Agile Navigation), a safe reinforcement learning framework for quadruped navigation in cluttered environments. A differentiable control barrier function (CBF) shield constrains the policy to produce safe velocity commands. An adaptive collision-state initialization mechanism increases the probability of learning from safety-critical near-collision experience. An action regularization term further suppresses infeasible commands for physical deployment. The policy converges after about one hour of training on a single RTX 4090 and transfers zero-shot to real-world cluttered scenes.
Shiyi Chen, Mingye Yang, Haiyi Liu +6
Tsinghua University, Beijing, China. · Imperial College London, London, UK.
Existing visual navigation models often aim to develop foundation models that can generalize robot navigation across diverse platforms. However, many of these models are prone to collisions when deployed in complex indoor environments, particularly in structured layouts and narrow passages. To address this problem, we propose a depth image- and point-goal-conditioned navigation system, DevGRU. The proposed system employs an action predictor (AP) that generates collision-aware future trajectories, enabling effective avoidance of immediate obstacles. In conjunction with a collision predictor, the AP further compensates for errors accumulated in the goal pose estimation and proactively mitigates future deviations. To evaluate our method, we conducted experiments across nine different scenes and three state-of-the-art approaches - ViNT, NoMaD, and NavDP - as well as four additional variants of ViNT and NoMaD. In terms of navigation performance, DevGRU significantly outperforms ViNT and NoMaD by a large margin. In addition, the proposed model has a relatively small number of trainable parameters, resulting in the fastest inference time among the baselines, particularly outperforming NavDP by 7x in model size and 17x in inference time.