cs.ROMar 2, 2026

Learning Vision-Based Omnidirectional Navigation: A Teacher-Student Approach Using Monocular Depth Estimation

Authors: Jan Finke, Wayne Paul Martis, Adrian Schmelter, Lars Erbach, Christian Jestel, Marvin Wiedemann

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reliable obstacle avoidance in industrial settings demands 3D scene understanding, but widely used 2D LiDAR sensors perceive only a single horizontal slice of the environment, missing critical obstacles above or below the scan plane. We present a teacher-student framework for vision-based mobile robot navigation that eliminates the need for LiDAR sensors. A teacher policy trained via Proximal Policy Optimization (PPO) in NVIDIA Isaac Lab leverages privileged 2D LiDAR observations that account for the full robot footprint to learn robust navigation. The learned behavior is distilled into a student policy that relies solely on monocular depth maps predicted by a fine-tuned Depth Anything V2 model from four RGB cameras. The complete inference pipeline, comprising monocular depth estimation (MDE), policy execution, and motor control, runs entirely onboard an NVIDIA Jetson Orin AGX mounted on a DJI RoboMaster platform, requiring no external computation for inference. In simulation, the student achieves success rates of 82-96.5%, consistently outperforming the standard 2D LiDAR teacher (50-89%). In real-world experiments, the MDE-based student outperforms the 2D LiDAR teacher when navigating around obstacles with complex 3D geometries, such as overhanging structures and low-profile objects, that fall outside the single scan plane of a 2D LiDAR.

Figures & tables

Fig. 1 : Parallel training environments in NVIDIA Isaac Lab. Each environment contains a randomized set of industrial obstacles for domain randomization during policy training.
Fig. 2 : Teacher-student pipeline for learning vision-based navigation from privileged LiDAR observations.
Fig. 3 : Data collection setup for fine-tuning the MDE backbone: (a) recording platform with Orbbec Femto Bolt ToF camera, (b) example RGB image, and (c) corresponding ground-truth depth map.
PlatformJetPackLatency [ms]Avg. FPS
NVIDIA Jetson Orin NX6.198.63±2.0698.63\pm 2.0610.1
NVIDIA Jetson Orin AGX6.150.16±8.2250.16\pm 8.2219.9
NVIDIA Jetson Thor7.019.02±4.8119.02\pm 4.8152.6
TABLE I : Inference performance of the fine-tuned MDE model on NVIDIA Jetson platforms.
Fig. 4 : Collision mesh representation used for privileged LiDAR -based observations in the teacher training phase.
Fig. 5 : Teacher policy network architecture with separate actor and critic. Both process exteroceptive LiDAR observations oteo_{t}^{e} via an IMPALA encoder and proprioceptive input otpo_{t}^{p} via an MLP .
ParameterValueParameterValue
wΔdw_{\Delta d}0.5wΔaw_{\Delta a}-0.1
wlidarw_{\text{lidar}}-0.1rat_goaltr_{\text{at\_goal}}^{t}10.0
dmind_{\text{min}}0.7 mdgoalthreshd_{\text{goal}}^{\text{thresh}}0.5 m
waw_{a}-0.05rcollisionr_{\text{collision}}-50.0
rfinishedr_{\text{finished}}50.0  
TABLE II : Reward function parameters and weights.
ParameterValueParameterValue
Learning rate ( α\alpha )1×10−31\times 10^{-3}Discount factor ( γ\gamma )0.99
Horizon ( TT )96Clip range ( ϵ\epsilon )0.2
Mini-epochs5Batch size98304
Minibatch size24576Environments ( NenvN_{\text{env}} )1024
TABLE III : PPO hyperparameters for teacher policy training.
Fig. 6 : Student network architecture. It is similar to the teacher actor network but instead of 2D LiDAR data, it processes 4 depth maps through an IMPALA-CNN encoder followed by an MLP .
Fig. 7 : DJI RoboMaster platform: (a) simulation model, (b) real robot, (c) Mecanum wheel with sphere colliders, and (d) detailed collider geometry of a single Mecanum roller.
Fig. 8 : Training reward progression during teacher policy training.
Fig. 9 : Success rate distribution of the teacher policy across evaluation episodes.
ObstacleTeacherTeacherStudent
CountsLiDAR (Std.)LiDAR (Priv.)MDE (Std.)
1089%99.5%96.5%
1573.5%93%94.5%
2058%95%92%
2550%89.5%82%
TABLE IV : Teacher and student success rates in simulation for varying obstacle counts. Std. and Priv. refer to standard and privileged collision meshes, respectively (see Fig. 4 ).
Fig. 10 : Real-world test environment: an 8 m ×\times 8 m arena with obstacles of varying size and geometry used for sim-to-real evaluation.
ScenarioTeacherStudent
 LiDARMDE
Endpoint A40%80%
Endpoint B30%60%
Endpoint C40%100%
TABLE V : Real-world success rate for teacher and student across three endpoints with 10 runs each.
Fig. 11 : Visualization of the starting point and the three endpoints (axes in meters). Three trajectories for Endpoint A are shown for both the teacher and student policies during real-world deployment.
Fig. 12 : Point cloud comparison between ground-truth LiDAR (red) and MDE -predicted depth. The scale mismatch increases at larger distances.

Explore similar work

CardsList

  1. An Open-Source LiDAR and Monocular Off-Road Autonomous Navigation Stack

    Apr 3, 2026Rémi Marsal, Quentin Picard, Adrien Poiré +5Off-Road Robot NavigationMonocular Depth Estimation

    U2IS, ENSTA, Institut Polytechnique de Paris, Palaiseau, France · AMIAD, Pˆole Recherche, France

  2. Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models for Robots in Unstructured Environments

    Jul 8, 2026Erik Jagnandan, Mulugeta Haile, Gregory Barber +1Collision AvoidanceDynamic Obstacle Avoidance

    GRASP Laboratory, University of Pennsylvania, PA · U.S. Army Research Laboratory, Aberdeen Proving Ground, MD

  3. Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

    Mar 21, 2026Kai Li, Shiyu ZhaoVision-Based Robot ControlMobile Robotics

    College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.