Reliable obstacle avoidance in industrial settings demands 3D scene understanding, but widely used 2D LiDAR sensors perceive only a single horizontal slice of the environment, missing critical obstacles above or below the scan plane. We present a teacher-student framework for vision-based mobile robot navigation that eliminates the need for LiDAR sensors. A teacher policy trained via Proximal Policy Optimization (PPO) in NVIDIA Isaac Lab leverages privileged 2D LiDAR observations that account for the full robot footprint to learn robust navigation. The learned behavior is distilled into a student policy that relies solely on monocular depth maps predicted by a fine-tuned Depth Anything V2 model from four RGB cameras. The complete inference pipeline, comprising monocular depth estimation (MDE), policy execution, and motor control, runs entirely onboard an NVIDIA Jetson Orin AGX mounted on a DJI RoboMaster platform, requiring no external computation for inference. In simulation, the student achieves success rates of 82-96.5%, consistently outperforming the standard 2D LiDAR teacher (50-89%). In real-world experiments, the MDE-based student outperforms the 2D LiDAR teacher when navigating around obstacles with complex 3D geometries, such as overhanging structures and low-profile objects, that fall outside the single scan plane of a 2D LiDAR.
Figures & tables
Fig. 1 : Parallel training environments in NVIDIA Isaac Lab. Each environment contains a randomized set of industrial obstacles for domain randomization during policy training.
Fig. 2 : Teacher-student pipeline for learning vision-based navigation from privileged LiDAR observations.
Fig. 3 : Data collection setup for fine-tuning the MDE backbone: (a) recording platform with Orbbec Femto Bolt ToF camera, (b) example RGB image, and (c) corresponding ground-truth depth map.
Platform
JetPack
Latency [ms]
Avg. FPS
NVIDIA Jetson Orin NX
6.1
98.63±2.06
10.1
NVIDIA Jetson Orin AGX
6.1
50.16±8.22
19.9
NVIDIA Jetson Thor
7.0
19.02±4.81
52.6
TABLE I : Inference performance of the fine-tuned MDE model on NVIDIA Jetson platforms.
Fig. 4 : Collision mesh representation used for privileged LiDAR -based observations in the teacher training phase.
Fig. 5 : Teacher policy network architecture with separate actor and critic. Both process exteroceptive LiDAR observations ote via an IMPALA encoder and proprioceptive input otp via an MLP .
Parameter
Value
Parameter
Value
wΔd
0.5
wΔa
-0.1
wlidar
-0.1
rat_goalt
10.0
dmin
0.7 m
dgoalthresh
0.5 m
wa
-0.05
rcollision
-50.0
rfinished
50.0
TABLE II : Reward function parameters and weights.
Parameter
Value
Parameter
Value
Learning rate ( α )
1×10−3
Discount factor ( γ )
0.99
Horizon ( T )
96
Clip range ( ϵ )
0.2
Mini-epochs
5
Batch size
98304
Minibatch size
24576
Environments ( Nenv )
1024
TABLE III : PPO hyperparameters for teacher policy training.
Fig. 6 : Student network architecture. It is similar to the teacher actor network but instead of 2D LiDAR data, it processes 4 depth maps through an IMPALA-CNN encoder followed by an MLP .
Fig. 7 : DJI RoboMaster platform: (a) simulation model, (b) real robot, (c) Mecanum wheel with sphere colliders, and (d) detailed collider geometry of a single Mecanum roller.
Fig. 8 : Training reward progression during teacher policy training.
Fig. 9 : Success rate distribution of the teacher policy across evaluation episodes.
Obstacle
Teacher
Teacher
Student
Counts
LiDAR (Std.)
LiDAR (Priv.)
MDE (Std.)
10
89%
99.5%
96.5%
15
73.5%
93%
94.5%
20
58%
95%
92%
25
50%
89.5%
82%
TABLE IV : Teacher and student success rates in simulation for varying obstacle counts. Std. and Priv. refer to standard and privileged collision meshes, respectively (see Fig. 4 ).
Fig. 10 : Real-world test environment: an 8 m × 8 m arena with obstacles of varying size and geometry used for sim-to-real evaluation.
Scenario
Teacher
Student
LiDAR
MDE
Endpoint A
40%
80%
Endpoint B
30%
60%
Endpoint C
40%
100%
TABLE V : Real-world success rate for teacher and student across three endpoints with 10 runs each.
Fig. 11 : Visualization of the starting point and the three endpoints (axes in meters). Three trajectories for Endpoint A are shown for both the teacher and student policies during real-world deployment.
Fig. 12 : Point cloud comparison between ground-truth LiDAR (red) and MDE -predicted depth. The scale mismatch increases at larger distances.
Off-road autonomous navigation demands reliable 3D perception for robust obstacle detection in challenging unstructured terrain. While LiDAR is accurate, it is costly and power-intensive. Monocular depth estimation using foundation models offers a lightweight alternative, but its integration into outdoor navigation stacks remains underexplored. We present an open-source off-road navigation stack supporting both LiDAR and monocular 3D perception without task-specific training. For the monocular setup, we combine zero-shot depth prediction (Depth Anything V2) with metric depth rescaling using sparse SLAM measurements (VINS-Mono). Two key enhancements improve robustness: edge-masking to reduce obstacle hallucination and temporal smoothing to mitigate the impact of SLAM instability. The resulting point cloud is used to generate a robot-centric 2.5D elevation map for costmap-based planning. Evaluated in photorealistic simulations (Isaac Sim) and real-world unstructured environments, the monocular configuration matches high-resolution LiDAR performance in most scenarios, demonstrating that foundation-model-based monocular depth estimation is a viable LiDAR alternative for robust off-road navigation. By open-sourcing the navigation stack and the simulation environment, we provide a complete pipeline for off-road navigation as well as a reproducible benchmark. Code available at https://github.com/LARIAD/Offroad-Nav.
Rémi Marsal, Quentin Picard, Adrien Poiré +5
U2IS, ENSTA, Institut Polytechnique de Paris, Palaiseau, France · AMIAD, Pˆole Recherche, France
Dynamic obstacle avoidance in unstructured outdoor environments remains a critical challenge for autonomous mobile robots, particularly when large-scale robot-specific training data and simulation-based policies are impractical. We present a data-efficient, interpretable method for vision-based dynamic obstacle avoidance that operates entirely on real-world data, avoiding the sim-to-real transfer problem inherent in simulation-trained policies. Our approach leverages UniDepth, a large pretrained monocular depth estimation model, to produce dense depth maps from RGB video without requiring stereo cameras or LiDAR at inference time. Dynamic obstacle avoidance is achieved by extending the SuperPoint and SuperGlue feature correspondence pipeline to track keypoints across long frame sequences, projecting their 2D pixel-space positions into 3D using camera intrinsics and predicted depth, running bundle adjustment initialized from these 3D keypoints, and computing per-keypoint time-to-collision (TTC). A 2D motion primitive in the ground plane is then selected to move the robot away from the closest point of approach of the minimum-TTC keypoint. Evaluated on real-world data from the M3ED dataset, our pipeline achieves a precision of 0.49 and a recall of 0.38 in identifying frames with a ground truth TTC below 1 second, and correctly generates the evasive motion direction in 84% of true positive detections. Crucially, it detects at least one frame with TTC less than 1 second for 20 out of 22 unique physical obstacles present in our test sequences. Unlike end-to-end learned methods that demand thousands of hours of robot-specific training data, our approach eliminates model training entirely, requiring only 74 seconds of data for hyperparameter tuning. This demonstrates exceptional data efficiency while preserving interpretable and generalizable behavior across diverse obstacle types.
Erik Jagnandan, Mulugeta Haile, Gregory Barber +1
GRASP Laboratory, University of Pennsylvania, PA · U.S. Army Research Laboratory, Aberdeen Proving Ground, MD
Vision-based policies are widely applied in robotics for tasks such as manipulation and locomotion. On lightweight mobile robots, however, they face a trilemma of limited scene transferability, restricted onboard computation resources, and sensor hardware cost. To address these issues, we propose a knowledge distillation approach that transfers knowledge from an information-rich, appearance invariant omniview depth policy to a lightweight monocular policy. The key idea is to train the student not only to mimic the expert actions but also to align with the latent embeddings of the omni view depth teacher. Experiments demonstrate that omni-view and depth inputs improve the scene transfer and navigation performance, and that the proposed distillation method enhances the performance of a singleview monocular policy, compared with policies solely imitating actions. Real world experiments further validate the effectiveness and practicality of our approach. Code will be released publicly.
Kai Li, Shiyu Zhao
College of Computer Science and Technology at Zhejiang University, Hangzhou, China. · School of Engineering at Westlake University, Hangzhou, China.