Reliable navigation with a single monocular limited field-of-view camera is challenging because the lack of depth measurements and limited spatial awareness complicate collision checking. Existing approaches to tackle these challenges fall into two categories. Model-based planners can incorporate explicit collision-avoidance guidance which leads to better generalization but are often computationally expensive and require manual tuning of the algorithmic parameters. In contrast, end-to-end approaches trained by imitation learning can be flexible but typically encode safety only implicitly and thus can suffer from high collision rates in novel scenarios. In this paper, we present a middle ground through model-based imitation learning. The model-based part is a learned collision model that can map a sequence of past images and a candidate control sequence to a distribution of worst-case obstacle clearance. This predictive distribution in turn is used to define a collision-risk cost in a trajectory optimization formulation. The imitation component uses human demonstrations to learn scene-dependent balancing of collision risk with other task costs such as goal progress. We also use it to learn the algorithmic parameters (e.g., a warm start) of our optimizer to accelerate the solution process. We show that our approach can demonstrate highly reliable navigation in cluttered environments on robots with different embodiments. We also demonstrate a substantial reduction in collision rate over recent end-to-end and model-based baselines.
Figures & tables
Fig. 1: Overview of the model-based imitation-learning policy. A frozen depth estimator processes monocular image history. The policy network combines visual features, robot state, and goal to predict cost weights and optimizer parameters. A differentiable optimizer then plans controls using a pre-trained, fixed collision model.
Fig. 2: Training pipeline for the history-aware probabilistic collision model. The control-dependent clearance branch is shown. Monocular history is converted to point clouds in their acquisition-time robot frames and temporally encoded to predict the worst-case clearance distribution. Fixed midpoint quantiles yield collision-constraint residuals h and the empirical risk surrogate remp . Gaussian NLL fits the distribution, while one-sided safe/unsafe supervision shapes the risk surrogate.
Fig. 3: Policy-network training through soft-CEM. The perception encoder is an independent copy of the architecture in Fig. 2 , with separate parameters. Its context cfeat , the current state x0 , and goal g determine the cost weights wh,wn and optimizer hyperparameters ϕ=(umean(0),ρ(0),η) . The Alg. 1 box represents the complete soft-CEM procedure with L refinement iterations. The updated means define Ltask-risk . The final mean’s induced planar trajectory is compared with the expert trajectory through Limit . Gradients of the weighted loss pass through Alg. 1 to train the policy network while the clearance-model parameters remain fixed.
Fig. 4: Panels (a)–(c) show the Grid-Difficult, Grid-Easy, and Corridor environments, respectively, used to evaluate our method and the baselines on the Jackal. Panel (d) shows the more challenging grid environment used for ablations A1 and A2. Panel (e) shows the environment used for the Parrot Bebop experiments.
Fig. 5: Comparison of safety predictions using current-frame observations and image history. Estimated point clouds and ground-truth LiDAR clouds illustrate obstacles near the robot and an unsafe motion primitive. Here t denotes the current frame, while t−2 and t−5 denote previous frames.
Environment
Method
Collision
Linear speed (m/s)
Angular speed (rad/s)
rate (%)
Avg.
Mean peak
Avg.
Mean peak
Corridor
Alg. 1 (Ours)
7
0.77
1.00
0.26
0.82
MonoMPC [ 5 ]
37
0.28
0.86
0.39
0.93
Grid-Easy
Alg. 1 (Ours)
3
0.81
1.00
0.26
0.78
MonoMPC [ 5 ]
23
0.36
0.80
0.35
0.87
Grid-Difficult
Alg. 1 (Ours)
0
0.76
1.00
0.30
0.93
TABLE I: Comparison with MonoMPC across three environments (30 different runs in each environment per method).
Fig. 6: Per-run steering-oscillation metrics for Alg. 1 and MonoMPC, reported separately for the Corridor, Grid-Easy, and Grid-Difficult environments. The top row shows excess heading variation, while the bottom row shows mean absolute angular-velocity command variation. The lower distributions for our method indicate fewer back-and-forth heading fluctuations and less abrupt corrective steering input.
Fig. 7: Comparison of Alg. 1 (Ours) with FlowNav [ 7 ] in the Corridor environment (30 runs per method).
Fig. 8: Ablation of Alg. 1 (30 runs per method). A1 and A2 use fixed initialization with learned and perturbed weights and temperature, respectively.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Supplementary material
Main paper
Sec. B : Collision-risk formulation and analysis
Sec. III-B ; Eqs. ( 6 )–( 8 )
Secs. C-A – C-D : Network architecture
Secs. III-B3 , III-C ; Figs. 2 , 3
Secs. C-E – C-F : Training and computation time
Secs. III-C , IV-A ; Alg. 1
Sec. D-A : MonoMPC trajectories and oscillations
Sec. IV-B ; Table I ; Fig. 6
Sec. D-B : FlowNav trajectories
Sec. IV-C ; Fig. 7
Sec. D-C : CARE comparison and trajectories
Sec. I : Safety augmentation of visual policies
Appendix
TABLE S1: Guide to the supplementary material and corresponding main-paper content.
Component
Layer specification
Point-wise encoder
Five Conv1D layers: 2→64→64→64→128→D
Pooling
Global maximum over the 300 points
Pooled projection
Linear D→256 , LayerNorm, ReLU, dropout, linear 256→D
Frame projection
Linear D→D
Position embedding
Learned array of size 21×D
Temporal attention
One pre-normalized self-attention block; four heads of width D/4
Appendix
TABLE S2: Perception encoder used independently by each network. D=40 for the collision model and D=64 for the policy. Arrows denote successive channel dimensions.
Component
Dimensions and output
State encoder
2→256→64 ; two layers
Goal encoder
2→256→64 ; two layers
Fusion MLP
192→256→256→256 ; three layers
Mean head
256→100 ; entries of umean(0)
Variance head
256→100 ; entries of ρ(0)
Goal-weight head
256→1 ; wh
Appendix
TABLE S3: Policy layers following the independent perception encoder. Each arrow denotes one fully connected layer with a bias.
Fig. S1: Qualitative trajectories in (a) Grid-Difficult, (b) Grid-Easy and (c) Corridor. Within each panel, the left and right columns show Alg. 1 and MonoMPC, respectively. RViz views with the goal highlighted appear above the corresponding third-person views.
Fig. S3: Alg. 1 (left) and FlowNav (right) in the Corridor environment. The top row shows the RViz view, and the bottom row shows the third-person view.
Method
Collision
Linear speed (m/s)
rate (%)
Average
Mean peak
Alg. 1 (Ours)
4
0.80
1.00
CARE [ 8 ]
90
0.43
0.57
Appendix
TABLE S4: Comparison with CARE in the Corridor environment.
Fig. S4: Alg. 1 (left) and CARE applied to NoMaD (right) in the Corridor environment. RViz views appear above the corresponding third-person views.
Fig. S5: Alg. 1 (left), A1 (center), and A2 (right). A1 and A2 use fixed initialization with learned and perturbed weights and temperature, respectively. RViz views are shown above the corresponding third-person views.
Fig. S6: The complete navigation pipeline, including Alg. 1 , deployed on the Parrot Bebop without modification or retraining. The RViz view (top) marks the goal position, and the third-person view (bottom) shows the drone.
Learning robust navigation policies remains a core challenge in robotics. Offline imitation learning suffers from distribution shift and compounding errors at rollout, while reinforcement learning requires reward engineering and learns inefficiently. In this paper, we propose NavOL, an online imitation learning paradigm that interacts with a simulator and updates itself using expert demonstrations gathered online. Built upon a pretrained navigation diffusion policy that maps local observations to future waypoints, NavOL trains in a rollout update loop: during rollout, the policy acts in the simulator and queries a global planner which has privileged access to the global environment for the optimal path segment as ground truth trajectory labels; during update, the policy is trained on the online collected observation trajectory pairs. This online imitation loop removes the need for reward design, improves learning efficiency, and mitigates distribution shift by training on the policy own explored rollouts. Built on IsaacLab with fast, high-fidelity parallel rendering and domain randomization of camera pose and start-goal pairs, our system scales across 50 scenes on 8 RTX 4090 GPUs, collecting over 2,000 new trajectories per hour, each averaging more than 400 steps. We also introduce an indoor visual navigation benchmark with predefined start and goal positions for zero-shot generalization. Extensive evaluations on simulation benchmarks, including the NavDP benchmark and our proposed benchmark, as well as carefully designed real-world experiments, demonstrate the effectiveness of NavOL, showing consistent performance gains in online imitation learning.
Xiaofei Wei, Chun Gu, Li Zhang
School of Data Science, Fudan University · 2Shanghai Innovation Institute.
While visual navigation has advanced through imitation learning from cross-platform demonstrations, fully leveraging such data remains challenging. First, directly learning from image-trajectory pairs entangles navigation behavior with platform-dependent camera geometry. This hinders consistent learning by forcing the policy to implicitly infer camera geometry from visual observations, an inherently ill-posed problem. Second, imitation learning from demonstrated trajectories captures the expert's chosen motion but leaves the intermediate decisions underlying that motion implicit. To address these issues, we propose CanonNav, a visual navigation framework that disentangles navigation behavior from camera geometry and incorporates complementary planning supervision into learning from cross-platform demonstrations. CanonNav introduces camera geometry canonicalization, which transforms visual observations and trajectories into a camera-consistent representation space. Building on this representation, we derive safety and local-progress supervision using pseudo-labels from an offline traversability estimator. Safety supervision penalizes unsafe trajectories, while local-progress supervision guides where the robot should advance. Experiments across diverse camera configurations and environments show that, despite using only RGB at inference, CanonNav consistently outperforms RGB-based baselines and even surpasses RGB-D-based methods in challenging scenarios.
Visual navigation policies have grown to hundreds of millions of parameters trained on billions of frames, with geometry, mapping, and control learned implicitly. We propose a decomposed point-goal navigation system in which operations with known closed-form structure, such as projective geometry, occupancy, and coordinate transforms, are computed analytically and serve as interfaces between three small learned modules: an egress predictor that grounds the episode goal as a local subgoal in the current view, a navigation predictor that estimates a goal-conditioned posterior over where trajectories travel, and an endpoint-pinned residual diffusion generator that samples trajectory shapes from this posterior. Only 0.58M out of 23M parameters are trained, on 44k frames, in under one GPU-hour. Across 6060 point-goal episodes in 60 environments, the system attains competitive success rates with the lowest collision rate among evaluated methods. We further show that under this decomposition, the frozen image encoder can be replaced by a 0.54M MobileNetV2 at a -2.0 SR cost, bringing the full system under 1.2M parameters. It also transfers to no-goal exploration by retraining only the 123k-parameter egress head, and its failure modes under sensor corruption are transparent and analytically correctable. We deploy and evaluate the system zero-shot on a low-cost UGV, running navigation and localization on a Jetson Orin Nano in real-time.
Edward Beng Wai Tan, Siew-Kei Lam
College of Computing and Data Science, Nanyang Technological University, Singapore.