Organizations: Dept. of Electrical and Computer Engineering, Univ. of California, Riverside, 900 University Avenue, Riverside, CA 92521, USA · Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91011, USA
The geometric understanding of a scene is a crucial component for safe and efficient robotic operations in novel and unstructured environments. The recent emergence of foundation models for Monocular Depth Estimation (MDE) has provided an opportunity to enrich robotic perception capabilities with dense long-range depth predictions, replacing expensive onboard hardware (e.g., LiDARs and stereo cameras). However, these offline models are primarily trained on affine-invariant depth representations, and their generalization to novel scenes remains unclear. In this work we present EM-ViT, a monocular depth estimation model that learns to adapt online relative depth inputs from a foundation model prior to the metric scale with observations collected during robotic operations. Motivated by the stereo camera parameterization, our method decomposes depth prediction via dense disparity refinement to recover the scene's relative structure and an effective baseline scalar prediction to determine the missing metric scale. The learning process leverages sparse LiDAR points and, optionally, the robot's Ego-Motion (EM) to rapidly train a lightweight Vision Transformer (ViT), while an internal gradient matching loss preserves the fine-grained semantic structure of the input disparity in real-time. We demonstrate the efficacy of our method on publicly available sequences, such as KITTI, and show that our online learning pipeline can adapt to novel scenes faster and with higher accuracy than existing MDE foundation models. The code and supplementary materials will be publicly available at https://emvit-mde.github.io.
Figures & tables
Fig. 1 : Real-time and onboard MDE metric adaptation. Sparse image frames and corresponding point cloud views collected during exploration are used to refine an affine-invariant relative depth map into metric and adapt to the new environment.
Fig. 2 : Self-supervised virtual stereo system for online metric adaptation in MDE. Based on an affine-invariant inverse depth input da and camera’s focal length f , our Ego Motion (EM)-ViT framework decodes depth reconstruction into dense disparity refinement d^ and an effective baseline prediction B^ , achieving scale-to-structure disentanglement. The integrated disparity refinement module (e.g., dRefNet ) uses a single image It , with optionally previously captured frames Iref and their corresponding poses p , to metrically fine-tune generated depth D^t , while pose corrections can be provided in real-time via the odometry refinement module (e.g., odomNet ). Our lightweight EM-ViT framework is self-supervised by sparse LiDAR annotations, photometric reconstruction, and gradient matching loss, to provide real-time, accurate, and fine-grained monocular depth estimation.
Fig. 3 : Virtual stereo and metric conversion ambiguity. Due to different baselines B and B′ , the points P and P′ appear to have the same disparity xleft−xright , regardless of their actual distance difference.
Fig. 4 : Online data collection and training pipeline. By selecting a training sequence duration dt=ti−ti−1 and a fixed sampling rate, we collect and synchronize data to train our model at times ti (black and gray circles). By having the trained model at offset times ti,off (white circles), namely EM-ViT i , we evaluate its MDE performance in the next scene sequence (ti+1,ti+1+dt] until the next training initiation at time ti+2 .
Fig. 5 : Qualitative evaluation of depth prediction during online learning in various datasets. The first column shows the selected input images, with the rest showcasing the pseudo-colored metric depth results of the comparing methods. Our method attains depth maps with higher metric accuracy and finer structural details, with important priors from the relative depth backbone being preserved (as the sky, or appeared objects), while also requiring less onboard training time. By using the known intrinsics, we showcase the resulted point maps of selected depth scenes along with the ground truth LiDAR red points.
Dataset
Method
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Total Training Time (s)
Garage Environment
DAV2 ViT-S Metric VKITTI2 (Unadapted) [ 21 ]
0.022
2.05
16.43
15.30
13.10
-
DAV2 ViT-S Metric HyperSim (Unadapted) [ 21 ]
0.659
0.19
5.10
2.38
1.01
-
Marsal R. etal. [ 28 ]
0.849
0.13
4.51
1.89
0.61
-
FusionDepth [ 52 ]
0.854
0.12
3.81
1.68
0.55
119.83
Online DAV2 ViT-S Metric [ 21 ]
0.859
0.13
3.37
1.58
0.63
100.38
EM-ViT (Ours)
0.910
0.10
3.29
1.32
0.43
82.37
TABLE I : Online training evaluation. Metric scores of our MDE pipeline compared with baseline models, evaluated on a NVIDIA AGX Orin 64GB. We include also scores from the metrically fine-tuned DAV2 ViT-S models, as provided [ 21 ] , for the corresponding indoor (HyperSim) and outdoor (VKITTI2) settings. The bold and underlined values denote the best and second-best scores across all methods, respectively.
Fig. 6 : Moments during online training. We demonstrate our model’s online MDE adaptation in the unexplored garage environment, by presenting the total training loss (top left), the δ1 score (top right), and depth prediction instances on unseen images (bottom).
Fig. 7 : Ground-truth density and depth map fidelity. Our method preserves high-fidelity structural detail during disparity reconstruction via the proposed gradient matching supervision scheme LGM(d^,da) with local corrections, even in cases with less LiDAR annotations (i.e. Garage and CitrusFarm).
Dataset
LiDAR Availability
odomNet
δ1↑
RMSE ↓
L1 80 (m) ↓
Garage Environment
All beams
✓
0.920
3.28
1.28
40%
✓
0.902
3.47
1.40
20%
✓
0.900
3.47
1.40
10%
✓
0.900
3.46
1.40
3%
✓
0.898
3.52
1.43
20 beams
✓
0.839
3.95
1.73
TABLE II : Limitation of LiDAR ground truth. Evaluation across different availability percentages of LiDAR points during self-supervision of EM-ViT. Inclusion of extreme cases using 20 or a single LiDAR beam (i.e. rangefinder case) for metric supervision, with and without the use of odomNet .
Dataset
Model
Sampling Rate
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Total Training Time (s)
Garage Environment
Online DAV2 ViT-S Metric [ 21 ]
0.75Hz
0.859
0.13
3.37
1.58
0.63
100.38
EM-ViT (Ours)
0.4Hz
0.895
0.11
3.53
1.43
0.46
94.29
0.3Hz
0.889
0.11
3.57
1.46
0.47
71.74
0.2Hz
0.866
0.12
3.59
1.52
0.54
50.91
KITTI [ 33 ] Scenes
Online DAV2 ViT-S Metric [ 21 ]
0.75Hz
0.666
0.24
4.20
2.40
1.43
67.75
EM-ViT (Ours)
0.4Hz
0.787
0.16
3.43
1.80
0.88
58.51
TABLE III : Examination of lower image sampling rates. Comparison of the superior baseline method with EM-ViT, which utilizes photometric loss LPh , pose refinement during online learning, and lower {It,Pt} sampling rates. The bold and underlined values denote the best and second-best scores across all methods, respectively.
Fig. 8 : Disparity map comparison through baseline value distribution on unseen images in a stereo dataset. Given the true disparity maps and the baseline value, our method ( ) generates disparity maps that distribute the per-pixel baseline values closer to the true value, compared with the input disparity da ( ).
Fig. 9 : Effective baseline value distribution on a single unseen image in a stereo dataset. Our dRefNet model generates a disparity map d^ that minimizes the variance of per-pixel baseline values and provides a predicted (effective) baseline value B^ close to the true baseline B .
Dataset
Method
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Garage Environment
DAV2 ViT-S Metric - DPT Head
0.927
0.09
2.59
1.14
0.48
DAV2 ViT-S Metric - Full Model
0.952
0.08
2.34
0.98
0.40
EM-ViT (Ours)
0.955
0.07
2.66
0.94
0.33
TABLE IV : Offline training evaluation. Metric scores between our MDE pipeline and the DAV2 models that were trained and tested on the full dataset. We train both the pretrained and head part of the DAV2 ViT-S model as proposed for metric fine-tuning. The bold and underlined values denote the best and second-best scores across all methods, respectively.
Dataset
Method
δ1↑
RMSE ↓
L1 80 (m) ↓
Garage Environment
L2 -Solver
0.736
4.12
2.45
EM-ViT (Ours)
0.822
3.66
2.05
TABLE V : Necessity of online disparity refinement. Comparison of EM-ViT with an L2 -solver for affine-invariant disparity map to metric fine-tuning.
Fig. 10 : Dense disparity reconstruction and correction. Example of MDE backbone inability to form correct disparity on specific areas of the scene with on-the-go structural correction of our dense disparity reconstruction.
Dataset
Method
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Perseverance Navcam [ 12 ]
DAV2 ViT-S Metric VKITTI2 (Unadapted)
0.213
0.56
4.62
3.14
1.95
DAV2 ViT-S Metric HyperSim (Unadapted)
0.086
0.48
5.42
3.61
2.08
DAV2 ViT-S Metric (Online Adapted)
0.931
0.09
1.19
0.62
0.35
EM-ViT (Ours)
0.975
0.06
0.97
0.48
0.25
TABLE VI : Online training evaluation on Perseverance Navcam dataset. Metric scores of our MDE pipeline compared with baseline models, evaluated on a NVIDIA AGX Orin 64GB. The bold values denote the best scores across all methods.
Fig. 11 : Qualitative and quantitative evaluation of depth prediction during online learning on the Perseverance Navcam [ 12 ] dataset. The first column shows the selected input images, with the rest showcasing the pseudo-colored metric depth results of the comparing methods. With FoundationStereo [ 70 ] as a pseudo ground-truth, we present Difference images illustrating the absolute distance error per annotated pixel for both methods.
Fig. 12 : Qualitative evaluation of depth prediction of our method in different Martian scenes. By using a single image our method is able to generate depth maps of the observed area without being affected by ambient conditions. Specifically, in cases of low solar phase angle and poor area texture, the applied Flight Stereo [ 11 ] method shows depth discontinuities, in contrast to the EM-ViT. We showcase our method’s generated point clouds for each Martian scene in different view points.
Monocular depth estimation (MDE) has been widely adopted in the perception systems of autonomous vehicles and mobile robots. However, existing approaches often struggle to maintain temporal consistency in depth estimation across consecutive frames. This inconsistency not only causes jitter but can also lead to estimation failures when the depth range changes abruptly. To address these challenges, this paper proposes a consistency-aware monocular depth estimation framework that leverages wheel odometry from a mobile robot to achieve stable and coherent depth predictions over time. Specifically, we estimate camera pose and sparse depth from triangulation using optical flow between consecutive frames. The sparse depth estimates are used to update a recursive Bayesian estimate of the metric scale, which is then applied to rescale the relative depth predicted by a pre-trained depth estimation foundation model. The proposed method is evaluated on the KITTI, TartanAir, MS2, and our own dataset, demonstrating robust and accurate depth estimation performance.
Foundation-model-based monocular depth estimation offers a viable alternative to active sensors for robot perception, yet its computational cost often prohibits deployment on edge platforms. Existing methods perform independent per-frame inference, wasting the substantial computational redundancy between adjacent viewpoints in continuous robot operation. This paper presents AsyncMDE, an asynchronous depth perception system consisting of a frozen foundation model and a lightweight fast path that amortizes the foundation model's computational cost over time. The foundation model periodically produces high-quality spatial features in the background, while the lightweight fast path runs asynchronously in the foreground, fusing cached memory with current observations through complementary fusion, outputting depth estimates, and autoregressively updating memory. This enables cross-frame feature reuse with bounded accuracy degradation. With 3.83M trainable fast-path parameters and a 97.5M frozen slow path, AsyncMDE's fast path operates at 237 FPS on an RTX 4090, recovering 77% of the accuracy gap to the foundation model. Across indoor static, dynamic, and synthetic extreme-motion benchmarks, AsyncMDE degrades predictably and reaches 161 FPS fast-path inference on a TensorRT-optimized Jetson AGX Orin, supporting real-time edge deployment.
Lianjie Ma, Yuquan Li, Bingzheng Jiang +3
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. · Yichang Testing Technology Research Institute, Yichang, China. · School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China.
Traditional monocular Visual-Inertial Odometry (VIO) systems struggle in low-texture environments where sparse visual features are insufficient for accurate pose estimation. To address this, dense Monocular Depth Estimation (MDE) has been widely explored as a complementary information source. While recent Vision Transformer (ViT) based complex foundational models offer dense, geometrically consistent depth, their computational demands typically preclude them from real-time edge deployment. Our work bridges this gap by integrating learned depth priors directly into the VINS-Mono optimization backend. We propose a novel framework that enforces affine-invariant depth consistency and pairwise ordinal constraints, explicitly filtering unstable artifacts via variance-based gating. This approach strictly adheres to the computational limits of edge devices while robustly recovering metric scale. Extensive experiments on the TartanGround and M3ED datasets demonstrate that our method prevents divergence in challenging scenarios and delivers significant accuracy gains, reducing Absolute Trajectory Error (ATE) by up to 28.3%. Code will be made available.
Arda Alniak, Sinan Kalkan, Mustafa Mert Ankarali +2
Dept. of Electrical & Electronics Engineering · Center for Image Analysis (OGAM), METU, Ankara, Türkiye · Dept. of Computer Engineering