Organizations: Dept. of Electrical and Computer Engineering, Univ. of California, Riverside, 900 University Avenue, Riverside, CA 92521, USA · Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91011, USA
The geometric understanding of a scene is a crucial component for safe and efficient robotic operations in novel and unstructured environments. The recent emergence of foundation models for Monocular Depth Estimation (MDE) has provided an opportunity to enrich robotic perception capabilities with dense long-range depth predictions, replacing expensive onboard hardware (e.g., LiDARs and stereo cameras). However, these offline models are primarily trained on affine-invariant depth representations, and their generalization to novel scenes remains unclear. In this work we present EM-ViT, a monocular depth estimation model that learns to adapt online relative depth inputs from a foundation model prior to the metric scale with observations collected during robotic operations. Motivated by the stereo camera parameterization, our method decomposes depth prediction via dense disparity refinement to recover the scene's relative structure and an effective baseline scalar prediction to determine the missing metric scale. The learning process leverages sparse LiDAR points and, optionally, the robot's Ego-Motion (EM) to rapidly train a lightweight Vision Transformer (ViT), while an internal gradient matching loss preserves the fine-grained semantic structure of the input disparity in real-time. We demonstrate the efficacy of our method on publicly available sequences, such as KITTI, and show that our online learning pipeline can adapt to novel scenes faster and with higher accuracy than existing MDE foundation models. The code and supplementary materials will be publicly available at https://emvit-mde.github.io.
Figures & tables
Fig. 1 : Real-time and onboard MDE metric adaptation. Sparse image frames and corresponding point cloud views collected during exploration are used to refine an affine-invariant relative depth map into metric and adapt to the new environment.
Fig. 2 : Self-supervised virtual stereo system for online metric adaptation in MDE. Based on an affine-invariant inverse depth input da and camera’s focal length f , our Ego Motion (EM)-ViT framework decodes depth reconstruction into dense disparity refinement d^ and an effective baseline prediction B^ , achieving scale-to-structure disentanglement. The integrated disparity refinement module (e.g., dRefNet ) uses a single image It , with optionally previously captured frames Iref and their corresponding poses p , to metrically fine-tune generated depth D^t , while pose corrections can be provided in real-time via the odometry refinement module (e.g., odomNet ). Our lightweight EM-ViT framework is self-supervised by sparse LiDAR annotations, photometric reconstruction, and gradient matching loss, to provide real-time, accurate, and fine-grained monocular depth estimation.
Fig. 3 : Virtual stereo and metric conversion ambiguity. Due to different baselines B and B′ , the points P and P′ appear to have the same disparity xleft−xright , regardless of their actual distance difference.
Fig. 4 : Online data collection and training pipeline. By selecting a training sequence duration dt=ti−ti−1 and a fixed sampling rate, we collect and synchronize data to train our model at times ti (black and gray circles). By having the trained model at offset times ti,off (white circles), namely EM-ViT i , we evaluate its MDE performance in the next scene sequence (ti+1,ti+1+dt] until the next training initiation at time ti+2 .
Fig. 5 : Qualitative evaluation of depth prediction during online learning in various datasets. The first column shows the selected input images, with the rest showcasing the pseudo-colored metric depth results of the comparing methods. Our method attains depth maps with higher metric accuracy and finer structural details, with important priors from the relative depth backbone being preserved (as the sky, or appeared objects), while also requiring less onboard training time. By using the known intrinsics, we showcase the resulted point maps of selected depth scenes along with the ground truth LiDAR red points.
Dataset
Method
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Total Training Time (s)
Garage Environment
DAV2 ViT-S Metric VKITTI2 (Unadapted) [ 21 ]
0.022
2.05
16.43
15.30
13.10
-
DAV2 ViT-S Metric HyperSim (Unadapted) [ 21 ]
0.659
0.19
5.10
2.38
1.01
-
Marsal R. etal. [ 28 ]
0.849
0.13
4.51
1.89
0.61
-
FusionDepth [ 52 ]
0.854
0.12
3.81
1.68
0.55
119.83
Online DAV2 ViT-S Metric [ 21 ]
0.859
0.13
3.37
1.58
0.63
100.38
EM-ViT (Ours)
0.910
0.10
3.29
1.32
0.43
82.37
TABLE I : Online training evaluation. Metric scores of our MDE pipeline compared with baseline models, evaluated on a NVIDIA AGX Orin 64GB. We include also scores from the metrically fine-tuned DAV2 ViT-S models, as provided [ 21 ] , for the corresponding indoor (HyperSim) and outdoor (VKITTI2) settings. The bold and underlined values denote the best and second-best scores across all methods, respectively.
Fig. 6 : Moments during online training. We demonstrate our model’s online MDE adaptation in the unexplored garage environment, by presenting the total training loss (top left), the δ1 score (top right), and depth prediction instances on unseen images (bottom).
Fig. 7 : Ground-truth density and depth map fidelity. Our method preserves high-fidelity structural detail during disparity reconstruction via the proposed gradient matching supervision scheme LGM(d^,da) with local corrections, even in cases with less LiDAR annotations (i.e. Garage and CitrusFarm).
Dataset
LiDAR Availability
odomNet
δ1↑
RMSE ↓
L1 80 (m) ↓
Garage Environment
All beams
✓
0.920
3.28
1.28
40%
✓
0.902
3.47
1.40
20%
✓
0.900
3.47
1.40
10%
✓
0.900
3.46
1.40
3%
✓
0.898
3.52
1.43
20 beams
✓
0.839
3.95
1.73
TABLE II : Limitation of LiDAR ground truth. Evaluation across different availability percentages of LiDAR points during self-supervision of EM-ViT. Inclusion of extreme cases using 20 or a single LiDAR beam (i.e. rangefinder case) for metric supervision, with and without the use of odomNet .
Dataset
Model
Sampling Rate
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Total Training Time (s)
Garage Environment
Online DAV2 ViT-S Metric [ 21 ]
0.75Hz
0.859
0.13
3.37
1.58
0.63
100.38
EM-ViT (Ours)
0.4Hz
0.895
0.11
3.53
1.43
0.46
94.29
0.3Hz
0.889
0.11
3.57
1.46
0.47
71.74
0.2Hz
0.866
0.12
3.59
1.52
0.54
50.91
KITTI [ 33 ] Scenes
Online DAV2 ViT-S Metric [ 21 ]
0.75Hz
0.666
0.24
4.20
2.40
1.43
67.75
EM-ViT (Ours)
0.4Hz
0.787
0.16
3.43
1.80
0.88
58.51
TABLE III : Examination of lower image sampling rates. Comparison of the superior baseline method with EM-ViT, which utilizes photometric loss LPh , pose refinement during online learning, and lower {It,Pt} sampling rates. The bold and underlined values denote the best and second-best scores across all methods, respectively.
Fig. 8 : Disparity map comparison through baseline value distribution on unseen images in a stereo dataset. Given the true disparity maps and the baseline value, our method ( ) generates disparity maps that distribute the per-pixel baseline values closer to the true value, compared with the input disparity da ( ).
Fig. 9 : Effective baseline value distribution on a single unseen image in a stereo dataset. Our dRefNet model generates a disparity map d^ that minimizes the variance of per-pixel baseline values and provides a predicted (effective) baseline value B^ close to the true baseline B .
Dataset
Method
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Garage Environment
DAV2 ViT-S Metric - DPT Head
0.927
0.09
2.59
1.14
0.48
DAV2 ViT-S Metric - Full Model
0.952
0.08
2.34
0.98
0.40
EM-ViT (Ours)
0.955
0.07
2.66
0.94
0.33
TABLE IV : Offline training evaluation. Metric scores between our MDE pipeline and the DAV2 models that were trained and tested on the full dataset. We train both the pretrained and head part of the DAV2 ViT-S model as proposed for metric fine-tuning. The bold and underlined values denote the best and second-best scores across all methods, respectively.
Dataset
Method
δ1↑
RMSE ↓
L1 80 (m) ↓
Garage Environment
L2 -Solver
0.736
4.12
2.45
EM-ViT (Ours)
0.822
3.66
2.05
TABLE V : Necessity of online disparity refinement. Comparison of EM-ViT with an L2 -solver for affine-invariant disparity map to metric fine-tuning.
Fig. 10 : Dense disparity reconstruction and correction. Example of MDE backbone inability to form correct disparity on specific areas of the scene with on-the-go structural correction of our dense disparity reconstruction.
Dataset
Method
δ1↑
AbsRel ↓
RMSE ↓
L1 80 (m) ↓
L1 10 (m) ↓
Perseverance Navcam [ 12 ]
DAV2 ViT-S Metric VKITTI2 (Unadapted)
0.213
0.56
4.62
3.14
1.95
DAV2 ViT-S Metric HyperSim (Unadapted)
0.086
0.48
5.42
3.61
2.08
DAV2 ViT-S Metric (Online Adapted)
0.931
0.09
1.19
0.62
0.35
EM-ViT (Ours)
0.975
0.06
0.97
0.48
0.25
TABLE VI : Online training evaluation on Perseverance Navcam dataset. Metric scores of our MDE pipeline compared with baseline models, evaluated on a NVIDIA AGX Orin 64GB. The bold values denote the best scores across all methods.
Fig. 11 : Qualitative and quantitative evaluation of depth prediction during online learning on the Perseverance Navcam [ 12 ] dataset. The first column shows the selected input images, with the rest showcasing the pseudo-colored metric depth results of the comparing methods. With FoundationStereo [ 70 ] as a pseudo ground-truth, we present Difference images illustrating the absolute distance error per annotated pixel for both methods.
Fig. 12 : Qualitative evaluation of depth prediction of our method in different Martian scenes. By using a single image our method is able to generate depth maps of the observed area without being affected by ambient conditions. Specifically, in cases of low solar phase angle and poor area texture, the applied Flight Stereo [ 11 ] method shows depth discontinuities, in contrast to the EM-ViT. We showcase our method’s generated point clouds for each Martian scene in different view points.
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. · Yichang Testing Technology Research Institute, Yichang, China. · School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China.