The Midas Touch for Metric Depth: Interpretable Metricization for Generalizable Depth Perception
Authors: Mengtan Zhang, Yu Ma, Shiqi Jiang, Yi Feng, Wei Ye, Xieyuanli Chen, Hanli Wang, Qijun Chen, Rui Fan
Organizations: Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai 201210, P. R. China · College of Electronic & Information Engineering, Tongji University, Shanghai 201804, P. R. China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, P. R. China · College of Electronic and Information Engineering, the School of Computer Science and Technology, and the Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University, Shanghai 201804, P. R. China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai Key Laboratory of Intelligent Autonomous Systems, Urban Mobility Institute, State Key Laboratory of Autonomous Intelligent Unmanned Systems, and Frontiers Science Center for Intelligent Autonomous Systems (Ministry of Education), The Key Laboratory of Road and Traffic Engineering (Ministry of Education), Tongji University, Shanghai 201804, China, as well as with the National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’An, Shaanxi 710049, China
Recent foundation models have drastically improved the generalization of monocular relative depth estimation, yet recovering accurate metric depth from their estimates for depth perception tasks remains challenging due to scale ambiguity and local metric-scale inconsistency. Existing approaches recover metric depth from sparse observations using either explicit global affine transformation algorithms or implicit data-driven depth completion networks. The former cannot accommodate spatially varying relative-to-metric relationships, while the latter are sensitive to the distribution of sparse observations and may not fully exploit generalizable relative depth priors. To simultaneously address these issues, this study extends our previous work MTD and proposes MTDv2, a hybrid metric depth perception framework that combines explicit depth metricization with data-driven metric depth refinement. The explicit depth metricization process performs segment-wise affine transformations to recover spatially varying metric scales while preserving the geometric structure of relative depth, followed by pixel-wise depth refinement that propagates metric observations along paths with minimal accumulated geometric variation. The data-driven depth refinement process subsequently extends the explicit single-path depth propagation to a learnable multi-path formulation while preserving its interpretability. Extensive experiments conducted on 16 public datasets demonstrate the robust and generalizable depth scale recovery capability of MTDv2 across diverse sources of sparse observations and different depth estimation paradigms, as well as its broad applicability to downstream 3D perception tasks. Code will be publicly available at https://mias.group/MTD upon publication.
Figures & tables
Fig. 1: Overview and capabilities of MTDv2. By combining training-free explicit depth metricization with data-driven depth refinement, MTDv2 provides an interpretable and generalizable solution for recovering dense metric depth from sparse observations acquired using camera-LiDAR systems, stereo cameras, and other range sensors. It remains effective under extremely sparse inputs and across different sensing devices.
Fig. 2: Framework overview of MTDv2. Given an RGB image, a relative depth map, and sparse metric observations, the optimization-based depth metricization is first performed. The affine transformation parameters are estimated for the qualified local segments and subsequently propagated to the unqualified segments via sparse graph optimization to generate a coarse metric depth map, which is then refined through single-path depth propagation based on dynamic programming, yielding the explicitly refined metric depth map. In the subsequent data-driven metric depth refinement process, the geometric features extracted from the explicitly refined metric depth and sparse metric observations are fused with textural features extracted from the RGB image and fed into a geometry-guided depth propagation head for learnable multi-path depth propagation, producing the final metric depth map.
Method
nuScenes
DDAD
Make3D
DIODE
ETH3D
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
CFormer [ 15 ] (CVPR’23)
15.672
10.528
9.606
3.328
10.749
5.035
2.297
0.976
2.796
1.940
LRRU [ 12 ] (ICCV’23)
13.660
8.472
9.164
2.738
13.023
5.893
2.795
1.947
3.016
2.337
BP-Net [ 14 ] (CVPR’24)
15.092
10.592
8.903
2.712
12.034
5.353
2.724
1.831
3.365
2.665
DMD 3 C [ 7 ] (CVPR’25)
5.556
3.112
7.766
2.498
12.019
5.575
2.262
1.127
0.935
0.285
PromptDA [ 66 ] (CVPR’25)
9.072
5.325
8.487
2.891
12.392
5.803
2.139
1.287
1.172
0.479
TABLE I: Quantitative comparisons of SoTA depth completion methods. The best results are shown in bold font.
Fig. 3: Qualitative comparisons with SoTA depth completion methods.
Model Paradigm
Method
Configurations
KITTI
NYUv2
ETH3D
ScanNet
DIODE
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Discriminative Relative Depth Estimation Network
MiDaS [ 3 ]
0.183
0.711
0.095
0.915
0.190
0.884
0.099
0.907
0.266
0.713
MiDaS [ 3 ]
+MTD
0.069
0.929
0.048
0.949
0.055
0.944
0.015
0.991
0.113
0.864
MiDaS [ 3 ]
+MTDv2
0.053
0.938
0.041
0.956
0.047
0.952
0.012
0.993
0.102
0.871
LeReS [ 67 ]
0.149
0.784
0.090
0.916
0.171
0.777
0.091
0.917
0.271
0.766
LeReS [ 67 ]
+MTD
0.035
0.974
0.014
0.994
0.022
0.984
0.013
0.994
0.097
0.885
TABLE II: Quantitative comparison among the global scale recovery strategy, MTD, and the proposed MTDv2. The best results in each comparison group are shown in bold font.
Method
Middlebury
ETH3D
KITTI-12
KITTI-15
BP-2 ↓
BP-1 ↓
D1 ↓
D1 ↓
CREStereo++ [ 73 ]
14.8
4.4
4.7
5.2
DSMNet [ 74 ]
13.8
6.2
6.2
6.5
Mask-CFNet [ 75 ]
13.7
5.7
4.8
5.8
HVT-RAFT [ 76 ]
10.4
3.0
3.7
5.2
RAFT-Stereo [ 77 ]
12.6
3.3
4.7
5.5
TABLE III: Quantitative comparison with SoTA stereo matching networks for zero-shot disparity estimation. The best results are shown in bold font.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
3DGS [ 55 ]
15.64
0.557
0.418
3.857
3DGS + ZoeDepth
18.96
0.573
0.324
2.163
3DGS + G2-MonoDepth
19.36
0.641
0.273
1.904
3DGS + OMNI-DC
20.38
0.660
0.229
0.838
3DGS + MTDv2
20.42
0.689
0.213
0.812
TABLE IV: Quantitative comparison of novel view synthesis methods on the ETH3D dataset. The best results are shown in bold font.
Method
Seq 00
Seq 02
Seq 05
Seq 06
ATE ↓
ATE ↓
ATE ↓
ATE ↓
DROID-SLAM (RGB) [ 64 ]
66.562
84.828
41.973
72.402
DROID-SLAM (RGB-D)
63.295
77.610
31.641
104.552
DROID-SLAM (RGB-D) + MTDv2
24.985
26.050
12.513
13.504
Method
Seq 07
Seq 08
Seq 09
Seq 10
ATE ↓
ATE ↓
ATE ↓
ATE ↓
TABLE V: Quantitative comparison of visual odometry methods on the KITTI Odometry dataset. The best results are shown in bold font.
Fig. 4: Qualitative comparisons of (a) rendered RGB images and (b) depth maps on test views obtained by the original 3DGS framework and by that incorporating different depth perception approaches.
Fig. 5: Qualitative comparisons of the visual odometry results achieved using RGB images only, RGB images with the original LiDAR measurements, and RGB images with dense metric depth recovered from the original sparse LiDAR measurements.
Configurations
KITTI
VOID
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
+ Segment-Wise Affine Transformation
(a0)
w/ LSF (inverse depth)
6.782
1.794
0.614
0.238
(a1)
w/ LSF
7.013
1.802
0.791
0.307
(a2)
w/ Median
10.891
2.169
0.898
0.358
(a3)
w/ Quantile matching
9.178
2.114
0.821
0.346
TABLE VI: Ablation studies on the KITTI and VOID datasets to validate the effectiveness of the proposed technical contributions in outdoor and indoor scenarios, respectively.
Fig. 6: Quantitative evaluation of MTDv2’s sensitivity to varying sampling densities of metric observations on (a) the indoor subset of the ETH3D dataset and (b) the outdoor subset of the DIODE dataset, compared with MTD and global affine transformation using different foundation models.
Fig. 7: Inference time breakdown and a representative failure case of the proposed method under challenging conditions.
Metric depth estimation predicts scene distances in physical units from images, supporting robotics, mapping, augmented reality, and 3D reconstruction.
College of Electronic and Information Engineering, Tongji University · 2Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · 3National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University