The Midas Touch for Metric Depth: Interpretable Metricization for Generalizable Depth Perception
Authors: Mengtan Zhang, Yu Ma, Shiqi Jiang, Yi Feng, Wei Ye, Xieyuanli Chen, Hanli Wang, Qijun Chen, Rui Fan
Organizations: Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai 201210, P. R. China · College of Electronic & Information Engineering, Tongji University, Shanghai 201804, P. R. China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, P. R. China · College of Electronic and Information Engineering, the School of Computer Science and Technology, and the Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University, Shanghai 201804, P. R. China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai Key Laboratory of Intelligent Autonomous Systems, Urban Mobility Institute, State Key Laboratory of Autonomous Intelligent Unmanned Systems, and Frontiers Science Center for Intelligent Autonomous Systems (Ministry of Education), The Key Laboratory of Road and Traffic Engineering (Ministry of Education), Tongji University, Shanghai 201804, China, as well as with the National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’An, Shaanxi 710049, China
Recent foundation models have drastically improved the generalization of monocular relative depth estimation, yet recovering accurate metric depth from their estimates for depth perception tasks remains challenging due to scale ambiguity and local metric-scale inconsistency. Existing approaches recover metric depth from sparse observations using either explicit global affine transformation algorithms or implicit data-driven depth completion networks. The former cannot accommodate spatially varying relative-to-metric relationships, while the latter are sensitive to the distribution of sparse observations and may not fully exploit generalizable relative depth priors. To simultaneously address these issues, this study extends our previous work MTD and proposes MTDv2, a hybrid metric depth perception framework that combines explicit depth metricization with data-driven metric depth refinement. The explicit depth metricization process performs segment-wise affine transformations to recover spatially varying metric scales while preserving the geometric structure of relative depth, followed by pixel-wise depth refinement that propagates metric observations along paths with minimal accumulated geometric variation. The data-driven depth refinement process subsequently extends the explicit single-path depth propagation to a learnable multi-path formulation while preserving its interpretability. Extensive experiments conducted on 16 public datasets demonstrate the robust and generalizable depth scale recovery capability of MTDv2 across diverse sources of sparse observations and different depth estimation paradigms, as well as its broad applicability to downstream 3D perception tasks. Code will be publicly available at https://mias.group/MTD upon publication.
Figures & tables
Fig. 1: Overview and capabilities of MTDv2. By combining training-free explicit depth metricization with data-driven depth refinement, MTDv2 provides an interpretable and generalizable solution for recovering dense metric depth from sparse observations acquired using camera-LiDAR systems, stereo cameras, and other range sensors. It remains effective under extremely sparse inputs and across different sensing devices.
Fig. 2: Framework overview of MTDv2. Given an RGB image, a relative depth map, and sparse metric observations, the optimization-based depth metricization is first performed. The affine transformation parameters are estimated for the qualified local segments and subsequently propagated to the unqualified segments via sparse graph optimization to generate a coarse metric depth map, which is then refined through single-path depth propagation based on dynamic programming, yielding the explicitly refined metric depth map. In the subsequent data-driven metric depth refinement process, the geometric features extracted from the explicitly refined metric depth and sparse metric observations are fused with textural features extracted from the RGB image and fed into a geometry-guided depth propagation head for learnable multi-path depth propagation, producing the final metric depth map.
Method
nuScenes
DDAD
Make3D
DIODE
ETH3D
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
CFormer [ 15 ] (CVPR’23)
15.672
10.528
9.606
3.328
10.749
5.035
2.297
0.976
2.796
1.940
LRRU [ 12 ] (ICCV’23)
13.660
8.472
9.164
2.738
13.023
5.893
2.795
1.947
3.016
2.337
BP-Net [ 14 ] (CVPR’24)
15.092
10.592
8.903
2.712
12.034
5.353
2.724
1.831
3.365
2.665
DMD 3 C [ 7 ] (CVPR’25)
5.556
3.112
7.766
2.498
12.019
5.575
2.262
1.127
0.935
0.285
PromptDA [ 66 ] (CVPR’25)
9.072
5.325
8.487
2.891
12.392
5.803
2.139
1.287
1.172
0.479
TABLE I: Quantitative comparisons of SoTA depth completion methods. The best results are shown in bold font.
Fig. 3: Qualitative comparisons with SoTA depth completion methods.
Model Paradigm
Method
Configurations
KITTI
NYUv2
ETH3D
ScanNet
DIODE
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Abs Rel ↓
δ1↑
Discriminative Relative Depth Estimation Network
MiDaS [ 3 ]
0.183
0.711
0.095
0.915
0.190
0.884
0.099
0.907
0.266
0.713
MiDaS [ 3 ]
+MTD
0.069
0.929
0.048
0.949
0.055
0.944
0.015
0.991
0.113
0.864
MiDaS [ 3 ]
+MTDv2
0.053
0.938
0.041
0.956
0.047
0.952
0.012
0.993
0.102
0.871
LeReS [ 67 ]
0.149
0.784
0.090
0.916
0.171
0.777
0.091
0.917
0.271
0.766
LeReS [ 67 ]
+MTD
0.035
0.974
0.014
0.994
0.022
0.984
0.013
0.994
0.097
0.885
TABLE II: Quantitative comparison among the global scale recovery strategy, MTD, and the proposed MTDv2. The best results in each comparison group are shown in bold font.
Method
Middlebury
ETH3D
KITTI-12
KITTI-15
BP-2 ↓
BP-1 ↓
D1 ↓
D1 ↓
CREStereo++ [ 73 ]
14.8
4.4
4.7
5.2
DSMNet [ 74 ]
13.8
6.2
6.2
6.5
Mask-CFNet [ 75 ]
13.7
5.7
4.8
5.8
HVT-RAFT [ 76 ]
10.4
3.0
3.7
5.2
RAFT-Stereo [ 77 ]
12.6
3.3
4.7
5.5
TABLE III: Quantitative comparison with SoTA stereo matching networks for zero-shot disparity estimation. The best results are shown in bold font.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
RMSE ↓
3DGS [ 55 ]
15.64
0.557
0.418
3.857
3DGS + ZoeDepth
18.96
0.573
0.324
2.163
3DGS + G2-MonoDepth
19.36
0.641
0.273
1.904
3DGS + OMNI-DC
20.38
0.660
0.229
0.838
3DGS + MTDv2
20.42
0.689
0.213
0.812
TABLE IV: Quantitative comparison of novel view synthesis methods on the ETH3D dataset. The best results are shown in bold font.
Method
Seq 00
Seq 02
Seq 05
Seq 06
ATE ↓
ATE ↓
ATE ↓
ATE ↓
DROID-SLAM (RGB) [ 64 ]
66.562
84.828
41.973
72.402
DROID-SLAM (RGB-D)
63.295
77.610
31.641
104.552
DROID-SLAM (RGB-D) + MTDv2
24.985
26.050
12.513
13.504
Method
Seq 07
Seq 08
Seq 09
Seq 10
ATE ↓
ATE ↓
ATE ↓
ATE ↓
TABLE V: Quantitative comparison of visual odometry methods on the KITTI Odometry dataset. The best results are shown in bold font.
Fig. 4: Qualitative comparisons of (a) rendered RGB images and (b) depth maps on test views obtained by the original 3DGS framework and by that incorporating different depth perception approaches.
Fig. 5: Qualitative comparisons of the visual odometry results achieved using RGB images only, RGB images with the original LiDAR measurements, and RGB images with dense metric depth recovered from the original sparse LiDAR measurements.
Configurations
KITTI
VOID
RMSE ↓
MAE ↓
RMSE ↓
MAE ↓
+ Segment-Wise Affine Transformation
(a0)
w/ LSF (inverse depth)
6.782
1.794
0.614
0.238
(a1)
w/ LSF
7.013
1.802
0.791
0.307
(a2)
w/ Median
10.891
2.169
0.898
0.358
(a3)
w/ Quantile matching
9.178
2.114
0.821
0.346
TABLE VI: Ablation studies on the KITTI and VOID datasets to validate the effectiveness of the proposed technical contributions in outdoor and indoor scenarios, respectively.
Fig. 6: Quantitative evaluation of MTDv2’s sensitivity to varying sampling densities of metric observations on (a) the indoor subset of the ETH3D dataset and (b) the outdoor subset of the DIODE dataset, compared with MTD and global affine transformation using different foundation models.
Fig. 7: Inference time breakdown and a representative failure case of the proposed method under challenging conditions.
Metric depth estimation predicts scene distances in physical units from images, supporting robotics, mapping, augmented reality, and 3D reconstruction.
Recent advances have markedly improved the cross-scene generalization of relative depth estimation, yet its practical applicability remains limited by the absence of metric scale, local inconsistencies, and low computational efficiency. To address these issues, we present Midas Touch for Depth (MTD), a mathematically interpretable approach that converts relative depth into metric depth using only extremely sparse 3D data. To eliminate local scale inconsistencies, it applies a segment-wise recovery strategy via sparse graph optimization, followed by a pixel-wise refinement strategy using a discontinuity-aware geodesic cost. MTD exhibits strong generalization and achieves substantial accuracy improvements over previous depth completion and depth estimation methods. Moreover, its lightweight, plug-and-play design facilitates deployment and integration on diverse downstream 3D tasks. Project page is available at https://mias.group/MTD.
Yu Ma, Zizhan Guo, Zuyi Xiong +5
College of Electronic and Information Engineering, Tongji University · 2Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · 3National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University
Scaling has powered recent advances in vision foundation models, yet extending this paradigm to metric depth estimation remains challenging due to heterogeneous sensor noise, camera-dependent biases, and metric ambiguity in noisy cross-source 3D data. We introduce Metric Anything, a simple and scalable pretraining framework that learns metric depth from noisy, diverse 3D sources without manually engineered prompts, camera-specific modeling, or task-specific architectures. Central to our approach is the Sparse Metric Prompt, created by randomly masking depth maps, which serves as a universal interface that decouples spatial reasoning from sensor and camera biases. Using about 20M image-depth pairs spanning reconstructed, captured, and rendered 3D data across 10000 camera models, we demonstrate-for the first time-a clear scaling trend in the metric depth track. The pretrained model excels at prompt-driven tasks such as depth completion, super-resolution and Radar-camera fusion, while its distilled prompt-free student achieves state-of-the-art results on monocular depth estimation, camera intrinsics recovery, single/multi-view metric 3D reconstruction, and VLA planning. We also show that using pretrained ViT of Metric Anything as a visual encoder significantly boosts Multimodal Large Language Model capabilities in spatial intelligence. These results show that metric depth estimation can benefit from the same scaling laws that drive modern foundation models, establishing a new path toward scalable and efficient real-world metric perception. We open-source MetricAnything at http://metric-anything.github.io/metric-anything-io/ to support community research.
Monocular depth estimation (MDE) typically produces depth estimations that are defined up to an unknown scale or shift. When only sparse metric anchors are available, recovering accurate metric depth becomes challenging yet necessary for practical applications. We address this problem by formulating metric depth recovery as image-adaptive scale field modeling. Instead of directly correcting the depth, we reformulate the correction as a low-dimensional linear combination of image-adaptive basis maps. These maps are derived from semantic and geometric cues encoded in the MDE estimations and intermediate representations. The weights of basis maps are efficiently determined from sparse metric anchors via a least-squares problem. This formulation yields improved metric depth accuracy, strong robustness under extreme anchor sparsity, and an interpretable decomposition of spatial scale variations. Extensive experiments across multiple datasets and representative MDE models demonstrate the effectiveness and general applicability of our approach.