Organizations: School of Software, Shandong University, Jinan 250100, China · Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China · School of Control Science and Engineering, Shandong University, Jinan, China · Shandong Institute of Information Technology Industry Development, Jinan, China · Research Institute of Interdisciplinary Innovation, Zhejiang Lab, Hangzhou, China · Institute of Microelectronics, Chinese Academy of Sciences, Beijing 100029, China · University of Chinese Academy of Sciences, Beijing 100049, China
Monocular Depth Estimation (MDE) is a fundamental computer vision task with important applications in 3D vision. The current mainstream MDE methods employ an encoder-decoder architecture with multi-level/scale feature processing. However, the limitations of the current architecture and the effects of different-level features on the prediction accuracy are not evaluated. In this paper, we first investigate the above problem and show that there is still substantial potential in the current framework if encoder features can be improved. Therefore, we propose to formulate the depth estimation problem from the feature restoration perspective, by treating pretrained encoder features as degraded features of an assumed ground truth feature that yields the ground truth depth map. Then an Invertible Transform-enhanced Indirect Diffusion (InvT-IndDiffusion) module is developed for feature restoration. Due to the absence of direct supervision on feature, only indirect supervision from the final sparse depth map is used. During the iterative procedure of diffusion, this results in feature deviations among steps. The proposed InvT-IndDiffusion solves this problem by using an invertible transform-based decoder under the bi-Lipschitz condition. Finally, a plug-and-play Auxiliary Viewpoint-based Low-level Feature Enhancement module (AV-LFE) is developed to enhance local details with auxiliary viewpoint when available. Experiments demonstrate that the proposed method achieves better performance than the state-of-the-art methods on various datasets. Specifically on the KITTI benchmark, compared with the baseline, the performance is improved by 4.09% and 37.77% under different training settings in terms of RMSE. Code is available at https://github.com/whitehb1/IID-RDepth.
Figures & tables
Fig. 1: Illustration of the feature deviation problem in the diffusion process due to the indirect supervision and iterative inference. It can be mitigated with the proposed InvT-IndDiffusion based on the invertible decoder under bi-Lipschitz condition.
Fig. 2: Effects of different levels of features on the final depth prediction performance in terms of RMSE (the smaller the better).
Fig. 3: Overview of the proposed IID-RDepth. The input image is first processed through a frozen encoder to extract the multi-level features. Then the high-level semantic features are combined as mutual conditional maps and fed into the InvT-IndDiffusion for restoration. For low-level detail features, AV-LFE is used as a plug-and-play module to enhance the identity connection when auxiliary viewpoint is available.
Type
Method
RMSE ↓
Δ RMSE ↑
Abs Rel ↓
Sq Rel ↓
RMESlog ↓
ζ1↑
ζ2↑
ζ3↑
Regression
Eigen et al. [ 51 ]
6.307
-203.07%
0.203
1.548
-
-
-
-
P3Depth [ 53 ]
2.842
-36.56%
0.071
0.270
0.103
0.953
0.993
0.998
DORN [ 54 ]
2.727
-31.04%
0.072
0.307
0.120
0.932
0.984
0.995
NeWCRFs [ 33 ]
2.129
-2.31%
0.052
0.155
0.079
0.974
0.997
0.999
URCDC [ 55 ]
2.032
2.35%
0.050
0.142
0.076
0.977
0.997
0.999
EVP [ 29 ]
2.015
3.17%
0.048
0.136
0.073
0.980
0.999
1.000
TABLE I: Quantitative results of the proposed method against the existing methods, using the Eigen split on the KITTI [ 52 ] dataset. ’ ↓ ’ indicates smaller is better, while ’ ↑ ’ denotes larger is better. Δ RMSE is calculated by comparing with the baseline PixelFormer [ 32 ] . † refers to results obtained with both InvT-IndDiffusion and AV-LFE in compatible mode. ‡ refers to results obtained with both InvT-IndDiffusion and AV-LFE in fully trainable mode.
Fig. 4: Qualitative results on the KITTI dataset comparing our IID-RDepth with existing methods. IID-RDepth utilizes the proposed InvT-IndDiffusion, while IID-RDepth † combines both InvT-IndDiffusion and AV-LFE in a compatible mode. With the proposed method, results of both fine-grained details and distant objects are improved. Some examples are zoomed such as the gaps between trees in the first row, the geometric structure of distant road signs in the second row, the edges of nearby white stone monuments along the road in the third row, the roadside signs in the fourth row.
Fig. 5: Visual illustration of the predicted depth maps and their differences when using the InvT-IndDiffusion with different interference steps, where the blue to red colors represent improvement from small to large. The residual map displays the differences of results obtained at different steps. The improvement map further illustrates the reduction in prediction error towards the ground truth depth map, derived from the residuals between the results and the ground truth depth map at different steps, with blue to red indicating the magnitude of the improvement.
DDAD Benchmark
Method
RMSE ↓
Δ RMSE ↑
Abs Rel ↓
Sq Rel ↓
RMESlog ↓
δ1↑
δ2↑
δ3↑
NeWCRFs [ 33 ]
5.21
9.23%
0.111
0.930
0.141
0.891
0.982
0.995
PixelFormer [ 32 ]
5.74
\
0.127
0.134
0.157
0.861
0.972
0.993
Ours-P
5.26
8.36%
0.121
0.988
0.146
0.881
0.977
0.994
Ours-N
4.82
16.03%
0.095
0.793
0.127
0.914
0.985
0.995
TABLE II: Quantitative result comparison on the DDAD [ 61 ] dataset. ’ ↓ ’indicates smaller is better, while ’ ↑ ’ denotes larger is better. Δ RMSE is calculated by comparing with the baseline PixelFormer [ 32 ] . ours-N (-P) means using NeWCRFs (PixelFormer) as the backbone.
Fig. 6: Qualitative Results of Ablation Studies on the AV-LFE Module in the KITTI [48] Dataset. A different pseudo color scheme, changing the colormap by replacing the default Magma colormap to Jet, is used to better illustrate the close-range depth result, with blue to red denoting increasing depth.
Method
RMSE ↓
Δ RMSE ↑
Sq Rel ↓
δ1↑
w/o Diffusoin
2.074
-
0.148
0.976
Diffusion-1
2.020
2.60%
0.142
0.978
Diffusion-6
1.996
3.76%
0.140
0.979
TABLE III: Ablation experiment on the InvT-IndDiffusion module. Diffusion-1 (-6) represents diffusion with 1 (6) inference steps.
Method
RMSE ↓
Δ RMSE ↑
Sq Rel ↓
δ1↑
TF
2.054
-
0.147
0.976
Conv
2.017
1.80%
0.144
0.977
Inv
1.996
2.82%
0.140
0.979
TABLE IV: Results using different types of decoders for the restored features by diffusion.
Methods
Range
RMSE ↓
Δ RMSE ↑
Sq Rel ↓
δ1↑
Pixelformer
0-20m
0.803
-
0.056
0.988
IID-RDepth
0.781
2.74%
0.053
0.989
Pixelformer
20-50m
3.514
-
0.433
0.935
IID-RDepth
3.477
2.41%
0.428
0.938
Pixelformer
50-80m
9.128
-
1.528
0.828
IID-RDepth
8.393
9.02%
1.307
0.862
TABLE V: Results in terms of different depth ranges, including near (0-20m), medium (20-50m) and far (50-80m).
Method
Total number
P<0.05
P<0.01
Ours / NewCRFs [ 33 ]
652
636
620
Ours / Pixelformer [ 32 ]
652
634
627
Ours / idisc [ 60 ]
652
632
623
TABLE VI: Paired t-test for improvement significance evaluation. The numerical values listed under each criterion indicate the number of pixels that our method outperforms the respective competitor.
Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web
Igor Pavlovic, Thiemo Wandel, Anton Obukhov +6
EPFL, HUAWEI Bayer Lab, Switzerland · HUAWEI Bayer Lab, Switzerland · University of Bologna, Italy +1
Monocular depth estimation has long stood as a fundamental challenge in computer vision, enabling a wide range of applications including 3D reconstruction, robotics, autonomous driving, and augmented reality. This survey traces the field's evolution from early learning-based methods to the emergence of transformative foundation models. We begin by framing the problem, distinguishing between relative and metric depth estimation, and highlighting the key challenges that have shaped a decade of research. We then present common problem formulations and introduce the most widely used datasets, covering indoor, outdoor, and synthetic data. Following this, we review major advances prior to the foundation model era, distilling core insights from influential methods that contributed to improvements in accuracy, efficiency, and robustness. The survey then turns to the recent surge of foundation-model-based approaches, categorizing them into discriminative and generative paradigms and emphasizing the critical roles of large-scale pretraining (e.g., DINOv3) and synthetic data. We compare representative models using both quantitative benchmarks and qualitative examples, and discuss natural extensions to video-based depth estimation. Further, to illustrate real-world impact, we highlight the integration of depth estimation into applications such as visual SLAM, content generation, and robot perception. Finally, we outline open challenges and promising research directions as the field advances further into the era of foundation models.
Muxin Liu, Xiaoyang Lyu, Yang-Tian Sun +4
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong, 999077.
Monocular depth estimation (MDE) has attracted increasing interest in the past few years, owing to its important role in 3D vision. MDE is the estimation of a depth map from a monocular image/video to represent the 3D structure of a scene, which is a highly ill-posed problem. To solve this problem, in this paper, we propose a LiftFormer based on lifting theory topology, for constructing an intermediate subspace that bridges the image color features and depth values, and a subspace that enhances the depth prediction around edges. MDE is formulated by transforming the depth value prediction problem into depth-oriented geometric representation (DGR) subspace feature representation, thus bridging the learning from color values to geometric depth values. A DGR subspace is constructed based on frame theory by using linearly dependent vectors in accordance with depth bins to provide a redundant and robust representation. The image spatial features are transformed into the DGR subspace, where these features correspond directly to the depth values. Moreover, considering that edges usually present sharp changes in a depth map and tend to be erroneously predicted, an edge-aware representation (ER) subspace is constructed, where depth features are transformed and further used to enhance the local features around edges. The experimental results demonstrate that our LiftFormer achieves state-of-the-art performance on widely used datasets, and an ablation study validates the effectiveness of both proposed lifting modules in our LiftFormer.
Shuai Li, Huibin Bai, Yanbo Gao +5
School of Control Science and Engineering, Shandong University, and Key Laboratory of Machine Intelligence and System Control, Ministry of Education, Jinan 250100, China. · School of Software, Shandong University, Jinan 250100, China · Shandong University-WeiHai Research Institute of Industrial Technology, Weihai 264209, China +2