Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation

Xinhao Cai; Gensheng Pei; Zeren Sun; Yazhou Yao; Fumin Shen; Wenguan Wang

cs.CVMar 17, 2026

Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation

Authors: Xinhao Cai, Gensheng Pei, Zeren Sun, Yazhou Yao, Fumin Shen, Wenguan Wang

Organizations: Nanjing University of Science and Technology · State Key Laboratory of Intelligent Manufacturing of Advanced Construction Machinery · Department of Electrical and Computer Engineering, Sungkyunkwan University · University of Electronic Science and Technology of China ·

Abstract

In this paper, we propose Iris, a deterministic framework for Monocular Depth Estimation (MDE) that integrates real-world priors into the diffusion model. Conventional feed-forward methods rely on massive training data, yet still miss details. Previous diffusion-based methods leverage rich generative priors yet struggle with synthetic-to-real domain transfer. Iris, in contrast, preserves fine details, generalizes strongly from synthetic to real scenes, and remains efficient with limited training data. To this end, we introduce a two-stage Priors-to-Geometry Deterministic (PGD) schedule: the prior stage uses Spectral-Gated Distillation (SGD) to transfer low-frequency real priors while leaving high-frequency details unconstrained, and the geometry stage applies Spectral-Gated Consistency (SGC) to enforce high-frequency fidelity while refining with synthetic ground truth. The two stages share weights and are executed with a high-to-low timestep schedule. Extensive experimental results confirm that Iris achieves significant improvements in MDE performance with strong in-the-wild generalization.

Figures & tables

Figure 1: Comparison of DAv2 and diffusion-based method. (a) Input. (b) DAv2 [ 46 ] yields accurate global layout and scale but smoother details. (d) The diffusion-based method ( i.e., Lotus [ 14 ] ) preserves fine details and sharper boundaries. This complementarity motivates our Priors-to-Geometry Deterministic (§ 3.1 ) framework; spectral disparity further motivates Spectral-Gated Distillation (§ 3.2 ), which transfers reliable low-frequency real-image priors while deferring high-frequency details.

Figure 2: Comparison of direct stage-1 and stage-2 outputs. (a) Input. (b) Unexpectedly, stage-1 operating at a high timestep with low-pass prior alignment produces crisp boundaries and richer textures. (d) The low-timestep stage-2 refined with synthetic ground truth yields smoother boundaries and more stable geometry. (c) Cumulative spectrum shows that stage-1 carries stronger high-frequency energy. These observations motivate using stage-1 as a high-frequency teacher via Spectral-Gated Consistency (§ 3.3 ).

Figure 3: Iris overview . Iris introduces a two-stage diffusion-based Priors-to-Geometry Deterministic framework that effectively injects real-world priors into the diffusion model. First prior stage injects real-world priors from a frozen teacher under a high-timestep state, while the second geometry stage refines metrically faithful predictions on synthetic supervision at a low-timestep state. In the prior stage, Spectral-Gated Distillation (§ 3.2 ) uses a lightweight low-pass gate to filter noisy teacher predictions into stable low-frequency layout priors, whereas in the geometry stage, Spectral-Gated Consistency (§ 3.3 ) applies a lightweight high-pass gate to transfer sharp boundaries and fine details from stage-1 to stage-2. The two U-Net blocks share weights. Please refer to § 3 for more details.

Figure 4: Visualization of Spectral-Gated Distillation. SGD aligns teacher and student in the low-frequency band, injecting real-world priors for layout and scale, suppressing high-frequency artifacts, and leaving high-frequency components unconstrained for next-stage refinement. See § 3.2 for more details.

Figure 5: Visualization of Spectral-Gated Consistency. Stage-1 naturally yields crisp detail and boundary cues. To leverage these internal cues, SGC encourages agreement between stages in the high-frequency band. See § 3.3 for more details.

Method	Training	KITTI (Outdoor)		NYUv2 (Indoor)		ETH3D (Various)		ScanNet (Indoor)		DIODE (Various)		All Avg	Group Avg
	Data $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	Ranking	Ranking
DiverseDepth [ 51 ]	320K	19.0	70.4	11.7	87.5	22.8	69.4	10.9	88.2	37.6	63.1	15.6	7.6
MiDaS [ 28 ]	2M	23.6	63.0	11.1	88.5	18.4	75.2	12.1	84.6	33.2	71.5	15.2	7.3
LeRes [ 53 ]	354K	14.9	78.4	9.0	91.6	17.1	77.7	9.1	91.7	27.1	76.6	12.3	5.3
Omnidata [ 8 ]	12.2M	14.9	83.5	7.4	94.5	16.6	77.8	7.5	93.6	33.9	74.2	12.2	4.8
DPT [ 27 ]	1.4M	10.0	90.1	9.8	90.3	7.8	94.6	8.2	93.4	18.2	75.8	9.9	3.5

Table 1: Quantitative comparison on zero-shot affine-invariant depth estimation . The upper section lists conventional discriminative methods, and the lower section lists diffusion-based methods. The best performances are bolded . All Avg Ranking averages the per-metric ranks over all methods, while Group Avg Ranking averages ranks computed within each section. ⋆ denotes methods relying on pre-trained Stable Diffusion [ 30 ] . Note that Iris is trained on 59K synthetic images and 100K real images with pseudo labels generated by DAv2 [ 46 ] .Compared with deterministic feed-forward models trained on massive real-image corpora (e.g., the DA [ 45 , 46 ] family), Iris retains a clear advantage in training data efficiency and achieves competitive average performance. Relative to prior diffusion-based methods, Iris further delivers considerable performance gains. Please refer to § 4.2 for more details.

Figure 6: Qualitative comparison on diverse scenes . Iris demonstrates consistent cross-scene generalization and accurate fine-detail modeling. See § 4.3 for details. For more qualitative results, please refer to the supplementary material.

Figure 7: Visualization of ablation studies. Note that two-stage denotes the Prior-to-Geometry pipeline. Single-step supervision with SGD alone only partially absorbs real-world priors. See § 4.4 .

Method	Marigold [ 19 ]	Lotus [ 14 ]	DAv2 [ 46 ]	Iris
Inference Time (s) $\downarrow$	377.7	0.8	2.2	1.3

Table 2: Inference efficiency comparison. Inference time (seconds) measured at

1536^{2}

resolution on a NVIDIA A100 GPU.

Components	KITTI (Outdoor)		ScanNet (Indoor)
	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$
(a)Stochastic	8.7	92.0	6.2	95.3
(b)Deter.	8.3	92.8	5.8	96.0
(c)Deter. + SGD	7.9	93.2	5.6	96.2
(d)Deter. + two-stage	8.2	93.0	5.7	96.0
(e)Deter. + two-stage + vanilla distil	7.5	94.0	5.3	96.6

Table 3: Ablation studies of essential components. Deterministic alone refers to single-step deterministic network. Deterministic + SGD stands for simultaneous supervision on synthetic ground truth and real pseudo labels after single-stage. Note that two-stage denotes Priors-to-Geometry pipeline. Collectively, Deterministic + two-stage + SGD + SGC define our final Priors-to-Geometry Deterministic (PGD) framework. See § 4.4 for details.

$(\alpha,\beta,\gamma)$	KITTI (Outdoor)		ETH3D (Various)		ScanNet (Indoor)
	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$
(0, 0, 0)	8.3	92.8	6.2	96.8	5.8	95.8
(0.5, 0, 0)	7.8	93.7	5.9	97.0	5.5	96.4
(1, 0, 0)	7.5	94.1	5.7	97.2	5.3	96.7
(0, 0, 1)	8.2	93.0	6.1	96.9	5.8	95.9
(1, 0, 1)	7.4	94.2	5.6	97.4	5.2	96.9

Table 4: Ablation studies of hyperparameters.

\alpha

\beta

, and

\gamma

control the relative strengths of SGD, SGC, and the reconstruction loss, respectively. See § 4.4 for details.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Training	KITTI (Outdoor)		NYUv2 (Indoor)		ETH3D (Various)		ScanNet (Indoor)		DIODE (Various)		DA-2K	Group Avg
Method	Data $\downarrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	Acc(%) $\uparrow$	Ranking
DiverseDepth [ 51 ]	320K	19.0	70.4	11.7	87.5	22.8	69.4	10.9	88.2	37.6	63.1	79.3	7.5
MiDaS [ 28 ]	2M	23.6	63.0	11.1	88.5	18.4	75.2	12.1	84.6	33.2	71.5	80.6	7.2
LeRes [ 53 ]	354K	14.9	78.4	9.0	91.6	17.1	77.7	9.1	91.7	27.1	76.6	81.1	5.2
Omnidata [ 8 ]	12.2M	14.9	83.5	7.4	94.5	16.6	77.8	7.5	93.6	33.9	74.2	76.8	5.0
DPT [ 27 ]	1.4M	10.0	90.1	9.8	90.3	7.8	94.6	8.2	93.4	18.2	75.8	83.2	3.5

Appendix

Table S1 : Quantitative comparison on zero-shot affine-invariant depth estimation .

Figure S1 : Visualizations of Joint depth and normal estimation. Iris enables simultaneous depth and normal estimation with fully shared parameters by swapping the task switcher $s_{y}^{\text{depth}}$ and $s_{y}^{\text{normal}}$ . See § B for details.

Figure S2 : Visualizations of Joint depth and normal estimation. Iris enables simultaneous depth and normal estimation with fully shared parameters by swapping the task switcher $s_{y}^{\text{depth}}$ and $s_{y}^{\text{normal}}$ . See § B for details.

Figure S3 : More qualitative results on indoor scenes and paintings. See § D for more details.

Figure S4 : More qualitative results on outdoor scenes. See § D for more details.

Monocular Depth Estimation

Monocular depth estimation infers scene geometry and distance from a single camera image, enabling 3D perception without stereo or depth sensors.

See all 132 papers

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Sep 8, 2026Igor Pavlovic, Thiemo Wandel, Anton Obukhov +6Diffusion Transformer OOD Generalization
EPFL, HUAWEI Bayer Lab, Switzerland · HUAWEI Bayer Lab, Switzerland · University of Bologna, Italy +1
OpenDepth: Repurposing Image Editing Priors for Robust Open-World Monocular Depth Estimation
Oct 13, 2026Mengke Zhang, Tianyi Yang, Xinzhuo Li +1Monocular Depth Estimation Diffusion Models
International School, Beijing University of Posts and Telecommunications, Beijing, China · International School, Queen Mary University of London, Beijing, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China
Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention
Aug 31, 2026Bingde Liu, Wu Ran, Jinglei Zhang +2Monocular Depth Estimation Pixel-Space Diffusion Models
MoE Key Lab of Artificial Intelligence, Institute of AI, Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China

Sep 8, 2026J/K move · Enter open · S save

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Igor Pavlovic, Thiemo Wandel, Anton Obukhov, Luca Bartolomei+5

EPFL, HUAWEI Bayer Lab, Switzerland · HUAWEI Bayer Lab, Switzerland · University of Bologna, Italy+1

Diffusion Transformer OOD Generalization Monocular Depth Estimation Surface Normal Estimation Diffusion Model Fine-Tuning

Open paper AskPDF arXiv

Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web

Abstract

Monocular Depth Estimation

Explore similar work

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

OpenDepth: Repurposing Image Editing Priors for Robust Open-World Monocular Depth Estimation

Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention