Depth-to-Image Synthesis-Driven Generative Unguided Depth Completion

Jiayi Yuan; Na Zhao; De Wen Soh

cs.CVSep 5, 2026

Depth-to-Image Synthesis-Driven Generative Unguided Depth Completion

Authors: Jiayi Yuan, Na Zhao, De Wen Soh

Organizations: Singapore University of Technology and Design

Abstract

Guided depth completion methods heavily depend on RGB quality and alignment, while unguided ones often suffer from limited precision due to the absence of explicit visual cues. In this paper, we present Depth-to-Image Synthesis-Driven Generative Unguided Depth Completion (GUDC), a new completion paradigm that innovatively bridges advanced 2D generative models with unguided depth completion, enabling semantics-aware depth inference without real RGB inputs. Our key idea is to exploit ControlNet's powerful depth-conditioned generation capability to synthesize pseudo-images directly from sparse depth, effectively converting the original unguided setting into a semantics-guided one. To address the potential image-depth misalignment caused by depth sparsity, we propose a multi-level dense-to-sparse representation distillation strategy for ControlNet fine-tuning, where dense-depth features act as teacher signals to distill consistent structural representations for sparse-depth inputs. Furthermore, during pseudo-image-guided completion, we propose a pseudo-image semantic attention fusion module to adaptively extract informative semantic cues from pseudo-images while suppressing artifacts (e.g., texture hallucinations). Extensive experiments on KITTI and NYUv2 validate that our GUDC achieves superior accuracy and robustness over existing methods.

Figures & tables

Figure 1: Comparison of conventional unguided depth completion and our proposed generative unguided depth completion framework (GUDC). Unlike prior methods that operate solely on sparse depth, our approach employs a depth-conditioned ControlNet to synthesize geometrically aligned pseudo-images, providing semantic priors and structural visual cues that effectively guide the completion process.

Figure 2: Framework of our proposed Generative Unguided Depth Completion . Given sparse depth, we first apply a depth-conditioned ControlNet to generate a geometrically consistent and high-quality pseudo-image. This pseudo-image serves as structural and contextual guidance for subsequent image-guided depth completion. To mitigate potential uncertainties in the generated image, Stable Diffusion is employed as a semantic encoder to extract robust semantic cues from the pseudo-image. These cues are fused with the sparse depth and pseudo-image within an image-guided completion network to predict the final dense depth.

Figure 3: Pipeline of our multi-level dense-to-sparse representation distillation , which transfers structural representations from a dense-depth teacher to a sparse-depth student.

Figure 4: Pipeline of our diffusion-based semantic feature extraction and comparative feature analysis . The left part illustrates the process of extracting high-level semantic features from the U-Net decoder of Stable Diffusion. The right part compares these diffusion-derived features with pseudo-image features extracted by the image encoder of the guided completion network [ Tang et al.(2024)Tang, Tian, An, Li, and Tan ] , revealing stronger contextual consistency, richer semantic abstraction, and better structure-awareness in the diffusion-derived representations.

Figure 5: Architecture of our Pseudo-image Semantic Attention Fusion module , which enhances pseudo-image features using semantic cues from diffusion features and adaptively fuses them with depth features for semantics-aware depth completion.

Figure 6: Qualitative comparisons with SOTA unguided methods on KITTI [ Uhrig et al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and Geiger ] test set, using official visualizations from the benchmark website. Our generative pseudo-images are shown in the second-to-last row for side-by-side comparison with real RGB images.

Methods	RMSE $\downarrow$	MAE $\downarrow$	iRMSE $\downarrow$	iMAE $\downarrow$
SI-CNN [ Uhrig et al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and Geiger ]	1601.33	481.27	4.94	1.78
DCCS [ Chodosh et al.(2019)Chodosh, Wang, and Lucey ]	1325.37	439.48	59.39	3.19
NConv-CNN [ Eldesokey et al.(2018)Eldesokey, Felsberg, and Khan ]	1268.22	360.28	4.67	1.52
Spade-sD [ Jaritz et al.(2018)Jaritz, De Charette, Wirbel, Perrotton, and Nashashibi ]	1035.29	248.32	2.60	0.98
PNCNN [ Eldesokey et al.(2020)Eldesokey, Felsberg, Holmquist, and Persson ]	960.05	251.77	3.37	1.05
S 3 2D d [ Ma et al.(2019)Ma, Cavalheiro, and Karaman ]	954.36	288.64	3.21	1.35

Table 1: Quantitative comparisons with SOTA unguided methods on KITTI [ Uhrig et al.(2017)Uhrig, Schneider, Schneider, Franke, Brox, and Geiger ] .

Panel of Figure 7 — Figure 7: Qualitative comparisons with SOTA methods on NYUv2 [ Silberman et al.(2012)Silberman, Hoiem, Kohli, and Fergus ] (200 samples).

Method	RMSE $\downarrow$
Effect of Multi-level Dense-to-sparse Distillation
Finetune_ControlNet w/o Distillation Loss	941.13
Finetune_ControlNet (#L=1)	934.11
Finetune_ControlNet (#L=2) ∗	932.24
Finetune_ControlNet (#L=3)	932.87
Effect of ControlNet Fine-Tuning Sample Size

Table 4: Ablation studies on KITTI validation dataset. (

\ast

) denotes the default configuration.

Figure 9: Left : Qualitative comparisons of pseudo-images generated by naïve fine-tuning and our distillation-based fine-tuning on KITTI. Right : Epoch-wise performance on NYUv2 test set for depth completion guided by both types of pseudo-images.

Depth Completion

Depth completion estimates dense, metrically grounded scene geometry from sparse depth measurements, often using aligned images and sensor data.

See all 27 papers

Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Mar 5, 2026Zhiyuan Zhou, Ruofeng Liu, Taichi Liu +4Depth Completion Prompt Tuning
Rutgers University, USA · Michigan State University, USA · JD Logistics, China +1
Need for Speed: Zero-Shot Depth Completion with Single-Step Diffusion
Mar 11, 2026Jakub Gregorek, Paraskevas Pegios, Nando Metzger +3Depth Completion Diffusion Models
DTU - Technical University of Denmark · Pioneer Centre for AI · ETH Zürich
Gaussian Belief Propagation Network for Depth Completion
Jan 29, 2026Jie Tang, Pingping Xie, Jian Li +1Markov Random Fields Depth Estimation
National University of Defense Technology, Changsha, China · Hong Kong University of Science and Technology, Hong Kong, China

Mar 5, 2026J/K move · Enter open · S save

Any to Full: Prompting Depth Anything for Depth Completion in One Stage

Zhiyuan Zhou, Ruofeng Liu, Taichi Liu, Weijian Zuo+3

Rutgers University, USA · Michigan State University, USA · JD Logistics, China+1

Depth Completion Prompt Tuning Domain Generalization

Open paper AskPDF arXiv

Accurate, dense depth estimation is crucial for robotic perception, but commodity sensors often yield sparse or incomplete measurements due to hardware limitations. Existing RGBD-fused depth completion methods learn priors jointly conditioned on training RGB distribution and specific depth patterns, limiting domain generalization and robustness to various depth patterns. Recent efforts leverage monocular depth estimation (MDE) models to introduce domain-general geometric priors, but current two-stage integration strategies relying on explicit relative-to-metric alignment incur additional computation and introduce structured distortions. To this end, we present Any2Full, a one-stage, domain-general, and pattern-agnostic framework that reformulates completion as a scale-prompting adaptation of a pretrained MDE model. To address varying depth sparsity levels and irregular spatial distributions, we design a Scale-Aware Prompt Encoder. It distills scale cues from sparse inputs into unified scale prompts, guiding the MDE model toward globally scale-consistent predictions while preserving its geometric priors. Extensive experiments demonstrate that Any2Full achieves superior robustness and efficiency. It outperforms OMNI-DC by 32.2% in average AbsREL and delivers a 1.4

\times

speedup over PriorDA with the same MDE backbone, establishing a new paradigm for universal depth completion. Codes and checkpoints are available at https://github.com/zhiyuandaily/Any2Full.

Abstract

Depth Completion

Explore similar work

Any to Full: Prompting Depth Anything for Depth Completion in One Stage

Need for Speed: Zero-Shot Depth Completion with Single-Step Diffusion

Gaussian Belief Propagation Network for Depth Completion