Need for Speed: Zero-Shot Depth Completion with Single-Step Diffusion

Jakub Gregorek; Paraskevas Pegios; Nando Metzger; Konrad Schindler; Theodora Kontogianni; Lazaros Nalpantidis

cs.CVMar 11, 2026

Need for Speed: Zero-Shot Depth Completion with Single-Step Diffusion

Authors: Jakub Gregorek, Paraskevas Pegios, Nando Metzger, Konrad Schindler, Theodora Kontogianni, Lazaros Nalpantidis

Organizations: DTU - Technical University of Denmark · Pioneer Centre for AI · ETH Zürich

Abstract

We introduce Marigold-SSD, a single-step, late-fusion depth completion framework that leverages strong diffusion priors while eliminating the costly test-time optimization typically associated with diffusion-based methods. By shifting computational burden from inference to finetuning, our approach enables efficient and robust 3D perception under real-world latency constraints. Marigold-SSD achieves significantly faster inference with a training cost of only 4.5 GPU days. We evaluate our method across four indoor and two outdoor benchmarks, demonstrating strong cross-domain generalization and zero-shot performance compared to existing depth completion approaches. Our approach significantly narrows the efficiency gap between diffusion-based and discriminative models. Finally, we challenge common evaluation protocols by analyzing performance under varying input sparsity levels. Page: https://dtu-pas.github.io/marigold-ssd/

Figures & tables

Figure 1: Performance vs speed trade-off. Comparison of our method Marigold-SSD with other diffusion-based approaches Marigold-DC [ 66 ] and Marigold-E2E [ 44 ] + LS (w/o sparse condition) as well as discriminative baselines [ 50 , 88 ] on KITTI dataset [ 18 ] . Marigold-SSD occupies a unique region in the trade-off space closing the efficiency gap to discriminative methods while retaining the benefit of the strong diffusion prior.

Panel of Figure 2 — Figure 2: Marigold-SSD for zero-shot depth completion . We present a single-step diffusion framework with end-to-end fine-tuning as an efficient alternative to the test-time optimization approach of Marigold-DC [ 66 ] . To this end, we introduce a conditional decoder with late fusion to incorporate sparse depth measurements. At inference, our method Marigold-SSD produces high-quality results in a single step, while Marigold-DC typically requires 50 optimization steps per inference and often ensembling 10 inferences for further improvements.

Figure 3: Internal architecture of the conditional decoder. $\mathcal{D}_{\mathbf{C}}$ consists of the VAE decoder $\mathcal{D}$ (top row) and blocks processing the sparse condition $\mathbf{C}$ (bottom row), adapted from the VAE encoder $\mathcal{E}$ (differing in down-sampling positions). Feature maps are concatenated channel‑wise ( $\oplus$ ) at five levels and the fusion blocks use $1\times 1$ convolutions (Eq. 1 ). Conv denotes standard convolution layers, UP, DOWN, and MID blocks are ResNet [ 23 ] -based, and MID blocks additionally containing an attention layer.

	Dataset	Marigold-DC		Marigold-SSD (Ours)
		Time (s)	FPS	Time (s) ( $\uparrow\times$ )	FPS
Indoor	ScanNet (640 $\times$ 480)	25.08	0.04	0.38 ( 66 $\times$ )	2.6
	IBims-1 (640 $\times$ 480)	24.62	0.04	0.38 ( 65 $\times$ )	2.6
	VOID (640 $\times$ 480)	24.98	0.04	0.38 ( 66 $\times$ )	2.6
	NYUv2 (640 $\times$ 480)	24.56	0.04	0.38 ( 65 $\times$ )	2.6
Outdoor	KITTI (1216 $\times$ 352)	35.10	0.03	0.53 ( 66 $\times$ )	1.9

Table 1: Runtime analysis. Average per-image inference time in seconds , throughput in FPS , and relative speedup compared to Marigold-DC. Runtimes are measured on an NVIDIA RTX 4090 GPU. Resolutions shown next to dataset names denote the input size used for the timing experiment. Marigold-DC is timed without ensembling (single run); ensembling (e.g., 10 predictions) would increase runtime approximately linearly.

Type	Method	Time (s) $\downarrow$	RMSE $\downarrow$
Discriminative	NLSPN [ 50 ] (ECCV ’20)	0.039	2.076
	BP-Net [ 62 ] (CVPR ’24)	0.078	-
	CFormer [ 88 ] (CVPR ’24)	0.151	1.935
	OGNI-DC [ 91 ] (ECCV ’24)	0.266	-
	VPP4DC [ 4 ] (3DV ’24)	0.164	1.609
	GBPN [ 63 ] (arXiv preprint ’26)	0.137	-

Table 2: Speed-performance trade-off. Zero-shot performance on KITTI [ 18 ] . All runtimes are evalauted on NVIDIA RTX 4090 GPUs. We time Marigold-SSD and Marigold-DC [ 66 ] (Table 1 ) and VPP4DC [ 4 ] . Performance is taken from [ 66 ] , and runtimes at original resolution for the other discriminative methods from [ 63 ] .

		Indoor								Outdoor				Average
Type	Method	ScanNet		IBims-1		VOID		NYUv2		KITTI		DDAD
Type	Method	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	Rank (Count)
Discriminative	NLSPN [ 50 ] (ECCV ’20)	0.036	0.127	0.049	0.191	0.210	0.668	0.440	0.716	1.335	2.076	2.498	9.231	0.761	2.168	8.50 (12)
	CFormer [ 88 ] (CVPR ’23)	0.120	0.232	0.058	0.206	0.216	0.726	0.186	0.374	0.952	1.935	2.518	9.471	0.675	2.157	10.00 (12)
	SpAgNet [ 12 ] (WACV ’23)	–	–	–	–	0.244	0.706	0.158	0.292	0.518	1.788	4.578	13.236	–	–	10.13 (8)
	BP-Net [ 62 ] (CVPR ’24)	0.122	0.212	0.078	0.289	0.270	0.742	–	–	–	–	2.270	8.344	–	–	11.75 (8)

Table 3: Comparison to state-of-the-art on six zero-shot benchmarks. Most values are taken from [ 66 ] , except for DMD 3 C [ 34 ] and GBPN [ 63 ] , which are reported from their original papers. Following [ 66 ] , we omit BP-Net [ 62 ] and OGNI-DC [ 91 ] on NYUv2 [ 46 ] and KITTI [ 18 ] and SpAgNet [ 12 ] on ScanNet [ 13 ] and IBims-1 [ 30 ] . Similar to Marigold [ 28 ] + optim and Marigold [ 28 ] + LS reported in [ 66 ] , we evaluate Marigold-E2E [ 44 ] + LS as a single-step diffusion baseline. Given the need for speed , we highlight best and second best excluding the ensemble approach of Marigold-DC [ 66 ] which increases runtime by an order of magnitude. Our model version trained on lower density-levels is denoted by ⋆. The rank expresses an average position of the method in the table per metric and dataset.

Figure 4: Qualitative results. Marigold-SSD generally produces smoother depth maps than Marigold-DC [ 66 ] , which tends to over-refine details that can lead to unrealistic scene structures. The black arrows highlight variations in the estimated depth, while the red and blue colors indicate the nearest and farthest regions.

Figure 5: Qualitative results. Both Marigold-SSD and Marigold-DC tend to underestimate sky depth on KITTI and DDAD, consistent with prior Marigold limitations and limited conditioning information in the sky, while they differ in how they estimate fine scene details.

Panel of Figure 6 — Figure 6: Evaluation under multiple levels of depth density on NYUv2 and ScanNet. Depth density is denoted by the number of depth samples (#). See the supplementary material for all datasets.

Panel of Figure 7 — Figure 7: Challenging the models on DDAD. At the commonly used sparsity level of 5000 points even sophisticated models can be outperformed by trivial Barycentric interpolation.

Method	Condition	ScanNet		IBims-1		VOID		NYUv2		KITTI		DDAD
Method	Condition	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$
Frozen VAE⋆	Sparse	0.038	0.075	0.099	0.212	0.254	0.721	0.089	0.173	1.899	3.642	6.612	12.295
Frozen VAE⋆	Interpolated	0.022	0.062	0.057	0.178	0.208	0.663	0.064	0.185	0.978	2.418	3.897	9.473
Conditional Encoder⋆	Sparse	0.063	0.103	0.131	0.257	0.222	0.615	0.124	0.208	1.521	3.059	7.637	13.779
Marigold-SSD⋆	Sparse	0.022	0.062	0.056	0.182	0.177	0.588	0.045	0.128	2.443	4.070	3.855	7.840
Frozen VAE	Sparse	0.045	0.087	0.111	0.237	0.315	0.820	0.105	0.201	1.280	2.860	5.976	11.511

Table 4: Late vs Early Fusion. Comparison with early-fusion: models with (i) frozen VAE or (ii) a trainable conditional encoder. i.e, an early fusion equivalent of our conditional decoder. Models trained with lower density-level are denoted by ⋆. The best and second best results are highlighted for each training setup. Our late fusion strategy outperforms in almost all cases early-fusion approaches.

Panel of Figure 8 — Figure 8: Sampling Density. Models (A), (B) & (C) fine-tuned on different densities. See the supplementary material for all datasets.

Explore similar work

Any to Full: Prompting Depth Anything for Depth Completion in One Stage
Mar 5, 2026Zhiyuan Zhou, Ruofeng Liu, Taichi Liu +4Depth Completion Prompt Tuning
Rutgers University, USA · Michigan State University, USA · JD Logistics, China +1
Depth Completion as Parameter-Efficient Test-Time Adaptation
Feb 16, 2026Bingxin Ke, Qunjie Zhou, Jiahui Huang +5Depth Completion Test-Time Adaptation
NVIDIA · ETH Zürich
Zero-Shot Depth from Defocus
Mar 27, 2026Yiming Zuo, Hongyu Wen, Venkat Subramanian +5Metric Depth Estimation Depth Estimation
Department of Computer Science, Princeton University

Mar 5, 2026J/K move · Enter open · S save

Any to Full: Prompting Depth Anything for Depth Completion in One Stage

Zhiyuan Zhou, Ruofeng Liu, Taichi Liu, Weijian Zuo+3

Rutgers University, USA · Michigan State University, USA · JD Logistics, China+1

Depth Completion Prompt Tuning Domain Generalization

Open paper AskPDF arXiv

Accurate, dense depth estimation is crucial for robotic perception, but commodity sensors often yield sparse or incomplete measurements due to hardware limitations. Existing RGBD-fused depth completion methods learn priors jointly conditioned on training RGB distribution and specific depth patterns, limiting domain generalization and robustness to various depth patterns. Recent efforts leverage monocular depth estimation (MDE) models to introduce domain-general geometric priors, but current two-stage integration strategies relying on explicit relative-to-metric alignment incur additional computation and introduce structured distortions. To this end, we present Any2Full, a one-stage, domain-general, and pattern-agnostic framework that reformulates completion as a scale-prompting adaptation of a pretrained MDE model. To address varying depth sparsity levels and irregular spatial distributions, we design a Scale-Aware Prompt Encoder. It distills scale cues from sparse inputs into unified scale prompts, guiding the MDE model toward globally scale-consistent predictions while preserving its geometric priors. Extensive experiments demonstrate that Any2Full achieves superior robustness and efficiency. It outperforms OMNI-DC by 32.2% in average AbsREL and delivers a 1.4

\times

speedup over PriorDA with the same MDE backbone, establishing a new paradigm for universal depth completion. Codes and checkpoints are available at https://github.com/zhiyuandaily/Any2Full.

Abstract

Explore similar work

Any to Full: Prompting Depth Anything for Depth Completion in One Stage

Depth Completion as Parameter-Efficient Test-Time Adaptation

Zero-Shot Depth from Defocus