VDPP: Video Depth Post-Processing for Speed and Scalability

Daewon Yoon; Injun Baek; Sangyu Han; Yearim Kim; Nojun Kwak

cs.CVApr 8, 2026

VDPP: Video Depth Post-Processing for Speed and Scalability

Authors: Daewon Yoon, Injun Baek, Sangyu Han, Yearim Kim, Nojun Kwak

Organizations: Seoul National University · Samsung Electronics

Abstract

Video depth estimation is essential for providing 3D scene structure in applications ranging from autonomous driving to mixed reality. Current end-to-end video depth models have established state-of-the-art performance. Although current end-to-end (E2E) models have achieved state-of-the-art performance, they function as tightly coupled systems that suffer from a significant adaptation lag whenever superior single-image depth estimators are released. To mitigate this issue, post-processing methods such as NVDS offer a modular plug-and-play alternative to incorporate any evolving image depth model without retraining. However, existing post-processing methods still struggle to match the efficiency and practicality of E2E systems due to limited speed, accuracy, and RGB reliance. In this work, we revitalize the role of post-processing by proposing VDPP (Video Depth Post-Processing), a framework that improves the speed and accuracy of post-processing methods for video depth estimation. By shifting the paradigm from computationally expensive scene reconstruction to targeted geometric refinement, VDPP operates purely on geometric refinements in low-resolution space. This design achieves exceptional speed (>43.5 FPS on NVIDIA Jetson Orin Nano) while matching the temporal coherence of E2E systems, with dense residual learning driving geometric representations rather than full reconstructions. Furthermore, our VDPP's RGB-free architecture ensures true scalability, enabling immediate integration with any evolving image depth model. Our results demonstrate that VDPP provides a superior balance of speed, accuracy, and memory efficiency, making it the most practical solution for real-time edge deployment. Our project page is at https://github.com/injun-baek/VDPP

Figures & tables

Category	Method	Speed	Spatial	Temporal	Plug &	Light-
		( $>30$ FPS)	Accuracy	Coherence	Play	weight
End-to-End	VDA-S [ 4 ]	✓	✓	$\triangle$	$\times$	$\times$
End-to-End	VDA-L [ 4 ]	$\times$	✓	✓	$\times$	$\times$
Image-to-Depth	DAv2 [ 31 ]	✓	✓	$\times$	-	✓
Post-Processing	NVDS [ 26 ]	$\times$	$\times$	$\triangle$	✓	$\times$
Post-Processing	VDPP (Ours)	✓	✓	✓	✓	✓

Table 1: Comparison of video depth estimation paradigms. VDPP provides a balanced profile for practical deployment, achieving SOTA-level coherence within a modular and efficient post-processing framework.

Figure 2: The architectural overview of VDPP. Our post-processing pipeline takes per-frame depth maps from any Image Depth Estimator as input. The pipeline begins with using a learnable Differentiable Scaler based on the median to normalize the scale of input depth maps. The normalized map is used to create a 3-channel geometric manifold through down-sampling . A transformer encoder (DINOv2 [ 17 ] ) and Spatial-Temporal Decoder (VDA [ 4 ] ) predict a residual map , which is then added back to the scaled input via a shortcut connection to produce the final depth. Training is supervised by comparing final depth with the ground truth depth using a composite temporal (TGM) and spatial (Affine Invariant) loss function [ 4 ] .

Method	NYUv2 [ 23 ]				Bonn [ 18 ]				Sintel [ 2 ]
Method	FPS ↑	Total (ms)	Image to Depth (ms)	Depth to Video (ms)	FPS ↑	Total (ms)	Image to Depth (ms)	Depth to Video (ms)	FPS ↑	Total (ms)	Image to Depth (ms)	Depth to Video (ms)
End-to-end
VDA-S	14	72.7	—	—	60	16.7	—	—	53	18.8	—	—
VDA-L	1	669.1	—	—	9	109.3	—	—	6	166.2	—	—
Image to Depth
DPT-L	82	12.2	12.2	—	85	11.8	11.8	—	63	15.9	15.9	—

Table 2: Inference speed comparison . Existing post-processing is slow and performs poorly, our VDPP shows superior speed. Notably, DAv2-B + VDPP and DPT-L + VDPP achieve superior speed among video depth estimation methods. Speed is measured in Frames Per Second (FPS) and time components in milliseconds (ms). Performance metrics are reported in Table 3 . Bold numbers indicate the best performance, and underlined numbers represent the second-best, excluding single-frame image to depth methods.

	NYUv2 [ 23 ]			Bonn [ 18 ]			Sintel [ 2 ]
Method	AbsRel $\downarrow$	$\delta_{1}\uparrow$	10 2 $\cdot$ TGSE $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	10 5 $\cdot$ TGSE $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	10 2 $\cdot$ TGSE $\downarrow$
End-to-end
VDA-S	$0.1118$	$0.8926$	$1.4338$	$0.0428$	$0.9885$	$4.9792$	$0.4517$	$0.6266$	$1.6015$
VDA-L	$0.0924$	$0.9312$	$1.0575$	$0.0387$	$0.9886$	$5.2152$	$0.3816$	$0.6722$	$1.2769$
Image to Depth
DPT-L	$0.2033$	$0.6589$	$3.2012$	$0.0633$	$0.9688$	$4.8596$	$0.6573$	$0.5633$	$2.7736$

Table 3: Performance comparison on video datasets. While end-to-end models achieve high accuracy at the expense of speed and scalability, and existing post-processing methods often suffer from poor spatial accuracy and temporal consistency, our VDPP achieves stable performance. In terms of spatial accuracy (AbsRel, δ₁) and temporal consistency (TGSE), VDPP performs on par with state-of-the-art end-to-end models VDA [ 4 ] , while significantly outperforming existing post-processing approaches such as NVDS [ 26 ] . The numbers in parentheses indicate the relative performance to the baselines (DPT [ 20 ] , DAv2 [ 31 ] ): Green / Red denote improvement/degradation. Among post-processing methods, Bold and underlined denote the best and next-best performance, respectively.

Figure 3: Quality comparison of depth methods. We visualize temporal stability using slit-scan image ( purple box ), generated by stacking a red pixel line over time. Ideally, static objects (e.g., swinging structure and arms on the desk) should appear as straight horizontal lines. However, DAv2-L exhibits severe temporal flickering (highlighted by blue arrows ). NVDS struggle with detail depth representation in all videos. Furthermore, VDA-S represents the closer object and the farther object at the similar depth. But VDPP achieve all detail depth representations.

Figure 4: VDPP running on NVIDIA Jetson Orin Nano (8GB) at 43.5 FPS with 5.5 GB memory usage. Other temporal methods (NVDS, VDA-L, VDA-S) failed to run on this edge device due to memory overflow as shown in Appendix E, validating VDPP’s practical deployment advantage.

Figure 5: Post-processing to LiDAR dataset. VDPP enhances LiDAR-derived depth quality on KITTI dataset [ 8 ] . Raw LiDAR point clouds are inpainted to generate dense depth maps, which are then refined by VDPP. Red boxes highlight the reduction of horizontal artifacts caused by the LiDAR scanning method. Blue boxes demonstrate significantly improved spatial detail depth representation, showcasing VDPP’s effectiveness as a sensor-agnostic post-processing solution.

Method	Spatial		Temporal
Method	AbsRel↓	$\delta_{1}$ ↑	10 2 $\cdot$ TGSE↓
Full Model	0.4142	0.6464	1.5616
w/o Diff. Scaler	0.4276	0.6335	1.5667
w/ Average Pooling	0.4325	0.6463	1.5623
w/o Surface Normal	0.4431	0.6476	1.6500
w/o Residual Pred.	0.4375	0.6443	1.8042

Table 4: We conducted an ablation study on the Sintel dataset [ 2 ] to validate the contributions of each component in addressing the trade-off between speed and fidelity. The table illustrates the performance degradation upon each is removal, confirming their individual importance. Bold numbers indicate the best performance, and underlined numbers represent the second-best.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative comparison of VDPP with NVDS (post-processing) and VDA (end-to-end). We compare video depth estimation by accumulating results along the time axis. The temporal profile ( purple box ) is a slit-scan image generated by stacking the red pixel line from each frame sequentially over time. (Top row) A basketball player runs forward and then moves away. Both NVDS and VDA show minimal depth change, whereas VDPP correctly captures the motion. (Second row) A hand moves through clothes on a hanger. The slit-scan image reveals NVDS suffers from severe flickering , even in areas that should be static. VDA lacks fine-grained depth expression. (Third row) Flames flicker inside a transparent fire pit. NVDS again exhibits heavy flickering, and VDA fails to distinguish the depth between the front and back of the pit. VDPP clearly shows the depth difference with almost no flickering. (Bottom row) A train moves laterally across tracks. NVDS fails to estimate depth during the fast motion, while VDA loses the details of the tracks. In all scenarios, VDPP demonstrates superior spatial accuracy and temporal coherence .

Figure 7: Qualitative comparison of fine-grained spatial fidelity across seven challenging scenarios. From left to right: RGB Input, NVDS, VDA-Small, VDA-Large (SOTA), and our VDPP. (Top row) The Water falling scene tests the ability to resolve fast-moving, non-rigid water and individual droplets. (Second row) The Power Lines scene highlights the challenge of preserving thin, static structures. (Third row) The Fence scene evaluates the separation of fine foreground posts from a complex background. (Fourth row) The Bus Stop tests robustness to dynamic background motion (passing buses) behind a thin object. (Fifth row) The Street Trees scene challenges the model to maintain structural integrity during complex camera and background motion. (Sixth row) The Swing scene focuses on resolving intricate details like thin ropes against a main structure. (Bottom row) The Train scene tests the difficult combination of transparent surfaces (windows) and fine details (antennas). In all examples, VDPP demonstrates a superior ability to preserve geometric integrity where other methods introduce artifacts, over-smoothing, or structural breaks.

Method	Speed				Performance Metrics
Method	FPS ↑	Total (ms)	I2D (ms)	D2V (ms)	AbsRel↓	$\delta_{1}\uparrow$	10 2 $\cdot$ TGSE $\downarrow$
End-to-end
VDA-S	65.6	15.3	–	–	0.355	0.681	1.760
VDA-L	18.0	55.5	–	–	0.324	0.729	1.424
Image to Depth
DPT-L	129	7.7	7.7	–	0.548	0.610	2.545

Appendix

Table 5: Performance and Speed Comparison on Sintel (384x384) . This table combines speed (FPS, TPF) and performance (AbsRel,

\delta_{1}

, TGSE) metrics. I2D refers to Image to Depth conversion time, and D2V refers to Depth to Video conversion time. The numbers in parentheses indicate the relative performance compared to the baselines(DPT, DAv2): green denotes improvement, and red denotes degradation. Among post-processing methods, Bold and underlined denote the best and next-best performance, respectively.

Figure 8: Qualitative validation of VDPP on real-world LiDAR data from the KITTI dataset [ 8 ] . We compare the “Raw Inpainted” depth (from LiDAR depth completion) against our refined “VDPP” output across four driving sequences. The red boxes highlight VDPP’s ability to suppress artifacts inherent to the inpainting process, such as horizontal streaking. The blue boxes demonstrate how VDPP enhances depth expression , clarifying object boundaries and improving geometric detail where the raw input was poorly defined. This validation confirms VDPP is not limited to refining single-image models but also serves as a powerful plug-and-play post-processor, significantly improving depth quality from real-world 3D sensor pipelines.

Figure 9: Failed deployment attempts of VDA-L, VDA-S, and NVDS on NVIDIA Jetson Orin Nano, showing Out-Of-Memory errors and system crashes.

Figure 10: Sensitivity and Qualitative Analysis of Scale Flicker. We analyzed metric behavior and visual quality changes while increasing the intensity of Scale Perturbation (

\lambda

) from 0% to 50%. Quantitative Sensitivity (Left): The graph shows metric changes according to perturbation levels. Our TGSE ( red line ) shows a strict monotonic sensitivity , accurately reflecting artifact severity. Conversely, AbsRel ( blue line ) fluctuates irregularly and exhibits unreliable behavior, specifically showing a sharp drop at perturbation level 0.3 despite the intensified flicker. Qualitative Temporal Profiles (Right): Visualization of temporal profiles (slit-scans) for representative Indoor (left) and Outdoor (right) scenes from the Sintel dataset. The top of each block is the Reference Depth Map, with a red horizontal line indicating the slice position. The bottom stack shows temporal slices for Ground truth,

\lambda

= 0.05,

\lambda

= 0.25, and

\lambda

= 0.5 (perturbation levels) in order. As perturbation increases, temporal discontinuities in the form of horizontal banding become evident. TGSE strongly correlates with this visual collapse, while AbsRel fails to capture it consistently.

Explore similar work

DVD: Deterministic Video Depth Estimation with Generative Priors
Mar 12, 2026Hongfei Zhang, Harold Haodong Chen, Chenfei Liao +12Depth Estimation Video Diffusion Models
HKUST(GZ) · HKUST · UCSD +5
Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
Jan 21, 2025Sili Chen, Hengkai Guo, Shengnan Zhu +4Depth Estimation Monocular Depth Estimation
ByteDance
ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning
Jul 2, 2026Xuanhua He, Jiaxin Xie, Mingzhe Zheng +1Video Diffusion Models Depth Estimation
The Hong Kong University of Science and Technology, Hong Kong SAR, China

Mar 12, 2026J/K move · Enter open · S save

DVD: Deterministic Video Depth Estimation with Generative Priors

Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He+11

HKUST(GZ) · HKUST · UCSD+5

Depth Estimation Video Diffusion Models

Open paper AskPDF arXiv

Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-video inference without requiring complex temporal alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across benchmarks. Furthermore, DVD successfully unlocks the profound geometric priors implicit in video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.

Abstract

Explore similar work

DVD: Deterministic Video Depth Estimation with Generative Priors

Video Depth Anything: Consistent Depth Estimation for Super-Long Videos

ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning