DVD: Deterministic Video Depth Estimation with Generative Priors

Hongfei Zhang; Harold Haodong Chen; Chenfei Liao; Jing He; Zixin Zhang; Haodong Li; Yihao Liang; Kanghao Chen

cs.CVMar 12, 2026

DVD: Deterministic Video Depth Estimation with Generative Priors

Authors: Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He, Zixin Zhang, Haodong Li, Yihao Liang, Kanghao Chen, +7 more

Organizations: HKUST(GZ) · HKUST · UCSD · Princeton University · MBZUAI · SZU · Knowin · UniTrento

Abstract

Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-video inference without requiring complex temporal alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across benchmarks. Furthermore, DVD successfully unlocks the profound geometric priors implicit in video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.

Figures & tables

Figure 1: ( Top ) Comparisons on a $1500$ -frame in-the-wild video highlight a fundamental paradigm trade-off: representative generative models ( e.g. , DepthCrafter ( Hu et al., 2025 ) ) suffer from geometric hallucination , while leading discriminative baselines ( e.g. , VDA ( Chen et al., 2025d ) ) face semantic ambiguity . DVD resolves this dilemma, delivering consistent, high-fidelity geometry. ( Bottom ) DVD achieves superior performance on both short and long videos (averaged on KITTI ( Geiger et al., 2012 ) , ScanNet ( Dai et al., 2017 ) , and Bonn ( Palazzolo et al., 2019 ) ), while successfully unlocking the rich priors implicit in video foundation models using remarkably minimal task-specific data, e.g. , less than $1\%$ of VDA’s training set.

Figure 2: Overview of DVD . ( Top ) A video DiT ( $\mathcal{F}_{\theta}$ ) performs single-pass depth regression, modulated by a structural anchor ( $\tau_{0}$ ). Latent manifold rectification (LMR) mitigates mean collapse via differential constraints. ( Bottom ) For long video depth estimation, overlapping windows ( $\mathcal{W}_{A},\mathcal{W}_{B}$ ) are seamlessly aligned using a closed-form least-squares solver, leveraging the model’s global affine coherence.

Figure 3: Timestep as a structural anchor. Visualizations on NYU ( Nathan Silberman and Fergus, 2012 ) demonstrate a fidelity-stability trade-off. Low ( $\tau=0.0$ ) recovers sharp boundaries but lacks global consistency, whereas high ( $\tau=0.8$ ) causes detail loss ( e.g. , blur). An optimal anchor ( $\tau=0.5$ ) balances these regimes, achieving a trade-off between detail recovery and metric accuracy. More detailed quantitative analyses are shown in Figure 10 .

Figure 4: Timestep embedding similarity. Cosine similarity matrix of timestep embeddings ( $t\in[0,1]$ , stride $0.1$ ). While embeddings are broadly consistent, mid-range timesteps exhibit high similarity with a wider range of states.

Figure 5: LMR mitigates mean collapse. Naive regression ( 2nd Row ) exhibits mean collapse, losing high-frequency details. In contrast, our LMR ( 3rd Row ) enforces differential constraints to rectify the latent manifold, recovering both sharp spatial boundaries and temporal coherence. Quantitative analyses are placed in Figure 11 .

Figure 6: Inter-window overlap consistency ( Geiger et al., 2012 ; Palazzolo et al., 2019 ; Dai et al., 2017 ) . Unlike generative baselines with high alignment error and variance, our deterministic regression yields minimal MSE and zero variance, validating our global affine coherence that bounds inter-window discrepancies to linear transformations.

Method	Train Frames	KITTI		ScanNet		Bonn		Sintel
Method	Train Frames	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
DAv2-L ( Yang et al., 2024c )	-	10.9	0.913	6.4	0.967	6.9	0.957	50.0	0.557
Marigold v1.1 ( Ke et al., 2025c )	-	9.5	0.936	7.6	0.940	9.5	0.936	65.1	0.411
RollingDepth ( Ke et al., 2025a )	-	9.8	0.912	5.8	0.964	5.9	0.966	43.7	0.500
ChoronDepth ( Shao et al., 2025 )	381K	15.2	0.775	17.1	0.818	16.8	0.901	52.8	0.504
DepthCrafter ( Hu et al., 2025 )	$\sim$ 30M	9.9	0.907	7.1	0.960	5.9	0.959	37.1	0.664

Table 1: Zero-shot video depth estimation results . The best and the second best results are highlighted.

Method	Paradigm	Bonn		ScanNet		KITTI
Method	Paradigm	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
DAv2-L ( Yang et al., 2024c )	Diff.+D	8.7	0.952	9.5	0.940	11.9	0.879
Marigold v1.1 ( Ke et al., 2025c )	Diff.+G	11.6	0.890	12.0	0.870	24.7	0.569
RollingDepth ( Ke et al., 2025a )	Diff.+G	7.2	0.966	7.5	0.957	11.1	0.911
ChronoDepth ( Shao et al., 2025 )	Diff.+G	17.3	0.859	21.2	0.715	13.0	0.846
DepthCrafter ( Hu et al., 2025 )	Diff.+G	8.5	0.962	11.4	0.866	12.0	0.858

Table 2: Zero-shot long video depth estimation results. Paradigm denotes the backbone type and inference paradigm, where ”Diff.”, ”ViT”, ”D”, and ”G” denote Diffusion-based, ViT-based, Discriminative, and Generative, respectively.

Method	Bonn		ScanNet		KITTI
Method	B-Recall $\uparrow$	B-F1 $\uparrow$	B-Recall $\uparrow$	B-F1 $\uparrow$	B-Recall $\uparrow$	B-F1 $\uparrow$
ChronoDepth ( Shao et al., 2025 )	0.221	0.319	0.144	0.204	0.049	0.090
DepthCrafter ( Hu et al., 2025 )	0.282	0.185	0.115	0.173	0.082	0.044
VDA ( Chen et al., 2025d )	0.223	0.325	0.147	0.210	0.047	0.088
DVD(Ours)	0.336	0.422	0.208	0.259	0.217	0.285

Table 3: Zero-shot boundary metrics, i.e. , Recall and F1. Higher values indicate sharper boundaries and finer details.

Method	KITTI		DIODE		NYUv2
Method	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$	AbsRel $\downarrow$	$\delta_{1}$ $\uparrow$
ChronoDepth ( Shao et al., 2025 )	-	-	80.3	0.549	21.2	0.767
DepthCrafter ( Hu et al., 2025 )	11.0	0.877	53.3	0.592	17.1	0.868
VDA ( Chen et al., 2025d )	8.3	0.933	27.0	0.730	4.7	0.977
DVD (Ours)	8.1	0.944	23.1	0.738	5.5	0.969

Table 4: Zero-shot single-image depth estimation results.

Figure 7: Qualitative comparison on indoor and outdoor scenes. DVD consistently produces higher fidelity depth with noticeably sharper structural boundaries

Figure 8: ( Left ) Data scaling curve of long video performance on ScanNet. ( Middle ) Inference latency & FPS comparisons. ( Right ) Stability of long video inference on KITTI. These results demonstrate DVD ’s remarkable data efficiency, competitive inference speed, and consistent long-video stability.

Figure 9: Qualitative results on long-horizon indoor navigation. DVD leverages global affine coherence to better preserve sharp boundaries and globally coherent geometry across thousands of frames compared to prior SOTA methods.

Figure 10: Ablation of timestep $\tau$ . The structural anchor $\tau$ dictates a fidelity-stability trade-off. While outdoor scenes are more sensitive, both datasets achieve a balance at $\tau=0.5$ .

Figure 11: ( Left ) Ablation of latent manifold rectification. ( Middle ) Ablation of the sampling strategy. ( Right ) Ablation of training schema. The results demonstrate that latent manifold rectification, deterministic adaptation, and joint image-video training effectively enhance geometric accuracy and structural stability.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 12: Visualization of inter-window affine alignment. ( Left ) Joint pixel density distribution between the current and reference windows in the overlapping region. The strictly linear correlation (red line) confirms that the inter-window discrepancy is predominantly affine. ( Middle ) After applying our calculated affine transformation, the predictions tightly cluster along the ideal $y=x$ diagonal. ( Right ) The histogram of pixel-wise residuals (Aligned minus Reference) exhibits a zero-mean, tightly bounded distribution with minimal variance, demonstrating the high precision of our global affine coherence strategy.

Setting	Hyperparameter	Value
Model Architecture $\&$ Adaptation
Backbone	pre-trained model	Wan2.1-1.3B
VAE	spatial compression	$8\times$
	temporal compression	$4\times$
LoRA	target modules	$W_{q},W_{k},W_{v},W_{o},W_{ffn}$
	rank ( $r$ ) / alpha ( $\alpha$ )	$512/512$

Appendix

Table 5: Hyperparameter configurations for DVD .

	Wan2.1-1.3B	CogVideoX-5B (by $\tau$ )
Metric	DVD	0.1	0.3	0.5	0.7	0.9
AbsRel $\downarrow$	6.7	7.8	7.6	7.4	7.4	9.5
$\delta\uparrow$	0.967	0.931	0.934	0.938	0.935	0.898

Appendix

Table 6: Cross-backbone generalization on KITTI. We deploy DVD on CogVideoX-5B ( Hong et al., 2022 ) . Consistent with our default Wan2.1-1.3B backbone, the mid-range timestep (

\tau=0.5

) serves as the optimal structural anchor.

Figure 13: Qualitative results of DVD on CogVideoX ( Hong et al., 2022 ) . Despite employing a different foundation architecture, our deterministic paradigm still preserves significantly sharper high-frequency details (highlighted in red boxes) compared to the leading baseline.

Timestep	AbsRel $\downarrow$	$\delta_{1}\uparrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$
w/o $\tau$	11.3	0.940	13.7	0.904
$\tau=0.1$	8.6	0.957	9.2	0.918
$\tau=0.2$	6.9	0.972	8.3	0.954
$\tau=0.3$	6.0	0.975	7.5	0.960
$\tau=0.4$	6.1	0.974	7.0	0.966
$\tau=0.5$	5.5	0.974	6.7	0.967

Appendix

Table 7: Analysis of structural anchor

\tau

, which is evaluated on ScanNet ( Left ) and KITTI ( Right ).

Regularizer	AbsRel $\downarrow$	$\bm{\delta_{1}}\uparrow$	B-F1 $\uparrow$
$\mathcal{L}_{2}$ only (Eq.( 2 ))	8.5	0.966	0.210
$+$ RGB reconstruction	10.5	0.951	0.174
$+$ Edge-aware smoothness	7.5	0.978	0.193
$+$ Multi-scale gradient matching	8.2	0.969	0.257
$+$ LMR (Ours)	7.3	0.977	0.259

Appendix

Table 8: Analysis of regularization strategies. All variants use the same backbone, training, and inference settings.

Overlap Size	AbsRel $\downarrow$	$\bm{\delta_{1}}\uparrow$	Rel. Time $\downarrow$
$O=3$	7.9	0.937	1.00 $\times$
$O=6$	7.7	0.941	1.04 $\times$
$O=9$	7.3	0.945	1.17 $\times$
$O=14$	7.2	0.948	1.34 $\times$
$O=19$	7.1	0.947	1.55 $\times$

Appendix

Table 9: Analysis of overlap size on KITTI. Increasing the number of overlapping frames (

O

) improves geometric accuracy but incurs diminishing returns and computational overhead (Rel. Time).

LoRA Rank	AbsRel $\downarrow$	$\delta_{1}\uparrow$
256	7.7	0.974
512	7.3	0.977
1024	7.3	0.979

Appendix

Table 10: Analysis of LoRA ranks.

Figure 14: Failure case analysis on $1100$ -frame sequence with massive scene transitions. While both DVD and VDA ( Chen et al., 2025d ) inevitably suffer from global scale drift across disjointed scenes ( e.g. , contrasting the absolute depth representations between Frame # $1$ and # $500$ ), DVD consistently maintains significantly sharper local structural fidelity ( e.g. , hands in Frames # $800$ –# $900$ ).

Figure 15: More results demonstrations on short videos.

Figure 16: More results demonstrations on short videos.

Figure 17: More results demonstrations on short videos.

Figure 18: More results demonstrations on long videos.

Figure 19: More results demonstrations on long videos.

Figure 20: More results demonstrations on long videos.

Explore similar work

ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning
Jul 2, 2026Xuanhua He, Jiaxin Xie, Mingzhe Zheng +1Video Diffusion Models Depth Estimation
The Hong Kong University of Science and Technology, Hong Kong SAR, China
VDPP: Video Depth Post-Processing for Speed and Scalability
Apr 8, 2026Daewon Yoon, Injun Baek, Sangyu Han +2Depth Estimation Monocular Depth Estimation
Seoul National University · Samsung Electronics
Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
Jan 21, 2025Sili Chen, Hengkai Guo, Shengnan Zhu +4Depth Estimation Monocular Depth Estimation
ByteDance

Jul 2, 2026J/K move · Enter open · S save

ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning

Xuanhua He, Jiaxin Xie, Mingzhe Zheng, Qifeng Chen

The Hong Kong University of Science and Technology, Hong Kong SAR, China

Video Diffusion Models Depth Estimation Monocular Depth Estimation In-Context Learning

Open paper AskPDF arXiv

Monocular video depth estimation requires temporal consistency, geometric accuracy, and generalization across diverse scenarios, yet existing methods struggle to achieve all three simultaneously. Discriminative models excel at per-frame accuracy but suffer from temporal drift due to limited context windows, while generative methods improve consistency and generalization at the cost of extensive training data (10M+ samples) and lack of geometric precision. In response to these issues, we introduce ICDepth, a framework that adapts pre-trained text-to-video diffusion transformers for video depth estimation via In-Context Conditioning (ICC), leveraging their rich spatial-temporal priors. To address key challenges in transferring ICC from generation to dense prediction, we propose: (1)SAND-Attention, which ensures precise spatial-temporal alignment via shared RoPE and enforces unidirectional attention to prevent noise contamination; (2)SRFM, which injects DINOv2 semantic and resolution priors to enhance geometric precision. ICDepth achieves state-of-the-art results on multiple benchmarks with remarkable data efficiency, trained on only 0.8M frames (

6

13\times

less than competing generative methods), while demonstrating strong zero-shot generalization to diverse domains.

Abstract

Explore similar work

ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning

VDPP: Video Depth Post-Processing for Speed and Scalability

Video Depth Anything: Consistent Depth Estimation for Super-Long Videos