Video Depth Anything: Consistent Depth Estimation for Super-Long Videos

Sili Chen; Hengkai Guo; Shengnan Zhu; Feihu Zhang; Zilong Huang; Jiashi Feng; Bingyi Kang

cs.CVJan 21, 2025

Video Depth Anything: Consistent Depth Estimation for Super-Long Videos

Authors: Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, Bingyi Kang

Organizations: ByteDance

Abstract

Depth Anything has achieved remarkable success in monocular depth estimation with strong generalization ability. However, it suffers from temporal inconsistency in videos, hindering its practical applications. Various methods have been proposed to alleviate this issue by leveraging video generation models or introducing priors from optical flow and camera poses. Nonetheless, these methods are only applicable to short videos (< 10 seconds) and require a trade-off between quality and computational efficiency. We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos (over several minutes) without sacrificing efficiency. We base our model on Depth Anything V2 and replace its head with an efficient spatial-temporal head. We design a straightforward yet effective temporal consistency loss by constraining the temporal depth gradient, eliminating the need for additional geometric priors. The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Moreover, a novel key-frame-based strategy is developed for long video inference. Experiments show that our model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization ability. Comprehensive evaluations on multiple video benchmarks demonstrate that our approach sets a new state-of-the-art in zero-shot video depth estimation. We offer models of different scales to support a range of scenarios, with our smallest model capable of real-time performance at 30 FPS.

Figures & tables

Figure 2: Overall pipeline and the spatio-temporal head . Left: Our model is composed of a backbone encoder from Depth Anything V2 and a newly proposed spatio-temporal head. We jointly train our model on video data using ground-truth depth labels for supervision and on unlabeled images with pseudo labels generated by a teacher model. During training, only the head is learned. Right: Our spatiotemporal head inserts several temporal layers into the DPT head, while preserving the original structure of DPT head [ 28 ] .

Figure 3: Inference strategy for long videos . $N$ is the video clip lenght consumed by our model. Each inference video clip is built by $N-T_{o}-T_{k}$ future frames, $T_{o}$ overlapping/adjacent frames, and $T_{k}$ key frames. The key frames are selected by taking every $\Delta_{k}$ -th frame going backward. Then, the new depth predictions will be scale-shift-aligned to the previous frames based on the $T_{k}$ overlapping frames. We use $N=32,T_{o}=8,T_{k}=2,\Delta_{k}=12$ .

Method / Metrics	KITTI [ 11 ]		Scannet [ 7 ]		Bonn [ 24 ]		NYUv2 [ 22 ]		Sintel [ 5 ] ( ~ 50 frames)		Scannet (170 frames [ 41 ] )	$\delta_{1}$ Rank
Method / Metrics	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	TAE (↓)	$\delta_{1}$ Rank
DAv2-L [ 43 ]	0.137	0.815	0.150	0.768	0.127	0.864	0.094	0.928	0.390	0.541	1.140	5.5
NVDS [ 39 ]	0.233	0.614	0.207	0.628	0.199	0.674	0.217	0.598	0.408	0.464	2.176	8.5
NVDS [ 39 ] + DAv2-L [ 43 ]	0.227	0.617	0.194	0.658	0.191	0.700	0.184	0.679	0.449	0.503	2.536	7.7
ChoronDepth [ 32 ]	0.243	0.576	0.190	0.682	0.159	0.793	0.199	0.665	0.551	0.478	1.022	6.6
DepthCrafter [ 13 ]	0.164	0.753	0.169	0.730	0.153	0.803	0.141	0.822	0.299	0.695	0.639	5.0

Table 1: Zero-shot video depth estimation results . We compare with representative single-image [ 43 ] and video depth estimation models [ 39 , 32 , 13 , 41 ] . “VDA-S” and “VDA-L” denote our model with ViT-Small and ViT-Large backbones, respectively. Models denoted with “-Syn” are trained exclusively on public datasets, as detailed in supplementary. The best and the second best results are highlighted.

Method / Metrics	KITTI		Sintel		NYUv2		ETH3D		DIODE		$\delta_{1}$ Rank
Method / Metrics	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	$\delta_{1}$ Rank
DepthCrafter [ 13 ]	0.107	0.891	0.568	0.652	0.082	0.936	0.179	0.793	0.141	0.857	4
DepthAnyVideo [ 41 ]	0.073	0.946	0.687	0.692	0.058	0.963	0.123	0.881	0.072	0.942	2.4
DAv2-L [ 43 ]	0.074	0.946	0.487	0.752	0.045	0.979	0.131	0.865	0.066	0.952	1.4
VDA-L (Ours)	0.075	0.946	0.496	0.754	0.046	0.978	0.132	0.863	0.067	0.950	2

Table 2: Zero-shot single-image depth estimation results . We compare with representative single-image [ 43 ] and video depth estimation models [ 13 , 41 ] with single-frame inputs. “VDA-L” denotes our model with ViT-Large backbone. The best and the second best results are highlighted.

Figure 4: Video depth estimation accuracy for different frame length. We compare our model (VDA-L) with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] from 110 to 500 frames on Bonn [ 24 ] , Scannet [ 7 ] , and NYUv2 [ 22 ] .

Figure 5: Qualitative comparison for real-world long video depth estimation. We compare our model with DAv2-L [ 43 ] and DepthCrafter [ 13 ] on 500-frame videos from Scannet [ 7 ] and Bonn [ 24 ] .

Figure 6: Qualitative comparison for in-the-wild short video depth estimation. We compare with Depth-Anything-V2 [ 43 ] , DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] on videos with less than 100 frames from DAVIS [ 26 ] . Red boxes show incorrect depth estimation while blue boxes show inconsistent depth estimation.

Method	Precision	Latency (ms)
ChronoDepth	FP16	506
DepthCrafter	FP16	910
DepthAnyVideo	FP16	159
NVDS	FP32	204
DAv2-L	FP32	60
VDA-L (Ours)	FP32	67

Table 3: Inference latency comparisons for video depth estimation. We measure average runtime for each frame on a single A100 GPU with a resolution of

518\times 518

Loss	AbsRel (↓)	$\delta_{1}$ (↑)	TAE (↓)
VideoAlign	0.151	0.846	1.326
VideoAlign+SSI	0.151	0.848	1.207
OPW [ 39 ] +SSI	0.182	0.771	0.918
SE+SSI	0.160	0.836	0.753
TGM+SSI (Ours)	0.166	0.832	0.767

Table 4: Ablation studies on the effectiveness of the temporal losses. “VideoAlign” denotes the spatial loss with a shared scale-shift alignment applied to the entire video. “SSI” is the image-level spatial loss used in [ 43 ] . “OPW” refers to the optical flow-based warping loss described in [ 39 ] . “SE” refers to the stable error as introduced in Equation 2 . “TGM” represents our proposed temporal gradient matching loss.

Strategy	Window	AbsRel (↓)	$\delta_{1}$ (↑)	TAE (↓)
Baseline	16	0.157	0.826	0.874
OA	16	0.146	0.845	0.792
OI	16	0.157	0.826	0.783
OI+KR(Ours)	16	0.145	0.849	0.761
OI+KR(Ours)	32	0.144	0.851	0.718
OI+KR(Ours)	48	0.143	0.852	0.732

Table 5: Ablation studies on the effectiveness of different inference strategies and window sizes. “Baseline” denotes directly inference for video clips without overlapping. “OA” denotes inference with a overlap of 4 frames and perform scale-shift alignment across windows. “OI” denotes depth clip stitching with a overlap of 4 frames. “OI+KR” combines the “OI” with our proposed key-frame referencing with extra 2 key-frames.

Figure 7: Qualitative comparisons of different inference strategies. We compare overlap alignment (OA) with our proposed overlap interpolation and key-frame referencing (OI + KR) on a self-captured video with 7320 frames.

Datasets	Image-datasets		Video-datasets
Datasets	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	TAE (↓)
Video	0.180	0.876	0.145	0.849	0.761
Video + Image	0.167	0.883	0.142	0.852	0.742

Table 6: Ablation studies on the effectiveness of the image dataset distillation. “Video” denotes training using only video datasets. “Video + Image” merges video and image datasets for training using image-level distillation [ 43 ] .

Figure 8: Qualitative comparison for static image depth estimation. We compare our model with Depth-Anything-V2 [ 43 ] , DepthCrafter [ 13 ] , and Depth Any Video [ 41 ] on static image depth estimation. Our model demonstrates visualization results comparable to those of Depth-Anything-V2 [ 43 ] .

Figure 9: Qualitative comparison for real-world long video depth estimation. We compare with Depth-Anything-V2 [ 43 ] and DepthCrafter [ 13 ] on 500-frames videos from Scannet [ 7 ] and Bonn [ 24 ] . We show changes in color and depth over time at the vertical red line in videos. White boxes show inconsistent estimation. Blue boxes show our algorithm has higher accuracy.

Method / Metrics	Params(M)	# Video Training Data(M)	KITTI(110) [ 11 ]		Bonn(110) [ 24 ]		Scannet(90) [ 7 ]
Method / Metrics	Params(M)	# Video Training Data(M)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)	AbsRel (↓)	$\delta_{1}$ (↑)
DepthCrafter	2156.7	10.5 ~ 40.5	0.111	0.885	0.066	0.979	0.125	0.848
DepthAnyVideo	1422.8	6	0.073	0.957	0.051	0.981	0.112	0.883
VDA-L (Ours)	381.8	0.55	0.079	0.950	0.053	0.972	0.075	0.954

Table 7: Zero-shot short video depth estimation results . We compare with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] in short video depth benchmark. “VDA-L” denotes our model with ViT-Large backbone. The default inference resolution of our model is set to 518 pixels on the short side, maintaining the aspect ratio. The best and the second best results are highlighted.

Figure 10: Temporal layer. The feature shape is adjusted for temporal attention.

Figure 11: 3D Video Conversion. A video from the DAVIS dataset [ 26 ] is transformed into a 3D video using our model.

Figure 12: Dense point cloud generation. We compare our model with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] for dense point cloud generation on the KITTI dataset [ 11 ] . Our model generates a clean and regular point cloud from multiple frames spanning approximately 5 seconds. In contrast, the point cloud generated by DepthCrafter [ 13 ] contains several obvious discontinuous layers. DepthAnyVideo [ 41 ] produces a point cloud with numerous noisy outliers and noticeable distortion in distant views.

Figure 13: Point cloud generation for wild videos. We compare our method with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] using three videos from DAVIS dataset [ 26 ] . Camera intrinsics, along with aligned scale and shift parameters, are derived from processing the first frame of each video through MoGe [ 36 ] . Point cloud distortions are highlighted with red boxes.

Explore similar work

DVD: Deterministic Video Depth Estimation with Generative Priors
Mar 12, 2026Hongfei Zhang, Harold Haodong Chen, Chenfei Liao +12Depth Estimation Video Diffusion Models
HKUST(GZ) · HKUST · UCSD +5
ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning
Jul 2, 2026Xuanhua He, Jiaxin Xie, Mingzhe Zheng +1Video Diffusion Models Depth Estimation
The Hong Kong University of Science and Technology, Hong Kong SAR, China
3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation
Jun 14, 2026Yuanye Liu, Ke Zhang, Junzhe Jiang +3Depth Estimation Monocular Depth Estimation
Fudan University, Shanghai, China · Johns Hopkins University, Baltimore, USA

Mar 12, 2026J/K move · Enter open · S save

DVD: Deterministic Video Depth Estimation with Generative Priors

Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Jing He+11

HKUST(GZ) · HKUST · UCSD+5

Depth Estimation Video Diffusion Models

Open paper AskPDF arXiv

Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-video inference without requiring complex temporal alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across benchmarks. Furthermore, DVD successfully unlocks the profound geometric priors implicit in video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.

Abstract

Explore similar work

DVD: Deterministic Video Depth Estimation with Generative Priors

ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning

3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation