Depth Anything has achieved remarkable success in monocular depth estimation with strong generalization ability. However, it suffers from temporal inconsistency in videos, hindering its practical applications. Various methods have been proposed to alleviate this issue by leveraging video generation models or introducing priors from optical flow and camera poses. Nonetheless, these methods are only applicable to short videos (< 10 seconds) and require a trade-off between quality and computational efficiency. We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos (over several minutes) without sacrificing efficiency. We base our model on Depth Anything V2 and replace its head with an efficient spatial-temporal head. We design a straightforward yet effective temporal consistency loss by constraining the temporal depth gradient, eliminating the need for additional geometric priors. The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Moreover, a novel key-frame-based strategy is developed for long video inference. Experiments show that our model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization ability. Comprehensive evaluations on multiple video benchmarks demonstrate that our approach sets a new state-of-the-art in zero-shot video depth estimation. We offer models of different scales to support a range of scenarios, with our smallest model capable of real-time performance at 30 FPS.
Figures & tables
Figure 2: Overall pipeline and the spatio-temporal head . Left: Our model is composed of a backbone encoder from Depth Anything V2 and a newly proposed spatio-temporal head. We jointly train our model on video data using ground-truth depth labels for supervision and on unlabeled images with pseudo labels generated by a teacher model. During training, only the head is learned. Right: Our spatiotemporal head inserts several temporal layers into the DPT head, while preserving the original structure of DPT head [ 28 ] .
Figure 3: Inference strategy for long videos . N is the video clip lenght consumed by our model. Each inference video clip is built by N−To−Tk future frames, To overlapping/adjacent frames, and Tk key frames. The key frames are selected by taking every Δk -th frame going backward. Then, the new depth predictions will be scale-shift-aligned to the previous frames based on the Tk overlapping frames. We use N=32,To=8,Tk=2,Δk=12 .
Method / Metrics
KITTI [ 11 ]
Scannet [ 7 ]
Bonn [ 24 ]
NYUv2 [ 22 ]
Sintel [ 5 ] ( ~ 50 frames)
Scannet (170 frames [ 41 ] )
δ1 Rank
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
TAE (↓)
DAv2-L [ 43 ]
0.137
0.815
0.150
0.768
0.127
0.864
0.094
0.928
0.390
0.541
1.140
5.5
NVDS [ 39 ]
0.233
0.614
0.207
0.628
0.199
0.674
0.217
0.598
0.408
0.464
2.176
8.5
NVDS [ 39 ] + DAv2-L [ 43 ]
0.227
0.617
0.194
0.658
0.191
0.700
0.184
0.679
0.449
0.503
2.536
7.7
ChoronDepth [ 32 ]
0.243
0.576
0.190
0.682
0.159
0.793
0.199
0.665
0.551
0.478
1.022
6.6
DepthCrafter [ 13 ]
0.164
0.753
0.169
0.730
0.153
0.803
0.141
0.822
0.299
0.695
0.639
5.0
Table 1: Zero-shot video depth estimation results . We compare with representative single-image [ 43 ] and video depth estimation models [ 39 , 32 , 13 , 41 ] . “VDA-S” and “VDA-L” denote our model with ViT-Small and ViT-Large backbones, respectively. Models denoted with “-Syn” are trained exclusively on public datasets, as detailed in supplementary. The best and the second best results are highlighted.
Method / Metrics
KITTI
Sintel
NYUv2
ETH3D
DIODE
δ1 Rank
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
DepthCrafter [ 13 ]
0.107
0.891
0.568
0.652
0.082
0.936
0.179
0.793
0.141
0.857
4
DepthAnyVideo [ 41 ]
0.073
0.946
0.687
0.692
0.058
0.963
0.123
0.881
0.072
0.942
2.4
DAv2-L [ 43 ]
0.074
0.946
0.487
0.752
0.045
0.979
0.131
0.865
0.066
0.952
1.4
VDA-L (Ours)
0.075
0.946
0.496
0.754
0.046
0.978
0.132
0.863
0.067
0.950
2
Table 2: Zero-shot single-image depth estimation results . We compare with representative single-image [ 43 ] and video depth estimation models [ 13 , 41 ] with single-frame inputs. “VDA-L” denotes our model with ViT-Large backbone. The best and the second best results are highlighted.
Figure 4: Video depth estimation accuracy for different frame length. We compare our model (VDA-L) with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] from 110 to 500 frames on Bonn [ 24 ] , Scannet [ 7 ] , and NYUv2 [ 22 ] .
Figure 5: Qualitative comparison for real-world long video depth estimation. We compare our model with DAv2-L [ 43 ] and DepthCrafter [ 13 ] on 500-frame videos from Scannet [ 7 ] and Bonn [ 24 ] .
Figure 6: Qualitative comparison for in-the-wild short video depth estimation. We compare with Depth-Anything-V2 [ 43 ] , DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] on videos with less than 100 frames from DAVIS [ 26 ] . Red boxes show incorrect depth estimation while blue boxes show inconsistent depth estimation.
Method
Precision
Latency (ms)
ChronoDepth
FP16
506
DepthCrafter
FP16
910
DepthAnyVideo
FP16
159
NVDS
FP32
204
DAv2-L
FP32
60
VDA-L (Ours)
FP32
67
Table 3: Inference latency comparisons for video depth estimation. We measure average runtime for each frame on a single A100 GPU with a resolution of 518×518 .
Loss
AbsRel (↓)
δ1 (↑)
TAE (↓)
VideoAlign
0.151
0.846
1.326
VideoAlign+SSI
0.151
0.848
1.207
OPW [ 39 ] +SSI
0.182
0.771
0.918
SE+SSI
0.160
0.836
0.753
TGM+SSI (Ours)
0.166
0.832
0.767
Table 4: Ablation studies on the effectiveness of the temporal losses. “VideoAlign” denotes the spatial loss with a shared scale-shift alignment applied to the entire video. “SSI” is the image-level spatial loss used in [ 43 ] . “OPW” refers to the optical flow-based warping loss described in [ 39 ] . “SE” refers to the stable error as introduced in Equation 2 . “TGM” represents our proposed temporal gradient matching loss.
Strategy
Window
AbsRel (↓)
δ1 (↑)
TAE (↓)
Baseline
16
0.157
0.826
0.874
OA
16
0.146
0.845
0.792
OI
16
0.157
0.826
0.783
OI+KR(Ours)
16
0.145
0.849
0.761
OI+KR(Ours)
32
0.144
0.851
0.718
OI+KR(Ours)
48
0.143
0.852
0.732
Table 5: Ablation studies on the effectiveness of different inference strategies and window sizes. “Baseline” denotes directly inference for video clips without overlapping. “OA” denotes inference with a overlap of 4 frames and perform scale-shift alignment across windows. “OI” denotes depth clip stitching with a overlap of 4 frames. “OI+KR” combines the “OI” with our proposed key-frame referencing with extra 2 key-frames.
Figure 7: Qualitative comparisons of different inference strategies. We compare overlap alignment (OA) with our proposed overlap interpolation and key-frame referencing (OI + KR) on a self-captured video with 7320 frames.
Datasets
Image-datasets
Video-datasets
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
TAE (↓)
Video
0.180
0.876
0.145
0.849
0.761
Video + Image
0.167
0.883
0.142
0.852
0.742
Table 6: Ablation studies on the effectiveness of the image dataset distillation. “Video” denotes training using only video datasets. “Video + Image” merges video and image datasets for training using image-level distillation [ 43 ] .
Figure 8: Qualitative comparison for static image depth estimation. We compare our model with Depth-Anything-V2 [ 43 ] , DepthCrafter [ 13 ] , and Depth Any Video [ 41 ] on static image depth estimation. Our model demonstrates visualization results comparable to those of Depth-Anything-V2 [ 43 ] .
Figure 9: Qualitative comparison for real-world long video depth estimation. We compare with Depth-Anything-V2 [ 43 ] and DepthCrafter [ 13 ] on 500-frames videos from Scannet [ 7 ] and Bonn [ 24 ] . We show changes in color and depth over time at the vertical red line in videos. White boxes show inconsistent estimation. Blue boxes show our algorithm has higher accuracy.
Method / Metrics
Params(M)
# Video Training Data(M)
KITTI(110) [ 11 ]
Bonn(110) [ 24 ]
Scannet(90) [ 7 ]
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
AbsRel (↓)
δ1 (↑)
DepthCrafter
2156.7
10.5 ~ 40.5
0.111
0.885
0.066
0.979
0.125
0.848
DepthAnyVideo
1422.8
6
0.073
0.957
0.051
0.981
0.112
0.883
VDA-L (Ours)
381.8
0.55
0.079
0.950
0.053
0.972
0.075
0.954
Table 7: Zero-shot short video depth estimation results . We compare with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] in short video depth benchmark. “VDA-L” denotes our model with ViT-Large backbone. The default inference resolution of our model is set to 518 pixels on the short side, maintaining the aspect ratio. The best and the second best results are highlighted.
Figure 10: Temporal layer. The feature shape is adjusted for temporal attention.
Figure 11: 3D Video Conversion. A video from the DAVIS dataset [ 26 ] is transformed into a 3D video using our model.
Figure 12: Dense point cloud generation. We compare our model with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] for dense point cloud generation on the KITTI dataset [ 11 ] . Our model generates a clean and regular point cloud from multiple frames spanning approximately 5 seconds. In contrast, the point cloud generated by DepthCrafter [ 13 ] contains several obvious discontinuous layers. DepthAnyVideo [ 41 ] produces a point cloud with numerous noisy outliers and noticeable distortion in distant views.
Figure 13: Point cloud generation for wild videos. We compare our method with DepthCrafter [ 13 ] and DepthAnyVideo [ 41 ] using three videos from DAVIS dataset [ 26 ] . Camera intrinsics, along with aligned scale and shift parameters, are derived from processing the first frame of each video through MoGe [ 36 ] . Point cloud distortions are highlighted with red boxes.
Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-video inference without requiring complex temporal alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across benchmarks. Furthermore, DVD successfully unlocks the profound geometric priors implicit in video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.
Hongfei Zhang, Harold Haodong Chen, Chenfei Liao +12
Monocular video depth estimation requires temporal consistency, geometric accuracy, and generalization across diverse scenarios, yet existing methods struggle to achieve all three simultaneously. Discriminative models excel at per-frame accuracy but suffer from temporal drift due to limited context windows, while generative methods improve consistency and generalization at the cost of extensive training data (10M+ samples) and lack of geometric precision. In response to these issues, we introduce ICDepth, a framework that adapts pre-trained text-to-video diffusion transformers for video depth estimation via In-Context Conditioning (ICC), leveraging their rich spatial-temporal priors. To address key challenges in transferring ICC from generation to dense prediction, we propose: (1)SAND-Attention, which ensures precise spatial-temporal alignment via shared RoPE and enforces unidirectional attention to prevent noise contamination; (2)SRFM, which injects DINOv2 semantic and resolution priors to enhance geometric precision. ICDepth achieves state-of-the-art results on multiple benchmarks with remarkable data efficiency, trained on only 0.8M frames (6--13× less than competing generative methods), while demonstrating strong zero-shot generalization to diverse domains.
Xuanhua He, Jiaxin Xie, Mingzhe Zheng +1
The Hong Kong University of Science and Technology, Hong Kong SAR, China
Reliable monocular video depth estimation is crucial for downstream 3D reasoning and embodied AI in endoscopic navigation. However, existing self-supervised approaches typically treat video frames independently or rely on weak temporal regularization. These methods, lacking a holistic perception of the underlying 3D scene, inevitably suffer from geometrically inconsistent predictions and severe cross-frame drift. To address these limitations, we introduce a new paradigm that recasts sequential video depth estimation as an unconstrained multi-view 3D reconstruction problem, enabling full exploitation of the powerful geometric priors embedded in recent 3D foundation models. The core of our approach is a 3D consistency optimization framework driven by three constraints: image-level photometric rendering, explicit world-coordinate geometric alignment, and multi-scale temporal gradient consistency. Such unified optimization elegantly anchors isolated frames to a globally coherent 3D structure. Our method has been validated in both the self-supervised training scenarios and challenging zero-shot clinical environments. Results show that the proposed approach achieves state-of-the-art spatial accuracy, outperforming the frame-based, video-based depth estimators and the multi-view 3D reconstruction baselines.
Yuanye Liu, Ke Zhang, Junzhe Jiang +3
Fudan University, Shanghai, China · Johns Hopkins University, Baltimore, USA