Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.
Figures & tables
Figure 2: Overview of the GeoNeXt fine-tuning protocol. (a) We start from the pretrained Stable Video Diffusion model and encode the RGB image x , depth d , and surface normal n into the latent space using the frozen Stable Diffusion VAE. (b) Noise is sampled and added to the latent z . The image latent is replicated across the geometry slots (depth and normal), and further conditioned by concatenating the geometry latents. (c) We fine-tune only the GeoNeXt U-Net. (d) The model is optimized with the standard diffusion objective over image, depth, and normal latents to ensure fine-grained alignment and consistency between image and geometry.
Method
Training Data
NYUv2
KITTI
ETH3D
ScanNet
DIODE
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
Large-scale discriminative depth models
MiDaS [ 41 ]
2M
11.1
88.5
23.6
63.0
18.4
75.2
12.1
84.6
33.2
71.5
Omnidata [ 13 ]
12.2M
7.4
94.5
14.9
83.5
16.6
77.8
7.5
93.6
33.9
74.2
DPT [ 40 ]
1.4M
9.8
90.3
10.0
90.1
7.8
94.6
8.2
93.4
18.2
75.8
DA [ 59 ]
62.6M
4.3
98.1
7.6
94.7
12.7
88.2
4.3
98.1
26.0
75.9
Table 1: Quantitative comparison on zero-shot affine-invariant depth estimation. We compare GeoNeXt with representative discriminative and generative methods across five datasets. The upper block lists large-scale discriminative approaches, the middle block reports generative methods trained for depth estimation only, and the lower block presents unified geometry estimation methods, which serve as our main comparison focus. * indicates results reproduced by us. Bold marks the best performance.
Figure 3: Qualitative comparison of monocular depth estimation methods across diverse datasets. GeoNeXt demonstrates superior reconstruction of fine structures.
Figure 4: Qualitative comparison of surface normal estimation methods across diverse datasets. GeoNeXt produces more accurate and detailed normal reconstructions, effectively capturing fine geometric structures and surface orientations.
Method
Training Data
NYUv2
ScanNet
iBims-1
Sintel
OASIS
mean ↓
11.25∘↑
mean ↓
11.25∘↑
mean ↓
11.25∘↑
mean ↓
11.25∘↑
mean ↓
11.25∘↑
Large-scale discriminative normal models
Omnidata [ 13 ]
12.2M
23.1
45.8
22.9
47.4
19.0
62.1
41.5
11.4
24.9
31.0
EESNU [ 2 ]
2.5M
16.2
58.6
-
-
20.0
58.5
42.1
11.5
27.7
24.0
DSINE [ 3 ]
160K
16.4
59.6
16.2
61.0
17.1
67.4
34.9
21.5
24.4
28.8
Generative methods for surface normal estimation only.
Table 2: Quantitative results on zero-shot surface normal estimation. We evaluate GeoNeXt against representative discriminative and generative approaches across five benchmark datasets. The top section summarizes large-scale discriminative baselines, the middle section reports generative methods trained exclusively for surface normal prediction, and the bottom section highlights unified geometry estimation frameworks for direct comparison. * indicates results reproduced by us. Bold marks the best performance.
Method
NYUv2
ETH3D
NYUv2
iBims-N
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Unified geometry estimation
(a) w/o Image Recon.
6.5
95.2
6.7
96.0
17.9
55.8
18.5
65.2
(b) w/ CLIP Embed.
5.6
96.5
5.8
97.0
16.9
59.6
16.5
69.0
(c) Full model
5.3
96.8
5.6
97.2
16.7
60.2
16.4
69.2
Separated geometry estimation
Table 3: Ablation on the generative formulation, model architecture, and joint estimation. Upper block: Models trained for unified geometry estimation; Lower block: Mfiodels trained for depth or surface normal estimation individually.
Method
NYUv2
ETH3D
NYUv2
iBims-N
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
normal-depth
5.7
96.6
5.5
97.2
16.6
60.0
16.5
69.4
depth-normal
5.3
96.8
5.6
97.2
16.7
60.2
16.4
69.2
Table 4: Ablation on reconstruction order. We compare two configurations that assign depth and normal to different frames. The results are nearly identical, demonstrating the robustness of the model to reconstruction order.
Method
ETH3D
iBims-N
NFEs
Inference time (s)
Param (B)
AbsRel ↓
δ1↑
Mean ↓
11.25∘↑
Separated geometry estimation
Marigold [ 29 ]
7.0
95.5
18.5
65.6
1×1
0.8
1.9
Marigold [ 29 ]
6.5
95.9
18.5
64.7
50×10
180.5
1.9
Lotus-G [ 22 ]
5.9
97.0
17.2
66.3
1×1
0.8
1.9
Unified geometry estimation
Table 5: Cost-effectiveness analysis. Upper : models trained on geometric targets; we report combined depth and normal inference time and parameter count. Lower : unified geometry estimation variants. NFEs : number of function evaluations required to obtain a prediction, defined as diffusion steps × ensemble size for diffusion models.
This work presents ViGeo, a feed-forward foundation model for recovering spatially dense and temporally consistent geometry from video sequences. Built upon a plain transformer architecture without task-specific architectural modifications, ViGeo supports streaming, full-sequence, and long-video inference within a unified model. The key design is dynamic chunking attention, which exposes the model to both bidirectional and causal temporal contexts during training and allows it to adapt its attention pattern at test time without retraining. To improve supervision quality, we further introduce a completion-based data refinement framework. This framework trains a video depth completion teacher that conditions on sparse and noisy annotations and exploits video/multi-view context to produce dense, temporally coherent, and geometrically reliable training targets. Beyond depth and point maps, ViGeo also predicts surface normals within the same framework. Trained solely on public datasets, ViGeo achieves state-of-the-art performance across online, offline, and long-video depth estimation, surface normal estimation, and video point map estimation.
Zhu Yu, Jingnan Gao, Runmin Zhang +9
Zhejiang University · Shanghai Jiao Tong University · Tongyi Lab, Alibaba Group +1
Monocular video depth estimation requires temporal consistency, geometric accuracy, and generalization across diverse scenarios, yet existing methods struggle to achieve all three simultaneously. Discriminative models excel at per-frame accuracy but suffer from temporal drift due to limited context windows, while generative methods improve consistency and generalization at the cost of extensive training data (10M+ samples) and lack of geometric precision. In response to these issues, we introduce ICDepth, a framework that adapts pre-trained text-to-video diffusion transformers for video depth estimation via In-Context Conditioning (ICC), leveraging their rich spatial-temporal priors. To address key challenges in transferring ICC from generation to dense prediction, we propose: (1)SAND-Attention, which ensures precise spatial-temporal alignment via shared RoPE and enforces unidirectional attention to prevent noise contamination; (2)SRFM, which injects DINOv2 semantic and resolution priors to enhance geometric precision. ICDepth achieves state-of-the-art results on multiple benchmarks with remarkable data efficiency, trained on only 0.8M frames (6--13× less than competing generative methods), while demonstrating strong zero-shot generalization to diverse domains.
Xuanhua He, Jiaxin Xie, Mingzhe Zheng +1
The Hong Kong University of Science and Technology, Hong Kong SAR, China
Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-video inference without requiring complex temporal alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across benchmarks. Furthermore, DVD successfully unlocks the profound geometric priors implicit in video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.
Hongfei Zhang, Harold Haodong Chen, Chenfei Liao +12