Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.
Figures & tables
Figure 2: Overview of the GeoNeXt fine-tuning protocol. (a) We start from the pretrained Stable Video Diffusion model and encode the RGB image x , depth d , and surface normal n into the latent space using the frozen Stable Diffusion VAE. (b) Noise is sampled and added to the latent z . The image latent is replicated across the geometry slots (depth and normal), and further conditioned by concatenating the geometry latents. (c) We fine-tune only the GeoNeXt U-Net. (d) The model is optimized with the standard diffusion objective over image, depth, and normal latents to ensure fine-grained alignment and consistency between image and geometry.
Method
Training Data
NYUv2
KITTI
ETH3D
ScanNet
DIODE
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
Large-scale discriminative depth models
MiDaS [ 41 ]
2M
11.1
88.5
23.6
63.0
18.4
75.2
12.1
84.6
33.2
71.5
Omnidata [ 13 ]
12.2M
7.4
94.5
14.9
83.5
16.6
77.8
7.5
93.6
33.9
74.2
DPT [ 40 ]
1.4M
9.8
90.3
10.0
90.1
7.8
94.6
8.2
93.4
18.2
75.8
DA [ 59 ]
62.6M
4.3
98.1
7.6
94.7
12.7
88.2
4.3
98.1
26.0
75.9
Table 1: Quantitative comparison on zero-shot affine-invariant depth estimation. We compare GeoNeXt with representative discriminative and generative methods across five datasets. The upper block lists large-scale discriminative approaches, the middle block reports generative methods trained for depth estimation only, and the lower block presents unified geometry estimation methods, which serve as our main comparison focus. * indicates results reproduced by us. Bold marks the best performance.
Figure 3: Qualitative comparison of monocular depth estimation methods across diverse datasets. GeoNeXt demonstrates superior reconstruction of fine structures.
Figure 4: Qualitative comparison of surface normal estimation methods across diverse datasets. GeoNeXt produces more accurate and detailed normal reconstructions, effectively capturing fine geometric structures and surface orientations.
Method
Training Data
NYUv2
ScanNet
iBims-1
Sintel
OASIS
mean ↓
11.25∘↑
mean ↓
11.25∘↑
mean ↓
11.25∘↑
mean ↓
11.25∘↑
mean ↓
11.25∘↑
Large-scale discriminative normal models
Omnidata [ 13 ]
12.2M
23.1
45.8
22.9
47.4
19.0
62.1
41.5
11.4
24.9
31.0
EESNU [ 2 ]
2.5M
16.2
58.6
-
-
20.0
58.5
42.1
11.5
27.7
24.0
DSINE [ 3 ]
160K
16.4
59.6
16.2
61.0
17.1
67.4
34.9
21.5
24.4
28.8
Generative methods for surface normal estimation only.
Table 2: Quantitative results on zero-shot surface normal estimation. We evaluate GeoNeXt against representative discriminative and generative approaches across five benchmark datasets. The top section summarizes large-scale discriminative baselines, the middle section reports generative methods trained exclusively for surface normal prediction, and the bottom section highlights unified geometry estimation frameworks for direct comparison. * indicates results reproduced by us. Bold marks the best performance.
Method
NYUv2
ETH3D
NYUv2
iBims-N
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
Unified geometry estimation
(a) w/o Image Recon.
6.5
95.2
6.7
96.0
17.9
55.8
18.5
65.2
(b) w/ CLIP Embed.
5.6
96.5
5.8
97.0
16.9
59.6
16.5
69.0
(c) Full model
5.3
96.8
5.6
97.2
16.7
60.2
16.4
69.2
Separated geometry estimation
Table 3: Ablation on the generative formulation, model architecture, and joint estimation. Upper block: Models trained for unified geometry estimation; Lower block: Mfiodels trained for depth or surface normal estimation individually.
Method
NYUv2
ETH3D
NYUv2
iBims-N
AbsRel ↓
δ1↑
AbsRel ↓
δ1↑
Mean ↓
11.25∘↑
Mean ↓
11.25∘↑
normal-depth
5.7
96.6
5.5
97.2
16.6
60.0
16.5
69.4
depth-normal
5.3
96.8
5.6
97.2
16.7
60.2
16.4
69.2
Table 4: Ablation on reconstruction order. We compare two configurations that assign depth and normal to different frames. The results are nearly identical, demonstrating the robustness of the model to reconstruction order.
Method
ETH3D
iBims-N
NFEs
Inference time (s)
Param (B)
AbsRel ↓
δ1↑
Mean ↓
11.25∘↑
Separated geometry estimation
Marigold [ 29 ]
7.0
95.5
18.5
65.6
1×1
0.8
1.9
Marigold [ 29 ]
6.5
95.9
18.5
64.7
50×10
180.5
1.9
Lotus-G [ 22 ]
5.9
97.0
17.2
66.3
1×1
0.8
1.9
Unified geometry estimation
Table 5: Cost-effectiveness analysis. Upper : models trained on geometric targets; we report combined depth and normal inference time and parameter count. Lower : unified geometry estimation variants. NFEs : number of function evaluations required to obtain a prediction, defined as diffusion steps × ensemble size for diffusion models.