Repurposing Geometric Foundation Models for Multi-view Diffusion

Wooseok Jang; Seonghu Jeon; Jisang Han; Jinhyeok Choi; Minkyung Kwon; Seungryong Kim; Saining Xie; Sainan Liu

cs.CVMar 23, 2026

Repurposing Geometric Foundation Models for Multi-view Diffusion

Authors: Wooseok Jang, Seonghu Jeon, Jisang Han, Jinhyeok Choi, Minkyung Kwon, Seungryong Kim, Saining Xie, Sainan Liu

Organizations: KAIST AI · New York University · Intel Labs

Abstract

While recent advances in generative latent spaces have driven substantial progress in single-image generation, the optimal latent space for novel view synthesis (NVS) remains largely unexplored. In particular, NVS requires geometrically consistent generation across viewpoints, but existing approaches typically operate in a view-independent VAE latent space. In this paper, we propose Geometric Latent Diffusion (GLD), a framework that repurposes the geometrically consistent feature space of geometric foundation models as the latent space for multi-view diffusion. We show that these features not only support high-fidelity RGB reconstruction but also encode strong cross-view geometric correspondences, providing a well-suited latent space for NVS. Our experiments demonstrate that GLD outperforms both VAE and RAE on 2D image quality and 3D consistency metrics, while accelerating training by more than 4.4x compared to the VAE latent space. Notably, GLD remains competitive with state-of-the-art methods that leverage large-scale text-to-image pretraining, despite training its diffusion model from scratch without such generative pretraining.

Figures & tables

Figure 1: (a) Prior multi-view diffusion operates in a view-independent VAE [ 36 ] latent space. (b) Geometric Latent Diffusion (GLD) instead diffuses in a geometrically consistent feature space of geometric foundation models [ 26 , 47 ] , enabling both RGB and geometry decoding. (c) GLD converges $\mathbf{4.4\times}$ faster than VAE [ 36 ] and DINO [ 56 ] .

Figure 2: Overview of the proposed framework. (a) Our NVS framework consists of three stages: (1) multi-view diffusion models synthesize features up to boundary layer $k{=}1$ via cascaded generation, (2) feature propagation derives deeper features through the frozen DA3 encoder, and (3) the RGB decoder maps the full multi-level features to target views. (b) The detailed architecture of a multi-view diffusion model.

Synthesis Configuration		RGB Metrics			Depth Metrics
Boundary $k$	Models $\{\mathcal{M}_{l}\}_{l=0}^{k}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	$\delta<1.25\uparrow$
$k=0$	$\{\mathcal{M}_{0}\}$	12.55	0.323	0.579	0.267	0.400	0.641
$k=1$	$\{\mathcal{M}_{0},\mathcal{M}_{1}\}$	13.61	0.366	0.555	0.191	0.311	0.744
$k=2$	$\{\mathcal{M}_{0},\mathcal{M}_{1},\mathcal{M}_{2}\}$	13.35	0.355	0.566	0.254	0.393	0.659
$k=3$	$\{\mathcal{M}_{0},\mathcal{M}_{1},\mathcal{M}_{2},\mathcal{M}_{3}\}$	13.35	0.355	0.567	0.260	0.402	0.647

Table 2: Boundary Selection. We evaluate NVS performance by varying the synthesis boundary

k

. Explicitly synthesizing up to level

l=1

(Boundary

k=1

) using

\mathcal{M}_{0}

and

\mathcal{M}_{1}

provides the best performance across all metrics.

Methods	From Scratch	2D Metrics			3D Metrics
Methods	From Scratch	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$	Reproj $\downarrow$	MEt3R $\downarrow$
RealEstate10K [ 58 ]
MVGenMaster [ 3 ]		15.226	0.588	0.456	0.282	6.42	0.526	0.664	0.339
Matrix3D [ 30 ]		14.490	0.580	0.448	0.413	8.93	0.638	0.666	0.344
CAMEO [ 22 ]	✗	13.800	0.561	0.522	0.446	12.93	0.790	0.661	0.344
NVComposer [ 24 ]		11.140	0.418	0.649	0.829	42.85	1.435	0.860	0.457

Table 3: Quantitative comparison of different methods across in-domain [ 58 , 27 ] and out-of-domain [ 2 ] datasets on 2D and 3D metrics. Bold and underlined values indicate the best and second-best results, respectively. We use reproduced CAT3D † [ 22 ] .

Figure 4: Qualitative results. We compare the rendering quality of target views given two source views. The rows from top to bottom correspond to the Re10K, DL3DV, and out-of-domain Mip-NeRF 360 datasets, respectively.

Source Views	Dataset	Methods	2D Metrics			3D Metrics
Source Views	Dataset	Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$	Reproj $\downarrow$	MEt3R $\downarrow$
		DINO [ 32 ]	13.49	0.543	0.541	0.442	22.95	0.972	0.735	0.301
	Re10K	VAE [ 36 ]	12.99	0.519	0.566	0.371	12.47	0.718	0.689	0.387
		GLD (Ours)	13.50	0.552	0.541	0.267	8.42	0.539	0.673	0.306
		DINO [ 32 ]	12.80	0.368	0.539	0.743	20.14	1.405	0.738	0.418
$N{=}1$	DL3DV	VAE [ 36 ]	12.47	0.360	0.569	0.880	25.93	1.828	0.702	0.420

Table 4: Ablation study across different numbers of source images. We report results with

N{=}1

and

N{=}4

input source views on Re10K and DL3DV. Bold values indicate best results.

Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
VAE [ 36 ]	34.53	0.939	0.028
VAE (SDXL) [ 33 ]	34.97	0.945	0.029
RAE (DINO) [ 56 ]	26.78	0.830	0.148
DA3 Dec. ( $\mathcal{D}_{\text{rgb}}$ )	35.41	0.960	0.019

Table 7: Reconstruction Fidelity Comparison. We evaluate the reconstruction quality using 4,000 randomly sampled images from the Re10K [ 58 ] test set.

	2D Metrics			3D Metrics
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$	Reproj $\downarrow$	MEt3R $\downarrow$
Independent	18.81	0.692	0.335	0.197	7.179	0.430	0.666	0.335
Cascaded (Ours)	19.00	0.695	0.327	0.182	6.694	0.397	0.652	0.326

Table 8: Cascaded vs. Independent Generation. We evaluate NVS performance on Re10K with

N{=}4

source views.

Methods	Depth			RGB
Methods	AbsRel $\downarrow$	SqRel $\downarrow$	$\delta_{1}$ $\uparrow$	PSNR $\uparrow$
Matrix3D [ 30 ]	0.197	0.475	0.731	14.13
GLD (Ours)	0.160	0.410	0.800	14.80

Table 9: Depth Evaluation. Comparison with Matrix3D [ 30 ] on ETH3D [ 39 ] .

Panel of Figure 6 — Figure 6: 3D Reconstruction Visualization. We unproject the synthesized novel views into 3D space using the jointly generated depth and ray maps.

Figure 7: 3D Reconstruction Comparison: Point clouds unprojected from source and target views RGBD outputs of GLD and Matrix3D [ 30 ] .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Architecture details. (A) Our multi-view diffusion model, featuring a condition encoder and a velocity decoder conditioned on camera poses and multi-view features. (B) The RGB decoder architecture that maps the multi-level latent features back to the pixel space.

Parameter	Condition Encoder	Velocity Decoder
Hidden dimension	$C_{1}=768$	$C_{2}=2048$
Number of blocks	28	6 ( $\mathcal{M}_{l}$ ), 2 ( $\mathcal{M}_{1\to 0}$ )
Attention heads	16	16
MLP activation	SwiGLU	SwiGLU
Normalization	RMSNorm	RMSNorm
Input channels	$2C=3072$ ( $\mathcal{M}_{l}$ )

Appendix

Table 10: DiT

{}^{\text{DH}}

configuration used for all level-wise diffusion models in GLD.

Component	Decoder $\mathcal{D_{\text{rgb}}}$	Multi-view Diffusion
optimizer	Adam	AdamW
max learning rate	$2\times 10^{-4}$	$5\times 10^{-5}$
min learning rate	$2\times 10^{-5}$	$5\times 10^{-5}$
learning rate schedule	cosine decay	constant
optimizer betas	$(0.9,\,0.95)$	$(0.9,\,0.95)$
weight decay	0.0	0.0

Appendix

Table 11: Training configuration for decoder and discriminator.

Dataset	Train	Eval
Re10K [ 58 ]	66,033	200
DL3DV [ 27 ]	10,176	55
HyperSim [ 35 ]	794	–
TartanAir [ 50 ]	369	–
MipNeRF-360 [ 2 ]	–	9
ETH3D [ 39 ]	–	13

Appendix

Table 12: Number of scenes per dataset used for training and evaluation.

Figure 9: Qualitative results of GLD w/ VGGT.

RealEstate10K [ 58 ]
Methods	2D Metrics			3D Metrics
Methods	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$	Reproj $\downarrow$	MEt3R $\downarrow$
DINO [ 32 ]	15.64	0.601	0.448	0.345	15.59	0.719	0.721	0.319
VAE [ 36 ]	15.66	0.606	0.456	0.278	8.68	0.552	0.681	0.375
GLD w/ VGGT [ 47 ]	16.17	0.596	0.429	0.216	7.17	0.440	0.666	0.325
GLD w/ DA3 [ 26 ]	16.36	0.630	0.431	0.211	7.07	0.444	0.673	0.328

Appendix

Table 13: Quantitative comparison of GLD using DA3 [ 26 ] and VGGT [ 47 ] backbones against baseline latent representations across in-domain [ 58 , 27 ] and out-of-domain [ 2 ] datasets on 2D and 3D metrics. Bold and underlined values indicate the best and second-best results, respectively.

Methods	From Scratch	2D Metrics			3D Metrics
Methods	From Scratch	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$	Reproj $\downarrow$	MEt3R $\downarrow$
Mip-NeRF 360 [ 2 ] (Out-of-domain)
MVGenMaster [ 3 ]	✗	14.170	0.304	0.511	0.320	10.92	0.587	0.676	0.402
MVGenMaster [ 3 ]	✓	11.217	0.197	0.665	1.804	61.291	3.423	0.720	0.479
GLD (Ours)	✓	14.542	0.288	0.504	0.589	15.97	1.071	0.630	0.406

Appendix

Table 14: Quantitative comparison of MVGenMaster [ 3 ] trained from scratch and GLD (Ours) on out-of-domain [ 2 ] datasets.

Figure 10: Additional qualitative results from one source views.

Figure 11: Additional qualitative results from two source views.

Figure 12: Additional qualitative results from four source views.

Figure 13: Additional 3D Visualizations. 3D reconstruction comparisons between Matrix3D and our method. (b) Ground truth is reconstructed using DA3 on GT RGB images. Our method yields significantly more consistent and geometrically accurate results.

Multi-View Diffusion Models

See all 10 papers

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis
Sep 27, 2026Kangjie Chen, Xiangyu Li, Dongbin Zhang +12Novel View Synthesis Conditional Diffusion Models
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University
GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space
Sep 28, 2026Kerui Ren, Tao Lu, Linning Xu +5Multi-View Consistency Novel View Synthesis
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
GeoNVS: Geometry Grounded Video Diffusion for Novel View Synthesis
Mar 16, 2026Minjun Kang, Inkyu Shin, Taeyeop Lee +3Camera-Controlled Video Generation Video Diffusion Models
KAIST, South Korea · Luma AI, USA

Sep 27, 2026J/K move · Enter open · S save

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis

Kangjie Chen, Xiangyu Li, Dongbin Zhang, Chaoda Zheng+11

XPeng Motors · The Chinese University of Hong Kong · Tsinghua University

Novel View Synthesis Conditional Diffusion Models Multi-View Consistency

Open paper AskPDF arXiv

We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.

Abstract

Multi-View Diffusion Models

Explore similar work

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

GeoNVS: Geometry Grounded Video Diffusion for Novel View Synthesis