While recent advances in generative latent spaces have driven substantial progress in single-image generation, the optimal latent space for novel view synthesis (NVS) remains largely unexplored. In particular, NVS requires geometrically consistent generation across viewpoints, but existing approaches typically operate in a view-independent VAE latent space. In this paper, we propose Geometric Latent Diffusion (GLD), a framework that repurposes the geometrically consistent feature space of geometric foundation models as the latent space for multi-view diffusion. We show that these features not only support high-fidelity RGB reconstruction but also encode strong cross-view geometric correspondences, providing a well-suited latent space for NVS. Our experiments demonstrate that GLD outperforms both VAE and RAE on 2D image quality and 3D consistency metrics, while accelerating training by more than 4.4x compared to the VAE latent space. Notably, GLD remains competitive with state-of-the-art methods that leverage large-scale text-to-image pretraining, despite training its diffusion model from scratch without such generative pretraining.
Figures & tables
Figure 1: (a) Prior multi-view diffusion operates in a view-independent VAE [ 36 ] latent space. (b) Geometric Latent Diffusion (GLD) instead diffuses in a geometrically consistent feature space of geometric foundation models [ 26 , 47 ] , enabling both RGB and geometry decoding. (c) GLD converges 4.4× faster than VAE [ 36 ] and DINO [ 56 ] .
Figure 2: Overview of the proposed framework. (a) Our NVS framework consists of three stages: (1) multi-view diffusion models synthesize features up to boundary layer k=1 via cascaded generation, (2) feature propagation derives deeper features through the frozen DA3 encoder, and (3) the RGB decoder maps the full multi-level features to target views. (b) The detailed architecture of a multi-view diffusion model.
Figure 3
Synthesis Configuration
RGB Metrics
Depth Metrics
Boundary k
Models {Ml}l=0k
PSNR ↑
SSIM ↑
LPIPS ↓
AbsRel ↓
RMSE ↓
δ<1.25↑
k=0
{M0}
12.55
0.323
0.579
0.267
0.400
0.641
k=1
{M0,M1}
13.61
0.366
0.555
0.191
0.311
0.744
k=2
{M0,M1,M2}
13.35
0.355
0.566
0.254
0.393
0.659
k=3
{M0,M1,M2,M3}
13.35
0.355
0.567
0.260
0.402
0.647
Table 2: Boundary Selection. We evaluate NVS performance by varying the synthesis boundary k . Explicitly synthesizing up to level l=1 (Boundary k=1 ) using M0 and M1 provides the best performance across all metrics.
Methods
From Scratch
2D Metrics
3D Metrics
PSNR ↑
SSIM ↑
LPIPS ↓
ATE ↓
RPE r ↓
RPE t ↓
Reproj ↓
MEt3R ↓
RealEstate10K [ 58 ]
MVGenMaster [ 3 ]
15.226
0.588
0.456
0.282
6.42
0.526
0.664
0.339
Matrix3D [ 30 ]
14.490
0.580
0.448
0.413
8.93
0.638
0.666
0.344
CAMEO [ 22 ]
✗
13.800
0.561
0.522
0.446
12.93
0.790
0.661
0.344
NVComposer [ 24 ]
11.140
0.418
0.649
0.829
42.85
1.435
0.860
0.457
Table 3: Quantitative comparison of different methods across in-domain [ 58 , 27 ] and out-of-domain [ 2 ] datasets on 2D and 3D metrics. Bold and underlined values indicate the best and second-best results, respectively. We use reproduced CAT3D † [ 22 ] .
Figure 4: Qualitative results. We compare the rendering quality of target views given two source views. The rows from top to bottom correspond to the Re10K, DL3DV, and out-of-domain Mip-NeRF 360 datasets, respectively.
Source Views
Dataset
Methods
2D Metrics
3D Metrics
PSNR ↑
SSIM ↑
LPIPS ↓
ATE ↓
RPE r ↓
RPE t ↓
Reproj ↓
MEt3R ↓
DINO [ 32 ]
13.49
0.543
0.541
0.442
22.95
0.972
0.735
0.301
Re10K
VAE [ 36 ]
12.99
0.519
0.566
0.371
12.47
0.718
0.689
0.387
GLD (Ours)
13.50
0.552
0.541
0.267
8.42
0.539
0.673
0.306
DINO [ 32 ]
12.80
0.368
0.539
0.743
20.14
1.405
0.738
0.418
N=1
DL3DV
VAE [ 36 ]
12.47
0.360
0.569
0.880
25.93
1.828
0.702
0.420
Table 4: Ablation study across different numbers of source images. We report results with N=1 and N=4 input source views on Re10K and DL3DV. Bold values indicate best results.
Figure 8
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
VAE [ 36 ]
34.53
0.939
0.028
VAE (SDXL) [ 33 ]
34.97
0.945
0.029
RAE (DINO) [ 56 ]
26.78
0.830
0.148
DA3 Dec. ( Drgb )
35.41
0.960
0.019
Table 7: Reconstruction Fidelity Comparison. We evaluate the reconstruction quality using 4,000 randomly sampled images from the Re10K [ 58 ] test set.
2D Metrics
3D Metrics
Methods
PSNR ↑
SSIM ↑
LPIPS ↓
ATE ↓
RPE r ↓
RPE t ↓
Reproj ↓
MEt3R ↓
Independent
18.81
0.692
0.335
0.197
7.179
0.430
0.666
0.335
Cascaded (Ours)
19.00
0.695
0.327
0.182
6.694
0.397
0.652
0.326
Table 8: Cascaded vs. Independent Generation. We evaluate NVS performance on Re10K with N=4 source views.
Methods
Depth
RGB
AbsRel ↓
SqRel ↓
δ1↑
PSNR ↑
Matrix3D [ 30 ]
0.197
0.475
0.731
14.13
GLD (Ours)
0.160
0.410
0.800
14.80
Table 9: Depth Evaluation. Comparison with Matrix3D [ 30 ] on ETH3D [ 39 ] .
Figure 6: 3D Reconstruction Visualization. We unproject the synthesized novel views into 3D space using the jointly generated depth and ray maps.
Figure 7: 3D Reconstruction Comparison: Point clouds unprojected from source and target views RGBD outputs of GLD and Matrix3D [ 30 ] .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Architecture details. (A) Our multi-view diffusion model, featuring a condition encoder and a velocity decoder conditioned on camera poses and multi-view features. (B) The RGB decoder architecture that maps the multi-level latent features back to the pixel space.
Parameter
Condition Encoder
Velocity Decoder
Hidden dimension
C1=768
C2=2048
Number of blocks
28
6 ( Ml ), 2 ( M1→0 )
Attention heads
16
16
MLP activation
SwiGLU
SwiGLU
Normalization
RMSNorm
RMSNorm
Input channels
2C=3072 ( Ml )
Appendix
Table 10: DiT DH configuration used for all level-wise diffusion models in GLD.
Component
Decoder Drgb
Multi-view Diffusion
optimizer
Adam
AdamW
max learning rate
2×10−4
5×10−5
min learning rate
2×10−5
5×10−5
learning rate schedule
cosine decay
constant
optimizer betas
(0.9,0.95)
(0.9,0.95)
weight decay
0.0
0.0
Appendix
Table 11: Training configuration for decoder and discriminator.
Dataset
Train
Eval
Re10K [ 58 ]
66,033
200
DL3DV [ 27 ]
10,176
55
HyperSim [ 35 ]
794
–
TartanAir [ 50 ]
369
–
MipNeRF-360 [ 2 ]
–
9
ETH3D [ 39 ]
–
13
Appendix
Table 12: Number of scenes per dataset used for training and evaluation.
Figure 9: Qualitative results of GLD w/ VGGT.
Methods
2D Metrics
3D Metrics
PSNR ↑
SSIM ↑
LPIPS ↓
ATE ↓
RPE r ↓
RPE t ↓
Reproj ↓
MEt3R ↓
RealEstate10K [ 58 ]
DINO [ 32 ]
15.64
0.601
0.448
0.345
15.59
0.719
0.721
0.319
VAE [ 36 ]
15.66
0.606
0.456
0.278
8.68
0.552
0.681
0.375
GLD w/ VGGT [ 47 ]
16.17
0.596
0.429
0.216
7.17
0.440
0.666
0.325
GLD w/ DA3 [ 26 ]
16.36
0.630
0.431
0.211
7.07
0.444
0.673
0.328
Appendix
Table 13: Quantitative comparison of GLD using DA3 [ 26 ] and VGGT [ 47 ] backbones against baseline latent representations across in-domain [ 58 , 27 ] and out-of-domain [ 2 ] datasets on 2D and 3D metrics. Bold and underlined values indicate the best and second-best results, respectively.
Methods
From Scratch
2D Metrics
3D Metrics
PSNR ↑
SSIM ↑
LPIPS ↓
ATE ↓
RPE r ↓
RPE t ↓
Reproj ↓
MEt3R ↓
Mip-NeRF 360 [ 2 ] (Out-of-domain)
MVGenMaster [ 3 ]
✗
14.170
0.304
0.511
0.320
10.92
0.587
0.676
0.402
MVGenMaster [ 3 ]
✓
11.217
0.197
0.665
1.804
61.291
3.423
0.720
0.479
GLD (Ours)
✓
14.542
0.288
0.504
0.589
15.97
1.071
0.630
0.406
Appendix
Table 14: Quantitative comparison of MVGenMaster [ 3 ] trained from scratch and GLD (Ours) on out-of-domain [ 2 ] datasets.
Figure 10: Additional qualitative results from one source views.
Figure 11: Additional qualitative results from two source views.
Figure 12: Additional qualitative results from four source views.
Figure 13: Additional 3D Visualizations. 3D reconstruction comparisons between Matrix3D and our method. (b) Ground truth is reconstructed using DA3 on GT RGB images. Our method yields significantly more consistent and geometrically accurate results.
Figure 14: Geometric correspondence of trained multi-view diffusion models.
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Kangjie Chen, Xiangyu Li, Dongbin Zhang +12
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University
Novel view synthesis from sparse images must reconcile faithful reconstruction of observed regions with plausible completion of unseen content, while maintaining world consistency across viewpoints. Existing geometry-based methods preserve observed scene structure but often struggle to complete unseen regions, whereas video generative models offer rich appearance priors but accumulate inconsistencies during sequential view generation. We propose GeoVerse, a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model. Specifically, GeoVerse extracts multilevel features from Wan2.2 VACE and injects them into the geometric latent diffusion model via a ControlNet-style adapter, incorporating video-learned appearance priors to enhance structural completion. To enforce cross-view coherence, a global spatial memory continuously aggregates observed and synthesized content, reprojecting target-aligned guidance to anchor subsequent predictions to a shared scene representation. Extensive experiments across diverse datasets demonstrate improved visual quality and geometric consistency, with a 2.23 dB higher PSNR on DL3DV and 32.4% lower ATE on Mip-NeRF360 compared to GLD.
Kerui Ren, Tao Lu, Linning Xu +5
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
Novel view synthesis requires strong 3D geometric consistency and the ability to generate visually coherent images across diverse viewpoints. While recent camera-controlled video diffusion models show promising results, they often suffer from geometric distortions and limited camera controllability. To overcome these challenges, we introduce GeoNVS, a geometry-grounded novel-view synthesizer that enhances both geometric fidelity and camera controllability through explicit 3D geometric guidance. Our key innovation is the Gaussian Splat Feature Adapter (GS-Adapter), which lifts input-view diffusion features into 3D Gaussian representations, renders geometry-constrained novel-view features, and adaptively fuses them with diffusion features to correct geometrically inconsistent representations. Unlike prior methods that inject geometry at the input level, GS-Adapter operates in feature space, avoiding view-dependent color noise that degrades structural consistency. Its plug-and-play design enables zero-shot compatibility with diverse feed-forward geometry models without additional training, and can be adapted to other video diffusion backbones. Experiments across 9 scenes and 18 settings demonstrate state-of-the-art performance, achieving 11.3% and 14.9% improvements over SEVA and CameraCtrl, with up to 2x reduction in translation error and 7x in Chamfer Distance.