Recent advances in generative modeling have substantially enhanced novel view synthesis, yet maintaining consistency across viewpoints remains challenging. Diffusion-based models rely on stochastic noise-to-data transitions, which obscure deterministic structures and yield inconsistent view predictions. We advocate a Data-to-Data Flow Matching framework that learns deterministic transformations between paired views, enhancing view-consistent synthesis through explicit data coupling. Building on this, we propose Probability Density Geodesic Flow Matching (PDG-FM), which aligns interpolation trajectories with density-based geodesics of a data manifold. To enable tractable geodesic estimation, we employ a teacher-student framework that distills density-based geodesic interpolants into an efficient ambient-space predictor. Empirically, our method surpasses diffusion-based baselines on Objaverse and GSO30 datasets, demonstrating improved structural coherence and smoother transitions across views. These results highlight the advantages of incorporating data-dependent geometric regularization into deterministic flow matching for consistent novel view generation.
Figures & tables
Figure 1: From Conditional Diffusion Model to Probability Density Geodesic Flow Matching. Conventional diffusion models learn stochastic noise-to-data transitions, often losing deterministic structure. We instead train a Data-to-Data Flow Matching network to learn continuous deterministic transformations between paired data samples (x0,x1) . To ensure geometric consistency, we propose to deploy a probability-density-based geodesic to align flow matching interpolants with high-density regions of the data manifold. This unified design couples accurate data coupling with manifold-aware regularization, yielding realistic, view-consistent transformations.
Figure 2: Overview of the Probability Density Geodesic Flow Matching (PDG-FM) framework. (a) Data-to-Data Flow Matching framework learns deterministic mappings between paired samples (x0,x1) by encoding source and target views into latent space and predicting intermediate states xt through either linear interpolants (Eq. 6 ) or geodesic interpolants (Eq. 17 ) predicted by the GeodesicNet ϕη . The velocity field vθ(xt,t,c) is conditioned on source view features and Plücker ray embeddings, and its decoded outputs yield novel views consistent with the target pose. (b) Variational Distillation of Geodesics trains GeodesicNet ϕη to produce manifold-aligned interpolants by minimizing path energy defined under the probability density geodesic metric ∥x∥G(x)=x⊤x/p(x)2 , where the density p(x) is estimated using a pretrained diffusion score function (Eq. 12 ). (c) The unified PDG-FM integrates both components, resulting in geometry-aware and manifold-consistent transformations for view-consistent novel view synthesis.
Objaverse
GSO30
Method
FID ↓
CLIP-S ↑
SSIM ↑
PSNR ↑
LPIPS ↓
FID ↓
CLIP-S ↑
SSIM ↑
PSNR ↑
LPIPS ↓
Zero-1-to-3
5.9959
88.6625
0.8446
19.5850
0.0962
12.5796
90.9683
0.8536
19.3038
0.0638
Zero-1-to-3-XL
7.2006
87.1176
0.7916
17.1722
0.1505
15.7340
89.0756
0.8280
18.6022
0.0910
EscherNet
7.9058
87.9020
0.8363
18.7560
0.1061
14.7044
90.1891
0.8595
20.1754
0.0689
Free3D
5.5434
88.9575
0.8537
20.3222
0.0873
12.0620
91.1424
0.8581
19.5610
0.0666
Naive FM
5.5093
88.8786
0.8622
20.8246
0.0809
15.2763
89.8226
0.8631
19.7273
0.0641
Table 1: Novel view synthesis performance on Objaverse and GSO datasets. Comparison of Linear-D2D-FM against diffusion and flow baselines on Objaverse and GSO30.
Figure 3: Qualitative comparisons on Objaverse. Comparison of Linear-D2D-FM, the Noise-to-Data FM (Naive FM) baseline and the diffusion-based Free3D model.
Figure 4: Qualitative comparisons on GSO30. Comparison of Free3D and our method (“Ours”) given the same input view.
Objaverse
Method
FID ↓
CLIP-S ↑
SSIM ↑
LPIPS ↓
Free3D
22.4533
77.9569
0.4470
0.4372
Free3D
8.9367
84.4197
0.7894
0.1493
Naive FM
5.7768
88.9998
0.8679
0.0779
Linear-D2D-FM
5.8223
88.9749
0.8688
0.0782
Table 2: Novel view synthesis performance with 10 NFE. Linear-D2D-FM preserves superior performance with 10 NFE.
Figure 5: Qualitative comparisons between PDG-FM (geodesic) and Linear-D2D-FM (linear) on Objaverse. Visual results showing that PDG-FM generates more geometrically faithful novel views. The improvement reflects the effect of energy-guided optimization along data-dependent geodesics.
Objaverse
Method
FID ↓
CLIP-S ↑
SSIM ↑
LPIPS ↓
Linear-D2D-FM
11.8124
94.3476
0.8736
0.0809
Metric FM
11.9609
92.0370
0.8663
0.0900
PDG-FM
10.4010
92.3368
0.8768
0.0804
Table 3: Effect of probability density geodesic alignment (PDG-FM) compared to linear interpolants (Linear-D2D-FM) and Metric FM under a reduced training setup.
Figure 6: Geodesic Interpolants vs Linear Interpolants. Geodesic interpolants traverse semantically meaningful regions of the data manifold, producing perceptually consistent interpolants. In contrast, linear interpolants merely blend the two endpoints, resulting in limited structural continuity and unrealistic transitions.
Method
Setting
PPL ↓
AOFM ↑
Ground-Truth
6 objs
0.2575
14.4716
Linear
6 objs
0.2130
1.0411
Initial
6 objs
0.5710
6.4773
Geodesic
6 objs
0.5023
13.6968
Table 4: Geodesic Path Study Results. We report PPL and AOFM for linear, DDIM-initialized, and geodesic interpolants, with ground-truth camera trajectories as the reference.
Figure 7: Comparison of Geodesic Gradient Norm across Time for geodesic interpolants, their pre-optimization counterparts, and linear interpolants on training dataset. The optimized geodesic paths have lower residuals thus indicating better satisfaction of the Euler-Lagrange condition, aligning with improved perceptual and geometric consistency.
Objaverse
Method
FID ↓
CLIP-S ↑
SSIM ↑
PSNR ↑
Linear-D2D-FM
5.4324
88.9855
0.8634
20.8447
Linear-D2D-FM(eps=50)
5.8050
88.3582
0.8532
20.0291
Linear-D2D-FM
5.8223
88.9749
0.8688
21.3045
Linear-D2D-FM, U10
5.5146
88.9185
0.8706
21.5952
Linear-D2D-FM, U4
7.4414
88.1954
0.8784
22.2092
Table 5: Ablation study of noise augmentation (top) and timestep sampling (bottom) in Linear-D2D-FM.
Figure 8: Extended Qualitative Results of PDG-FM (geodesic) and Linear-D2D-FM on Objaverse.
Figure 9: Extended Qualitative Results on Objaverse.
Novel view synthesis from sparse images must reconcile faithful reconstruction of observed regions with plausible completion of unseen content, while maintaining world consistency across viewpoints. Existing geometry-based methods preserve observed scene structure but often struggle to complete unseen regions, whereas video generative models offer rich appearance priors but accumulate inconsistencies during sequential view generation. We propose GeoVerse, a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model. Specifically, GeoVerse extracts multilevel features from Wan2.2 VACE and injects them into the geometric latent diffusion model via a ControlNet-style adapter, incorporating video-learned appearance priors to enhance structural completion. To enforce cross-view coherence, a global spatial memory continuously aggregates observed and synthesized content, reprojecting target-aligned guidance to anchor subsequent predictions to a shared scene representation. Extensive experiments across diverse datasets demonstrate improved visual quality and geometric consistency, with a 2.23 dB higher PSNR on DL3DV and 32.4% lower ATE on Mip-NeRF360 compared to GLD.
Kerui Ren, Tao Lu, Linning Xu +5
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Kangjie Chen, Xiangyu Li, Dongbin Zhang +12
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University
Generating 3D-consistent novel views from 2D diffusion models remains a fundamental challenge in computer vision. In this work, we introduce a framework that enhances geometric alignment for monocular view synthesis under drastic camera changes by jointly generating a target view and its corresponding depth map. By conditioning the diffusion pipeline on the single source image, source depth, and target camera rays, our approach forces the model to inherently reason in 3D space even under extreme viewpoint shifts. To guarantee spatial fidelity, we propose a novel per-pixel depth consistency constraint, which verifies and enforces 3D consistency across generated views. Our framework establishes a robust pipeline for geometrically consistent view synthesis, yielding superior visual accuracy and spatial alignment compared to current state-of-the-art methods. With this work, we pave the road towards robust 3D world models capable of understanding complex environments from a single view image.
Ge Ya Luo, Daniel Severo, Brian Karrer +3
FAIR at Meta · Mila – Québec AI Institute · Université de Montréal +3