We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Figures & tables
Figure 2: Overview of Vggt -Diff . The Visual Geometry Conditioner uses VGGT- Ω points and confidence to route appearance-bearing source features into query-aligned conditions. The diffusion state, clean-source image condition, Plücker rays, and routed visual condition are concatenated channel-wise, embedded independently within each view, and jointly processed by a Wan-initialized multi-view DiT. Geometry conditioning is stochastically regularized during training, while flow matching and PTRC supervise target fidelity and cross-view consistency, respectively.
Figure 3: Visual Geometry Conditioning. Our conditioner constructs a query-aligned hard anchor from z-buffered near-front source features. A lightweight layered router forms confidence-weighted front and secondary hypotheses and predicts a residual correction from them and routing statistics.
Figure 4: Qualitative comparison on diverse DL3DV scenes. Vggt -Diff better matches the requested viewpoints while preserving fine geometry, legible text, and object identity; previous methods exhibit blur, structural distortion, or content drift.
Figure 5: Qualitative comparison on Mip-NeRF 360. Without dataset-specific fine-tuning, Vggt -Diff better preserves fine structures, occlusion boundaries, and source-observed appearance while more accurately matching the target view.
Table 5
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Pose-difficulty protocol and representative predictions. Blue frustums denote the six source cameras and purple denotes the query. Rows distinguish trajectory interpolation from extrapolation; columns increase pose novelty from near to far. Frozen VGGT- Ω point clouds, fused from multiple continuous GT-view windows, provide scene context only. Insets compare Vggt -Diff with GT and do not determine the bins.
Figure 7: Evaluation metrics across the clean-10K optimization budget. Under the 192-case half-resolution protocol, PSNR, SSIM, and LPIPS-VGG are measured as training extends from 18K to 100K effective steps. Despite local fluctuations, all metrics improve substantially, with the best results at 100K. The open square denotes the clean-1K model evaluated at the matched 18K budget.
Figure 8: Training dynamics on DL3DV clean-10K. Curves connect non-overlapping 1K-step averages across the full training trajectory. The dashed line marks the original 18K fixed-compute budget. PTRC is reported as the raw, unweighted consistency loss. Its initial rise reflects scheduled activation and the increasingly difficult target curriculum rather than optimization divergence.
Figure 9: VGGT- Ω feature depth and optimization. Intermediate frozen features perform best, while jointly optimizing the Layer 23 visual pathway degrades PSNR.
Figure 10: Point-cloud reconstruction from jointly generated views. The upper and lower blocks use VGGT- Ω and Pi3, respectively. Within each block, normalized point-to-reference errors are shown above the corresponding reconstructions using the displayed color scale. Both reconstructors show fewer displaced points and more coherent geometry for Vggt -Diff , indicating stronger cross-view consistency.
Interpolation
Difficulty
PSNR ↑
SSIM ↑
LPIPS ↓
Direct
PTRC
Direct
PTRC
Direct
PTRC
Near
19.008
20.133
0.531
0.602
0.274
0.221
Mid
17.520
18.103
0.435
0.467
0.341
0.300
Far
15.210
15.670
0.308
0.352
0.440
0.397
Appendix
Table 11: PTRC versus direct velocity matching across pose difficulty. Both use the same half-resolution protocol; PTRC values are highlighted and LPIPS uses VGG features.
Figure 11: Target-view alignment and pixel error. Predicted RGB views are shown above their absolute pixel differences from GT under a shared error scale. Vggt -Diff produces lower and more localized errors than the baselines, indicating closer alignment with the GT viewpoint and fewer spatial or structural mismatches.