We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Figures & tables
Figure 2: Overview of Vggt -Diff . The Visual Geometry Conditioner uses VGGT- Ω points and confidence to route appearance-bearing source features into query-aligned conditions. The diffusion state, clean-source image condition, Plücker rays, and routed visual condition are concatenated channel-wise, embedded independently within each view, and jointly processed by a Wan-initialized multi-view DiT. Geometry conditioning is stochastically regularized during training, while flow matching and PTRC supervise target fidelity and cross-view consistency, respectively.
Figure 3: Visual Geometry Conditioning. Our conditioner constructs a query-aligned hard anchor from z-buffered near-front source features. A lightweight layered router forms confidence-weighted front and secondary hypotheses and predicts a residual correction from them and routing statistics.
Figure 4: Qualitative comparison on diverse DL3DV scenes. Vggt -Diff better matches the requested viewpoints while preserving fine geometry, legible text, and object identity; previous methods exhibit blur, structural distortion, or content drift.
Figure 5: Qualitative comparison on Mip-NeRF 360. Without dataset-specific fine-tuning, Vggt -Diff better preserves fine structures, occlusion boundaries, and source-observed appearance while more accurately matching the target view.
Table 5
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Pose-difficulty protocol and representative predictions. Blue frustums denote the six source cameras and purple denotes the query. Rows distinguish trajectory interpolation from extrapolation; columns increase pose novelty from near to far. Frozen VGGT- Ω point clouds, fused from multiple continuous GT-view windows, provide scene context only. Insets compare Vggt -Diff with GT and do not determine the bins.
Figure 7: Evaluation metrics across the clean-10K optimization budget. Under the 192-case half-resolution protocol, PSNR, SSIM, and LPIPS-VGG are measured as training extends from 18K to 100K effective steps. Despite local fluctuations, all metrics improve substantially, with the best results at 100K. The open square denotes the clean-1K model evaluated at the matched 18K budget.
Figure 8: Training dynamics on DL3DV clean-10K. Curves connect non-overlapping 1K-step averages across the full training trajectory. The dashed line marks the original 18K fixed-compute budget. PTRC is reported as the raw, unweighted consistency loss. Its initial rise reflects scheduled activation and the increasingly difficult target curriculum rather than optimization divergence.
Figure 9: VGGT- Ω feature depth and optimization. Intermediate frozen features perform best, while jointly optimizing the Layer 23 visual pathway degrades PSNR.
Figure 10: Point-cloud reconstruction from jointly generated views. The upper and lower blocks use VGGT- Ω and Pi3, respectively. Within each block, normalized point-to-reference errors are shown above the corresponding reconstructions using the displayed color scale. Both reconstructors show fewer displaced points and more coherent geometry for Vggt -Diff , indicating stronger cross-view consistency.
Interpolation
Difficulty
PSNR ↑
SSIM ↑
LPIPS ↓
Direct
PTRC
Direct
PTRC
Direct
PTRC
Near
19.008
20.133
0.531
0.602
0.274
0.221
Mid
17.520
18.103
0.435
0.467
0.341
0.300
Far
15.210
15.670
0.308
0.352
0.440
0.397
Appendix
Table 11: PTRC versus direct velocity matching across pose difficulty. Both use the same half-resolution protocol; PTRC values are highlighted and LPIPS uses VGG features.
Figure 11: Target-view alignment and pixel error. Predicted RGB views are shown above their absolute pixel differences from GT under a shared error scale. Vggt -Diff produces lower and more localized errors than the baselines, indicating closer alignment with the GT viewpoint and fewer spatial or structural mismatches.
Novel view synthesis requires strong 3D geometric consistency and the ability to generate visually coherent images across diverse viewpoints. While recent camera-controlled video diffusion models show promising results, they often suffer from geometric distortions and limited camera controllability. To overcome these challenges, we introduce GeoNVS, a geometry-grounded novel-view synthesizer that enhances both geometric fidelity and camera controllability through explicit 3D geometric guidance. Our key innovation is the Gaussian Splat Feature Adapter (GS-Adapter), which lifts input-view diffusion features into 3D Gaussian representations, renders geometry-constrained novel-view features, and adaptively fuses them with diffusion features to correct geometrically inconsistent representations. Unlike prior methods that inject geometry at the input level, GS-Adapter operates in feature space, avoiding view-dependent color noise that degrades structural consistency. Its plug-and-play design enables zero-shot compatibility with diverse feed-forward geometry models without additional training, and can be adapted to other video diffusion backbones. Experiments across 9 scenes and 18 settings demonstrate state-of-the-art performance, achieving 11.3% and 14.9% improvements over SEVA and CameraCtrl, with up to 2x reduction in translation error and 7x in Chamfer Distance.
Single-image novel view synthesis remains challenging because the underlying 3D geometry is highly ambiguous. Recent diffusion-based approaches produce plausible results, but they often struggle to preserve the geometric structure and spatial coherence of foreground objects. We present GenNVS, a framework for geometry-enhanced novel view synthesis via a disentangled 3D prior. Specifically, GenNVS models foreground objects and the background with 3D Gaussian Splatting and aligns them through a coarse-to-fine geometric optimization process to form a unified 3D scene. This scene conditions a video diffusion model through the proposed Dual-Stream Masking mechanism, which guides synthesis by jointly exploiting rendered validity masks and geometry-aware warping. Experimental results show that GenNVS performs favorably against recent methods in both visual quality and geometric accuracy, while naturally supporting flexible scene editing.
Sparse-view novel view synthesis is a central problem in 3D content creation, but diffusion-based approaches remain limited by iterative denoising, making multi-view generation expensive at inference time. We introduce NAMVIS, a diffusion-free framework that reformulates multi-view image synthesis as geometry-conditioned next-scale autoregression. Instead of generating target views through repeated denoising, NAMVIS predicts discrete visual tokens through a small number of coarse-to-fine scale steps, while sampling all tokens within each scale and across target views in parallel. To anchor this generation process to explicit camera geometry, we propose Multi-scale Projective Pose Encoding, which injects source and target camera transformations into both target-view self-attention and source-to-target cross-attention at every resolution. NAMVIS further combines global conditioning with dense geometry-aware cross-attention, enabling the model to preserve source-view appearance while maintaining target-view consistency. Across Objaverse, GSO, and OmniObject3D, NAMVIS outperforms diffusion-based baselines in PSNR, SSIM, and LPIPS, while running over 3 times faster than the evaluated diffusion baselines under the same evaluation setting. These results suggest that geometry-conditioned next-scale autoregression is a promising and efficient alternative to diffusion for sparse-view multi-view synthesis. Additional qualitative results, videos, and resources are available at https://corl-team.github.io/namvis/
Ramil Khafizov, Ilya Statsenko, Ruslan Rakhimov +3