We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Figures & tables
Figure 1: We demonstrate our BetterScene approach on diverse in-the-wild scenes. Given sparse inputs, recent novel view synthesis methods suffer from performance degradation due to insufficient visual information. BetterScene enhances novel view rendering quality by mitigating artifacts and recovering view-consistent details at inference time with an alias-free, representation-aligned video diffusion model.
Figure 2: The visual quality and reconstruction FID score (rFID) for autoencoders with different channel sizes. We trained all the autoencoders on the DL3DV-10K Ling et al. (2024) dataset. Results show that the original 4-channel autoencoder design Rombach et al. (2021) , which is widely used in diffusion models is unable to reconstruct fine details. Moreover, as shown in (b) and (c), increasing channel size leads to much better reconstructions. We choose to use a 64-channel BetterScene autoencoder for our video diffusion model.
Figure 3: Overview of our BetterScene . The training process consists of two stages. In the first stage, we train an autoencoder using a representation-aligned and equivariance-regularized objective function. In the second stage, we freeze the pretrained BetterScene-VAE and fine-tune the denoiser U-Net within the SVD framework. We leverage a feed-forward 3DGS rendering module, MVSplat, to generate both coarse synthesized views and corresponding Gaussian feature latents. The SVD module then processes these coarse features to decode enhanced high-quality images.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
MVSplat Chen et al. (2024a)
17.05
0.499
0.435
61.92
latentSplat Wewer et al. (2024)
17.79
0.527
0.391
34.55
MVSplat360 Chen et al. (2024b)
17.81
0.562
0.352
18.89
BetterScene (ours)
17.81
0.579
0.347
16.59
Table 1: A quantitative comparison of novel view synthesis performance using 5 input views . Experiments on the DL3DV-10K dataset follow the setting of Chen et al. (2024b) (Note that since all experimental settings remain identical, we directly adopt the evaluation results for baseline methods reported in Chen et al. (2024b) .)
Figure 4: A visual comparison of enhanced rendering results generated from 5 input views across scenes from the DL3DV benchmark test set. BetterScene demonstrates superior visual quality and enhanced detail consistency compared to existing state-of-the-art approaches.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
rFID ↓
SD-VAE
26.06
0.78
0.12
13.83
BetterScene-VAE(C=16)
25.14
0.758
0.12
13.41
BetterScene-VAE(C=32)
28.21
0.851
0.08
7.41
BetterScene-VAE(C=64)
31.21
0.92
0.04
4.90
Table 2: A quantitative comparison of reconstruction performance across latent channel sizes. The SD-VAE represents the original VAE architecture with 4 latent channels.
Single-image novel view synthesis remains challenging because the underlying 3D geometry is highly ambiguous. Recent diffusion-based approaches produce plausible results, but they often struggle to preserve the geometric structure and spatial coherence of foreground objects. We present GenNVS, a framework for geometry-enhanced novel view synthesis via a disentangled 3D prior. Specifically, GenNVS models foreground objects and the background with 3D Gaussian Splatting and aligns them through a coarse-to-fine geometric optimization process to form a unified 3D scene. This scene conditions a video diffusion model through the proposed Dual-Stream Masking mechanism, which guides synthesis by jointly exploiting rendered validity masks and geometry-aware warping. Experimental results show that GenNVS performs favorably against recent methods in both visual quality and geometric accuracy, while naturally supporting flexible scene editing.
The development of generalizable Novel View Synthesis (NVS) models is critically limited by the scarcity of large-scale training data featuring diverse and precise camera trajectories. While real-world captures are photorealistic, they are typically sparse and discrete. Conversely, synthetic data scales but suffers from a domain gap and often lacks realistic semantics. We introduce FreeScale, a novel framework that leverages the power of scene reconstruction to transform limited real-world image sequences into a scalable source of high-quality training data. Our key insight is that an imperfect reconstructed scene serves as a rich geometric proxy, but naively sampling from it amplifies artifacts. To this end, we propose a certainty-aware free-view sampling strategy identifying novel viewpoints that are both semantically meaningful and minimally affected by reconstruction errors. We demonstrate FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR on challenging out-of-distribution benchmarks. Furthermore, we show that the generated data can actively enhance per-scene 3D Gaussian Splatting optimization, leading to consistent improvements across multiple datasets. Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision. Project page: https://mvp-ai-lab.github.io/FreeScale.
Chenhan Jiang, Yu Chen, Qingwen Zhang +4
Hong Kong University of Science and Technology · National Univeristy of Singapore · KTH Royal Institute of Technology +3
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Kangjie Chen, Xiangyu Li, Dongbin Zhang +12
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University