Single-image novel view synthesis requires viewpoint-dependent structural changes while retaining source-scene information and inferring unobserved content. We formulate this task as a camera-conditioned initialization problem for a frozen pretrained latent diffusion model. At an intermediate DDIM inversion timestep, we decompose the source representation into a structural mean and a source noise residual. A pose-conditioned translation U-Net (TUNet) learns to predict the target-view mean from the source mean. We then introduce a score-guided fusion strategy: the predicted mean and source noise residual form a proposal that is evaluated by the frozen denoiser, whose noise prediction is combined with the translated mean to initialize DDIM sampling. Only TUNet is trained, while the pretrained model synthesizes the final view without explicit reconstruction or rendering of a shared 3D scene. Experiments on MVImgNet and RealEstate10K demonstrate competitive performance. Beyond image-level metrics, Wasserstein comparisons show that the corrected initialization yields trajectory-curvature distributions closer to those of ground-truth target-initialized sampling than raw fusion. These results support structural translation and source-conditioned initialization as a practical approach to reusing frozen diffusion priors for novel view synthesis.
Figures & tables
Figure 2: Overview: Given a single reference image xref , we first apply DDIM inversion up to t=600 to obtain the mean latent zref,μinv . This, together with camera intrinsics/extrinsics, class embeddings, and ray information, is fed into our translation network TUNet . TUNet predicts the target-view mean latent z~tar,μinv , which we combine with the corresponding noise component using our fusion strategy to form the initial DDIM latent z~tarinv . Finally, this latent is used for sampling by a pre-trained diffusion model to synthesize the novel view image.
Figure 3: Mean of the DDIM inverted latent. Top: t=400,600 . Bottom: t=800 and original image.
Figure 4: Qualitative results with our 167-class trained model.
Figure 5: Qualitative results with our 3-class trained model on MVImgNet test set.
Table 1: (a) Full-trajectory curvature Wasserstein distance to the ground-truth target diffusion trajectory (lower is better). (b) Bin-wise curvature analysis. FD is frame distance
Figure 6: (a) View-dependent geometric details absent in the mean prediction but re-synthesized in the final result. (b) Sampling trajectories under different initialization strategies. Row: (i) our diffusion-dynamics aligned latent z~tarinv , (ii) ground-truth target-view latent ztarinv (equation 5 ), and (iii) direct noise addition from the reference latent zfused (equation 3 ).
Table 2: (a) Comparison for 3 classes - chairs, sofa, tables. Resolution is 256×256 . (b) Comparison for 167 classes. Resolution is 90×90 .
Figure 7: (a) Comparison with NViST on unseen test scenes from five classes. (b) Multi-view consistency across a 30-frame sweep from a single input image.
Figure 8: Results on unseen classes of MVImgNet and OOD images
Table 3: (a) Ablation study using 3-class model. (b) Analysis of fusion strategies
t
LPIPS ↓
PSNR ↑
SSIM ↑
FID ↓
400
0.510
15.37
0.528
150.28
600
0.491
15.41
0.522
93.57
800
0.550
15.01
0.502
197.13
Table 4: Effect of diffusion timestep t .
Pipeline
LPIPS ↓
PSNR ↑
SSIM ↑
FID ↓
Stable Diffusion v1.5
0.491
15.41
0.522
93.57
Stable Diffusion 2.1
0.491
15.27
0.522
88.95
Dreamlike Photoreal 2.0
0.507
15.15
0.503
101.84
Table 5: Comparison across different SD pipelines.
Frame Distance
N
LPIPS ↓
PSNR ↑
SSIM ↑
0–5
196
0.4741±0.0455
16.28±1.92
0.543±0.19
6–10
3375
0.4863±0.0483
15.82±2.01
0.528±0.17
11–16
8811
0.4914±0.0499
15.63±2.06
0.525±0.17
17–21
4980
0.4949±0.0508
15.55±2.11
0.517±0.17
21–25
996
0.4996±0.0517
15.44±2.14
0.513±0.17
Table 6: Performance as a function of source–target frame distance. Metrics are reported as mean ± standard deviation across all test pairs in each bucket.
Mid-range (30–60 frames)
Long-range (60–120 frames)
Method
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
VIVID
0.523
13.83
0.439
0.594
12.69
0.410
Ours
0.503
15.04
0.479
0.609
13.44
0.448
Table 7: Results on 1K pairs of RealEstate10K. Images are uniformly sampled at random from different scenes.
Figure 9: We use textual prompts to edit the novel view reconstructions as desired. In the first row, we place a bag on the bench. In the second row, a dog is introduced in the scene. In the third row, a pair of shoes are placed by the bench.
Figure 10: MVImgNet failure cases. Each column represents one scene, while rows correspond to input view, predicted novel view, and target view.
Method
LPIPS ↓
PSNR ↑
SSIM ↑
FID ↓
Ours
0.556
16.162
0.578
69.604
Table S1: Our results at original resolution of 512×512
Figure S1 : Qualitative ablation results.
Figure S2 : Additional Qualitative results with our 167-class trained model on MVImgNet test set.
We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) methods face a fundamental trade-off: reconstruction-based approaches preserve observed geometry but struggle to synthesize unseen regions, while diffusion-based methods provide strong generative priors yet rely on implicit source-to-query correspondence. VGGT-Diff bridges these regimes by routing visual geometry latents from VGGT-Ω into a pretrained video diffusion model. Each visual token is associated with a 3D point and confidence, then transformed into query-aligned latent conditions through a confidence-aware Visual Geometry Router (VGR) that preserves front and back surface evidence. These conditions guide joint target-view denoising, while Point-Track Residual Consistency (PTRC) regularizes predicted-clean residuals along reliable 3D tracks, improving multi-view stability. We further introduce robust geometry conditioning, combining training-time regularization with inference-time guidance for improved robustness. Experiments show competitive or state-of-the-art performance across interpolation and extrapolation under different viewpoint difficulties. Our code is available at https://github.com/chenkangjie1123/VGGT-Diff.
Kangjie Chen, Xiangyu Li, Dongbin Zhang +12
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University
With the recent surge of generative models, diffusion-based approaches have become mainstream for view synthesis tasks, either in an explicit depth-warp-inpaint or in an implicit end-to-end manner. Despite their success, both paradigms often suffer from noticeable quality degradation, e.g., blurred details and distorted structures, caused by pixel-to-latent compression and diffusion hallucination. In this paper, we investigate diffusion degradation from three key dimensions (i.e., spatial, temporal, and backbone-related) and propose UniFixer, a universal reference-guided framework that fixes diverse degradation artifacts via a coarse-to-fine strategy. Specifically, a reference pre-alignment module is first designed to perform coarse alignment between the reference view and the degraded novel view. A global structure anchoring mechanism then rectifies geometric distortions to ensure structural fidelity, followed by a local detail injection module that recovers fine-grained texture details for high-quality view synthesis. Our UniFixer serves as a plug-and-play refiner that achieves zero-shot fixing across different types of diffusion degradation, and extensive experiments verify our state-of-the-art performance on novel view synthesis and stereo conversion.
Sihan Chen, Xiang Zhang, Yang Zhang +2
ETH Zürich, Switzerland · DisneyResearch|Studios, Switzerland
We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.
Yuci Han, Charles Toth, John E. Anderson +2
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL