SemanticNVS: Improving Semantic Scene Understanding in Generative Novel View Synthesis

Xinya Chen; Christopher Wewer; Jiahao Xie; Xinting Hu; Jan Eric Lenssen

cs.CVFeb 23, 2026

SemanticNVS: Improving Semantic Scene Understanding in Generative Novel View Synthesis

Authors: Xinya Chen, Christopher Wewer, Jiahao Xie, Xinting Hu, Jan Eric Lenssen

Organizations: Max Planck Institute for Informatics, Saarland Informatics Campus, Germany

Abstract

We present SemanticNVS, a camera-conditioned multi-view diffusion model for novel view synthesis (NVS), which improves generation quality and consistency by integrating pre-trained semantic feature extractors. Existing NVS methods perform well for views near the input view, however, they tend to generate semantically implausible and distorted images under long-range camera motion, revealing severe degradation. We speculate that this degradation is due to current models failing to fully understand their conditioning or intermediate generated scene content. Here, we propose to integrate pre-trained semantic feature extractors to incorporate stronger scene semantics as conditioning to achieve high-quality generation even at distant viewpoints. We investigate two different strategies, (1) warped semantic features and (2) an alternating scheme of understanding and generation at each denoising step. Experimental results on multiple datasets demonstrate the clear qualitative and quantitative (4.69%-15.26% in FID) improvement over state-of-the-art alternatives.

Figures & tables

Figure 2: Method Overview. SemanticNVS integrates semantic DINO features into the multi-view diffusion setup in two different ways. First, it provides warped features from the given input view and uses them as additional conditioning. Second, it extracts DINO features from intermediate generations $\hat{x}^{t}_{0}$ of the previous iteration and uses them to complete the warped DINO features.

Figure 3: Qualitative Comparison on RealEstate10K. ViewCrafter and Uni3C struggle to follow the long camera trajectory accurately. SEVA produces degraded views when moving far from the input. In contrast, our method better adheres to the target trajectory, generates more realistic novel views, and yields a more coherent underlying geometry when reconstructing the 3D scene from the generated frames. More results can be found in the appendix and the project page .

Figure 4: Qualitative Comparison on Tanks-and-Temples. ViewCrafter, Uni3C, and SEVA fail to follow the long camera trajectory and produce unrealistic or degraded views. In contrast, our method better adheres to the target trajectory, generates more realistic novel views, and yields a more coherent underlying geometry when reconstructing from the generated frames.

Figure 5: Qualitative Ablation on RealEstate10K. Warped RGB fails to generate the floor-to-ceiling glass window on the right side of the input view. Adding Warped DINO produces a clear window, but the chair remains incomplete. Further guiding sampling with intermediate-sample features (Iterative DINO) enables view-consistent synthesis of the entire scene.

Method	Short Trajectory (80 $\sim$ 100)									Long Trajectory ( $\geq$ 250)
	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE $\downarrow$	TE $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE $\downarrow$	TE $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
ViewCrafter	32.929	58.499	0.387	0.329	0.030	0.075	15.000	0.559	0.356	37.703	58.599	0.597	4.612	0.339	0.162	11.021	0.458	0.595
Uni3C	25.308	55.246	0.335	0.358	0.028	0.069	16.208	0.599	0.323	39.572	49.984	0.533	3.434	0.197	0.151	12.652	0.521	0.564
SEVA	26.376	59.145	0.357	0.202	0.014	0.067	15.517	0.579	0.357	31.344	57.971	0.511	0.535	0.055	0.110	12.871	0.517	0.519
Ours	22.726	62.060	0.251	0.189	0.019	0.067	18.024	0.646	0.251	26.561	66.458	0.364	0.468	0.054	0.109	13.817	0.542	0.450

Table 1: Quantitative Comparison on RealEstate10K. SemanticNVS achieves the best overall performance across a variety of metrics, showing significant improvements in generation quality, geometric consistency, and reconstruction quality.

Method	Short Trajectory (80 $\sim$ 100)									Long Trajectory ( $\geq$ 250)
	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE $\downarrow$	TE $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE $\downarrow$	TE $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
ViewCrafter	38.587	71.029	0.215	0.901	0.008	0.104	14.557	0.427	0.364	57.998	68.867	0.380	5.001	0.027	0.187	12.137	0.352	0.510
Uni3C	36.166	61.491	0.366	0.401	0.006	0.088	15.560	0.464	0.356	57.873	55.189	0.473	2.564	0.020	0.152	13.402	0.409	0.505
SEVA	44.749	68.344	0.340	0.381	0.002	0.098	12.978	0.400	0.474	46.564	66.258	0.436	0.640	0.005	0.127	12.196	0.373	0.531
Ours	30.749	70.985	0.219	0.197	0.003	0.085	16.240	0.495	0.275	44.381	69.829	0.266	0.577	0.008	0.119	13.825	0.426	0.405

Table 2: Quantitative Comparison on Tanks-and-Temples. While SEVA performs significantly worse on scenes out of training distribution, SemanticNVS is more robust and achieves the best overall performance on Tanks-and-Temples.

Method	Full Long Trajectory ( $\geq$ 250)
	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE ( ∘ ) $\downarrow$	TE (m) $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SEVA	25.347	57.816	0.554	0.297	0.035	0.070	13.310	0.529	0.502
Ours	21.842	64.463	0.359	0.310	0.032	0.070	14.155	0.553	0.443

Table 3: Full Trajectory Comparison on RealEstate10K. Applying the two-stage anchor generation and view interpolation for both methods, we beat SEVA in almost all metrics.

Method	Long Trajectory ( $\geq$ 250)
	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE $\downarrow$	TE $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SEVA	31.344	57.971	0.511	0.535	0.055	0.110	12.871	0.517	0.519
SEVA finetuned	31.896	56.794	0.603	0.539	0.076	0.114	12.609	0.515	0.556
+ Warped RGB	29.942	59.201	0.499	0.518	0.063	0.124	13.790	0.529	0.465
+ Warped DINO	28.908	60.452	0.424	0.506	0.052	0.111	13.809	0.538	0.452
+ Iterative RGB	28.260	62.820	0.399	0.472	0.056	0.110	13.815	0.539	0.451

Table 4: Ablation on Core Components on RealEstate10K. Warped and Iterative DINO both individually improve the overall performance. Using DINO features of intermediate samples (Iterative DINO) outperforms the equivalent Iterative RGB variant.

Method	Long Trajectory ( $\geq$ 250)
	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE $\downarrow$	TE $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DINOv3	29.058	64.590	0.416	0.458	0.055	0.112	13.544	0.549	0.456
VGGT	28.704	61.781	0.407	0.538	0.059	0.109	13.608	0.540	0.452
DINOv2	26.561	66.458	0.364	0.468	0.054	0.109	13.817	0.542	0.450

Table 5: Feature Ablation on RealEstate10K. DINOv2 achieves the best overall performance among different foundation models.

Method	Long Trajectory ( $\geq$ 250)
	FID $\downarrow$	ImQ $\uparrow$	Drift $\downarrow$	RE $\downarrow$	TE $\downarrow$	MEt3R $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
SEVA finetuned	31.896	56.794	0.603	0.539	0.076	0.114	12.609	0.515	0.556
+ REPA	30.579	63.231	0.384	0.721	0.067	0.116	12.787	0.503	0.538
+ Ours	26.561	66.458	0.364	0.468	0.054	0.109	13.817	0.542	0.450

Table 6: Comparison with REPA on RealEstate10K. While both making use of DINO as semantic prior, SemanticNVS outperforms REPA for generative NVS.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Qualitative Comparison on RealEstate10K.

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 282 papers

BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model
Feb 26, 2026Yuci Han, Charles Toth, John E. Anderson +2Novel View Synthesis Multi-View Image Synthesis
Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL
Reflection-aware Generative Novel View Synthesis
Sep 4, 2026GeonU Kim, Shin Dong-Yeon, Tae-Hyun OhMulti-View Diffusion Models Novel View Synthesis
KAIST
Global Pose Control for Generative View Synthesis in Normalized Object Coordinate Space
Jul 2, 2026Zhibing Li, Amogh Gupta, Behnoosh Parsa +1Novel View Synthesis Controllable Image Generation
The Chinese University of Hong Kong, China · Amazon, USA

Feb 26, 2026J/K move · Enter open · S save

BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model

Yuci Han, Charles Toth, John E. Anderson, William J. Shuart+1

Dept. of Electrical and Computer Engineering, The Ohio State University · USACE ERDC GRL

Novel View Synthesis Multi-View Image Synthesis Sparse-View 3D Reconstruction Video Diffusion Models 3D Reconstruction

Open paper AskPDF arXiv

We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.

Abstract

Novel View Synthesis

Explore similar work

BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model

Reflection-aware Generative Novel View Synthesis

Global Pose Control for Generative View Synthesis in Normalized Object Coordinate Space