Aligning Misaligned Benchmarks for Single-View Novel View Synthesis

Yang-Che Tseng; Erqun Dong; Jeong-gi Kwak; Gopal Sharma; Shakiba Kheradmand; Stephan Garbin; Hossam Isack; Kwang Moo Yi

cs.CVOct 28, 2026

Aligning Misaligned Benchmarks for Single-View Novel View Synthesis

Authors: Yang-Che Tseng, Erqun Dong, Jeong-gi Kwak, Gopal Sharma, Shakiba Kheradmand, Stephan Garbin, Hossam Isack, Kwang Moo Yi

Organizations: University of British Columbia · Google

Abstract

While single-view novel view synthesis methods have been dramatically enhanced recently, how to quantify their performance has yet to be standardized. In this work, we demonstrate that an important factor lies in how each method's own reference frame is aligned with the so-called ground-truth pose. Existing methods often rely on test-time optimization of target poses for this alignment, which inherently is ill-posed due to the complexity of the optimization landscape of camera poses vs render accuracy, leading to methods being evaluated at an arbitrary local optimum. Some recognize this limitation and thus rely on 3D reconstructions of entire scenes using training and test views to align poses, which may be infeasible. In this work, instead, we propose to use traditional 2D-to-2D and 2D-to-3D image matching techniques to align 3D reconstructions. We also evaluate only within co-visible regions, between different views, which is the only place where a `correct' ground truth can be defined. Our evaluations show that with proper coordinate alignment, novel-view synthesis methods that use explicit 3D Gaussians achieve the state of the art, despite being previously reported in the literature to be outperformed by pure image-based 2D methods. We further provide a simple baseline that directly utilizes an off-the-shelf video generator and a feed-forward geometry estimator. Our simple baseline is already comparable to state-of-the-art methods specifically trained for novel-view synthesis, without any task-specific training.

Figures & tables

Figure 1: – Evaluation can be misleading due to inaccurate test-time pose alignment – We visualize the error map with the conventional test-time optimization and our proposed protocol (red is higher error, blue is lower). With the same 3D scene reconstructed from the reference view, conventional test-time optimization can leave residual misalignment between the rendered and ground-truth views, leading to large quantitative error. In contrast, our evaluation protocol provides a more robust alignment strategy, enabling more reliable performance measurement.

Panel of Figure 2 — Figure 2: Extracting co-visible regions – We show example co-visible regions extracted by our protocol. We extract by projecting each pixel back and forth via DUSt3R [ 44 ] and checking their relative depths. We make a conservative estimate of co-visible regions to not disadvantage methods that do not extrapolate pixels outside the input view.

Panel of Figure 3 — Figure 3: Examples of misalignment leading to inaccurate evaluation – We show examples with Flash3D [ 38 ] for RealEstate10K (scene 01a5cc , fr. 50) and DTU (scan103, fr. 12), with test-time optimization only, and our proposed protocol. As shown, optimization alone leaves a large pose offset. Combined with the fact that many methods do not mask via co-visibility [ 6 , 48 , 51 , 5 , 18 ] , the PSNR values can be very low, up to a level where it effectively is at a level where two random images are being compared ( $<10$ dB). With our image-matching-based pre-alignment, on the other hand, images are better aligned, and reasonable PSNR values are extracted. The yellow tint marks the non-co-visible region that our protocol excludes from computing the metrics.

	Method	RealEstate10K [ 54 ] (SV)			RealEstate10K [ 54 ] (VC)			DL3DV-10K [ 29 ] (SV)			DTU [ 15 ]			Tank & Temples [ 22 ] (VC)			Average
	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Test-time optim. only	Ours	18.85	0.714	0.402	16.03	0.666	0.489	12.74	0.341	0.682	11.83	0.397	0.647	14.30	0.454	0.566	14.75	0.514	0.557
	Flash3D [ 38 ]	14.52	0.578	0.504	14.75	0.630	0.499	11.71	0.300	0.683	11.12	0.402	0.689	13.75	0.455	0.574	13.17	0.473	0.590
	DepthSplat [ 48 ]	16.66	0.612	0.409	16.45	0.633	0.421	12.77	0.300	0.644	10.67	0.346	0.623	13.59	0.441	0.534	14.03	0.466	0.526
	Lyra [ 3 ]	17.50	0.653	0.423	16.20	0.662	0.454	12.34	0.295	0.627	12.68	0.432	0.535	15.38	0.478	0.482	14.82	0.504	0.504
	AnySplat [ 17 ]	19.40	0.734	0.330	19.03	0.739	0.370	13.54	0.323	0.617	12.82	0.416	0.523	14.81	0.465	0.508	15.92	0.535	0.470

Table 1: Quantitative results – We summarize the Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM) [ 45 ] , and Learned Perceptual Image Patch Similarity (LPIPS) [ 52 ] for each method on our benchmark datasets, with the mean over all datasets reported on the right. Within each group, best results are in bold and second best are underlined . Using test-time optimization only reports wildly different results compared to when our evaluation protocol is used: Flash3D [ 38 ] is among the worst performing methods with a naive evaluation but, with better alignment, performs better than DepthSplat, while SHARP [ 32 ] leads the test-time-optimization-only group on average—its Gaussians are regressed directly in the input-camera frame, so it needs little alignment to begin with.

	Method	RealEstate10K [ 54 ] (SV)			RealEstate10K [ 54 ] (VC)			DL3DV-10K [ 29 ] (SV)			DTU [ 15 ]			Tank & Temples [ 22 ] (VC)			Average
	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Test-time optim. only	Ours	20.73	0.744	0.315	16.08	0.538	0.462	13.04	0.260	0.668	11.82	0.340	0.652	14.09	0.364	0.536	15.15	0.449	0.527
	LVSM [ 18 ]	19.76	0.690	0.283	19.54	0.672	0.270	13.37	0.266	0.632	12.58	0.372	0.602	16.16	0.443	0.440	16.28	0.489	0.445
	MVSplat [ 6 ]	15.51	0.589	0.401	14.54	0.535	0.404	11.91	0.227	0.657	10.34	0.275	0.643	12.41	0.351	0.563	12.94	0.395	0.534
	DepthSplat [ 48 ]	17.47	0.656	0.360	16.10	0.602	0.378	12.39	0.247	0.648	12.01	0.369	0.539	13.49	0.399	0.517	14.29	0.455	0.488
	NoPoSplat [ 49 ]	13.42	0.466	0.538	12.29	0.404	0.584	11.31	0.249	0.740	9.41	0.327	0.801	11.51	0.295	0.623	11.59	0.348	0.657

Table 2: Quantitative results (256

\times

256 square evaluation) – PSNR, SSIM [ 45 ] , and LPIPS [ 52 ] for the methods that support the square (256

\times

256 centre-crop) setting, under both test-time-optimization-only and our evaluation protocol, with the mean over all datasets reported on the right. Flash3D [ 38 ] and ViewCrafter [ 51 ] are omitted as they cannot produce a comparable 256

\times

256 cell. PF3plat [ 14 ] is evaluated with its RealEstate10K checkpoint on every dataset, as are MVSplat [ 6 ] and NoPoSplat [ 49 ] . Within each group, best results are in bold and second best are underlined .

Panel of Figure 4 — Figure 4: Qualitative results – We show qualitative examples for each dataset under both evaluation resolutions, with the ground-truth target view (co-visibility overlay in blue) on the left. Middle: native-resolution methods, Flash3D [ 38 ] , ViewCrafter [ 51 ] , Lyra [ 3 ] , SHARP [ 32 ] , and our own baseline ( Tab. 1 ). Right: methods that support the $256\times 256$ centre-crop setting, LVSM [ 18 ] , PF3plat [ 14 ] , and our baseline ( Tab. 2 ). Both halves show the same scene and held-out target view for each dataset, listed above each row. All methods are shown with our conservative co-visibility overlay. With our enhanced alignment protocol, the actual qualitative performances of Flash3D [ 38 ] , ViewCrafter [ 51 ] , and our baseline align well with the quantifications in Tab. 1 . Our suggested baseline, while simple, performs on par with Flash3D [ 38 ] .

Init	Train	RE10K [ 54 ]	RE10K [ 54 ] -VC	T&T [ 22 ]	DL3DV-10K [ 29 ]	DTU [ 15 ]	Avg
input only	input only	21.34	20.11	16.32	13.61	16.56	17.59
all views	input only	22.26	21.30	16.80	14.13	17.47	18.39
input only	all views	19.89	20.17	17.24	13.46	16.24	17.40
all views	all views	19.92	20.34	17.40	13.60	16.41	17.53

Table 3: Ablation on using generated views – We ablate when to use generated views for training our 3D Gaussians. Including generated views for initialization only and training Gaussians only with the real input view gives best results.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Example views generated by video models – We show example views generated by video models.

Panel of Figure 6 — Figure 6: Qualitative effect of generated views – For each scene we show the ground truth and the full init $\times$ train factorial of Tab. 3 : rows vary the initialization (input view only vs. all generated views) and columns the training supervision. Training on generated views smears fine geometry (top scene) and hallucinates floaters in disocclusions (bottom scene); using them only for initialization stays sharpest. The teal overlay marks pixels outside the co-visible region, excluded from the metric.

Panel of Figure 7 — Figure 7: ViewCrafter [ 51 ] point-cloud conditioning vs. diffusion output. ViewCrafter [ 51 ] rasterizes its single-image point cloud from the (canonical) target pose to form the conditioning frame (middle), then a video-diffusion model synthesizes the final view (right). The raster is geometrically and photometrically faithful wherever the single view provides coverage (the speckled regions are sparse splatting where coverage thins); the diffusion model fills the remainder but can hallucinate geometry and drift in color as the viewpoint rotates away from the input. Rows: RealEstate10K (SV), RealEstate10K (VC), DL3DV-10K, DTU, Tanks & Temples (VC), on the same scenes as Fig. 4 .

Novel View Synthesis

Novel View Synthesis creates images of a scene from camera viewpoints not present in the input, using learned geometry and appearance models.

See all 295 papers

NeoMap: Training-free Novel-View Synthesis from Single Images and Videos
Jul 2, 2026Jinxi Li, Tianyi Zhang, Yafei Yang +4Multi-View Consistency Video Diffusion Models
Shenzhen Research Institute, The Hong Kong Polytechnic University · vLAR Group, The Hong Kong Polytechnic University
SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis
Aug 17, 2026Yejun Zhang, Zihan Wang, Xu Ji +8Multi-View Diffusion Models Novel View Synthesis
Aalto University, Finland · Deep Render, UK · ELLIS Institute Finland +3
OrbitNVS: Harnessing Video Diffusion Priors for Novel View Synthesis
Mar 20, 2026Jinglin Liang, Zijian Zhou, Rui Huang +2Novel View Synthesis Single-View 3D Reconstruction
South China University of Technology · Agentic Labs, XGTech · King’s College London +1

Jul 2, 2026J/K move · Enter open · S save

NeoMap: Training-free Novel-View Synthesis from Single Images and Videos

Jinxi Li, Tianyi Zhang, Yafei Yang, Zihui Zhang+3

Shenzhen Research Institute, The Hong Kong Polytechnic University · vLAR Group, The Hong Kong Polytechnic University

Multi-View Consistency Video Diffusion Models Dynamic Novel View Synthesis

Open paper AskPDF arXiv

We study the challenging problem of novel view video synthesis from single images or monocular videos. Existing methods, which operate under the assumption that pre-trained video models lack native novel view synthesis capability and enforce view alignment via camera conditioning, task-specific fine-tuning, or stepwise hard denoising guidance, often suffer from artifacts and compromised global scene consistency. In this paper, we introduce NeoMap, a novel training-free framework designed to locate high-fidelity, view-consistent novel view solutions from general pre-trained video models. The key to our approach is the core insight that promising novel view solutions are inherently encoded within the natural video data manifold learned by pre-trained models, and the core challenge is simply to locate this optimal solution. We solve this via our core mechanism: convergent manifold alternating projection iterations that optimize the initial noise. Extensive experiments demonstrate that NeoMap significantly outperforms all existing methods across 3 standard novel view synthesis benchmarks, including the challenging Tanks-and-Temples, LLFF and DAVIS datasets, achieving state-of-the-art generation fidelity and top-tier view consistency.

Abstract

Novel View Synthesis

Explore similar work

NeoMap: Training-free Novel-View Synthesis from Single Images and Videos

SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis

OrbitNVS: Harnessing Video Diffusion Priors for Novel View Synthesis