GenRec: Knowing Where to Reconstruct and Where to Generate

Ata Çelen; Jaewoo Jung; Federico Tombari; Marc Pollefeys; Sunghwan Hong; Michael Niemeyer; Daniel Barath

cs.CVAug 18, 2026

GenRec: Knowing Where to Reconstruct and Where to Generate

Authors: Ata Çelen, Jaewoo Jung, Federico Tombari, Marc Pollefeys, Sunghwan Hong, Michael Niemeyer, Daniel Barath

Organizations: ETH Zürich · KAIST · Google · Microsoft

Abstract

Generative novel view synthesis from sparse input images is rarely all reconstruction or all generation: pixels visible in some source view have a unique correct value modulated only by view-dependent shading, while pixels in disocclusions or beyond the captured volume admit a distribution of plausible completions. Existing generative novel-view-synthesis methods conflate these regimes under a single uniform loss, blurring the line between geometric fidelity and creative hallucinations even when scene geometry is injected through warped point clouds or projected depth. We introduce GenRec, a multi-view flow matching model that builds the reconstruction--generation split directly into its architecture, supervision, and gradient flow. Guided by an observation mask derived from the source cameras and a monocular depth estimator, a flow matching backbone jointly denoises RGB and scene-coordinate maps across all target views, while a pixel-space refinement stage restores high-frequency detail on observed pixels; the same mask gates supervision so regression signals do not contaminate the generative prior. Across RealEstate10K, DL3DV-10K, and Mip-NeRF~360, in both single-view extrapolation and two-view interpolation, GenRec attains the best reconstruction fidelity in observed regions while also surpassing purely generative baselines on perceptual quality in unobserved ones, showing the effectiveness of our approach.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of GenRec (right) vs. prior generative reconstruction methods (left). Prior methods condition multi-view image or video models on warped point clouds or Plücker rays, supervising output pixels with one generative loss regardless of source coverage. GenRec factors the task by an observation mask: a generative backbone predicts every pixel, and a pixel-space reconstruction module refines the observed ones under a dedicated loss. The two pathways share neither parameters nor gradients, preserving the generative prior in unobserved regions.

Figure 2: Overview of GenRec . Given posed input views and target poses, monocular depth and forward warping produce per-target observation masks and warped renders. These condition a (I) multi-view flow matching backbone , which jointly denoises RGB and scene-coordinate latents through cross-view and cross-modal attention. A (II) reconstruction branch then refines only the observed pixels via sparse 3D attention guided by the predicted scene coordinates, while the observation mask gates gradients so the generative prior is preserved in unobserved regions.

	Global				Observed		Unobserved
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	Inf. Time
CameraCtrl [ 13 ]	13.40	0.4424	0.4761	57.42	15.53	0.4568	12.40	0.3935	$\sim$ 12s
ViewCrafter [ 56 ]	13.29	0.4792	0.4293	46.27	15.93	0.5196	11.83	0.3937	$\sim$ 200s
SEVA [ 59 ]	13.50	0.4839	0.4674	41.64	15.97	0.4999	12.40	0.4189	$\sim$ 20s
GF [ 47 ]	13.11	0.3806	0.4945	54.58	15.56	0.3986	11.96	0.3125	$\sim$ 40s
LVSM [ 18 ]	13.81	0.4094	0.5186	57.89	16.32	0.4224	12.61	0.3435	$\sim$ 1s

Table 1: Quantitative results on RealEstate10K in the single-view extrapolation setting. Global metrics are computed over the full target frame; Observed restricts them to pixels that receive source evidence under the shared observation mask, and Unobserved to pixels that do not. Inference time is reported per scene. Best results in bold .

	Global				Observed		Unobserved
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	Inf. Time
CameraCtrl [ 13 ]	13.26	0.3346	0.4893	69.49	13.45	0.3418	12.76	0.2957	$\sim$ 12s
ViewCrafter [ 56 ]	17.57	0.5302	0.3092	46.36	18.75	0.5659	15.36	0.4282	$\sim$ 200s
SEVA [ 59 ]	14.58	0.4300	0.4359	43.51	14.86	0.4401	13.92	0.3819	$\sim$ 20s
GLD [ 17 ]	15.54	0.4001	0.3979	51.85	16.10	0.4184	14.33	0.3368	$\sim$ 42s
Gen3R [ 16 ]	13.74	0.3754	0.5173	130.10	14.02	0.3886	12.95	0.3048	$\sim$ 90s

Table 2: Quantitative Results on DL3DV-10K. We compare our method with previous works on single-view conditioned generation. We follow the same evaluation protocol as in Table 1 .

	Global				Observed		Unobserved
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	Inf. Time
CameraCtrl [ 13 ]	10.89	0.1870	0.6534	214.2	11.31	0.1869	10.67	0.1821	$\sim$ 12s
ViewCrafter [ 56 ]	11.75	0.2197	0.5694	172.5	14.46	0.2510	10.74	0.1913	$\sim$ 200s
SEVA [ 59 ]	11.52	0.2231	0.5857	139.8	12.09	0.2315	11.30	0.2171	$\sim$ 20s
GLD [ 17 ]	12.03	0.1986	0.5631	162.1	13.32	0.2044	11.45	0.1900	$\sim$ 42s
Gen3R [ 16 ]	11.07	0.2056	0.6564	223.6	11.21	0.2072	10.96	0.2015	$\sim$ 90s

Table 3: Quantitative results on Mip-NeRF 360. Out-of-distribution comparison with prior methods in the single-view extrapolation setting. We follow the same evaluation protocol as in Table 1 .

Figure 3: Comparative qualitative results. Single-view extrapolation against our strongest baseline, Gen3C [ 34 ] . Each row shows the input view, ground-truth target, GenRec , and Gen3C.

	RE10K			DL3DV-10K
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
CameraCtrl [ 13 ]	12.87	0.4271	0.5022	15.60	0.3456	0.4705
ViewCrafter [ 56 ]	12.61	0.4550	0.4573	17.38	0.5214	0.3087
SEVA [ 59 ]	16.15	0.5833	0.3477	21.38	0.6906	0.1845
LVSM [ 18 ]	16.88	0.5493	0.3983	21.31	0.6801	0.2223
GLD [ 17 ]	17.64	0.5707	0.2900	19.49	0.5575	0.2376

Table 4: Quantitative results on RE10K and DL3DV-10K. Comparison with prior methods in the two-view interpolation setting.

		Global				Observed
	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
RE10K	Generative Backbone	15.86	0.5483	0.3402	37.82	19.20	0.6139
	+ VAE Skip & LoRA	15.86	0.5591	0.3394	35.27	19.23	0.6152
	+ Reconstruction Branch	17.05	0.6071	0.3220	33.09	20.85	0.6738
DL3DV	Generative Backbone	18.62	0.5796	0.2556	39.94	19.96	0.6163
	+ VAE Skip & LoRA	18.64	0.5828	0.2512	36.90	20.02	0.6215

Table 5: Ablation study on the effectiveness of individual components on RE10K and DL3DV. Starting from a vanilla multi-view flow matching baseline, we progressively add the VAE skip connections + LoRA weights, and the reconstruction branch.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Qualitative comparison on Mip-NeRF 360 [ 2 ] . Each row shows a held-out target view rendered from a single reference, compared against GEN3C. Mip-NeRF 360’s unbounded outdoor scenes and wide-baseline trajectories stress geometric consistency in the unobserved background; our model preserves thin structures and produces sharper far-field detail than the baselines.

Figure 5: Qualitative comparison on RealEstate10K [ 60 ] . Indoor real-estate walkthroughs feature predominantly forward camera motion and narrow baselines. We highlight cases with disocclusions of structured surfaces (walls, doorways, furniture edges); our method recovers plausible texture in unobserved regions while baselines exhibit blur or ghosting along occlusion boundaries.

Figure 6: Qualitative comparison on DL3DV-10K [ 23 ] . DL3DV-10K provides diverse, wide-baseline captures of indoor and outdoor scenes. The larger camera motions amplify the gap between methods that only inpaint warped pixels and methods that synthesize globally consistent geometry; our outputs remain sharp and view-consistent under aggressive target poses.

				Global				Observed		Unobserved
	Rec.	Gate	BP	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
(a)	$\times$	✓	–	16.51	0.5779	0.3459	36.84	19.88	0.6535	13.96	0.4667
(b)	✓	$\times$	$\times$	16.97	0.6020	0.3433	40.43	20.55	0.6724	14.28	0.4940
(c)	✓	✓	✓	16.99	0.6067	0.3429	34.92	20.77	0.6732	14.13	0.4738
(d)	✓	✓	$\times$	17.05	0.6071	0.3220	33.09	20.85	0.6738	14.18	0.4743

Appendix

Table 6: Ablation of the generation–reconstruction split on RealEstate10K, single-view extrapolation. Rec. : dedicated pixel-space reconstruction branch; in (a) the branch is removed and the mask-gated regression loss is applied to the backbone directly. Gate : mask-gated supervision, i.e., flow-matching supervision on all pixels with an additional regression supervision on observed ones. BP : reconstruction gradients back-propagated into the backbone through the last

K{=}1

sampling steps, following Clark et al. [6] . Rows (b)–(d) share the same architecture, parameter count, and training procedure. Best results in bold .

#Views	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
2	Gen3C [ 34 ]	15.70	0.5632	0.3282
2	GenRec	19.80	0.7043	0.2205
4	Gen3C [ 34 ]	20.80	0.7596	0.1860
4	GenRec (OOD)	23.68	0.7988	0.1387
6	Gen3C [ 34 ]	20.86	0.7656	0.1793
6	GenRec (OOD)	24.07	0.8061	0.1342

Appendix

Table 7: Multi-view interpolation on RE10K. GenRec is trained with at most two input views, so the 4- and 6-view results are zero-shot (marked OOD ), whereas Gen3C’s 3D cache ingests additional views natively. Best result per view count in bold .

	RE10K			DL3DV-10K
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
3DGS [ 20 ]	14.95	0.543	0.421	19.76	0.681	0.248
Difix3D+ [ 48 ]	15.06	0.543	0.387	19.27	0.655	0.250
pixelSplat [ 4 ]	16.96	0.614	0.392	19.29	0.625	0.328
MVSplat [ 5 ]	16.64	0.592	0.383	17.53	0.535	0.366
DepthSplat [ 52 ]	19.92	0.723	0.277	19.79	0.673	0.253

Appendix

Table 8: Comparison with reconstruction methods in the two-view interpolation setting on RE10K and DL3DV-10K. The baselines are specialised, deterministic reconstruction models; GenRec is generative. Best results in bold .

	RE10K			DL3DV
Method	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$	Inf. Time
GLD [ 17 ]	0.131	0.158	0.837	0.030	0.041	0.395	$\sim$ 42s
ViewCrafter [ 56 ]	0.134	0.148	1.446	0.045	0.060	0.535	$\sim$ 200s
Gen3R [ 16 ]	0.134	0.162	1.124	0.370	0.526	4.900	$\sim$ 90s
Gen3C [ 34 ]	0.085	0.083	0.809	0.037	0.052	0.430	$\sim$ 1200s
Ours	0.087	0.106	1.048	0.021	0.029	0.252	$\sim$ 11s

Appendix

Table 9: Camera pose accuracy and multi-view consistency on RE10K and DL3DV-10K . ATE and RPE t are reported in the scale of the dataset camera poses; RPE r in degrees. Inference time is reported per scene. Lower is better for all metrics.

Panel of Figure 7 — Figure 7: PSNR as a function of per-scene observation percentage on (a) RealEstate10K and (b) DL3DV-10K in the single-view conditioned setting. Each point is one test scene; lines are per-method linear fits. GenRec (red) sits above every baseline across the full range, from generation-heavy scenes on the left to near-reconstruction scenes on the right.

	RE10K				DL3DV-10K
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$
RGB	15.09	0.5384	0.3602	38.22	18.05	0.5681	0.2631	38.53
RGB + SC (Ours)	15.39	0.5457	0.3528	37.13	18.31	0.5749	0.2575	37.41

Appendix

Table 10: Ablation on multi-modal generation. We compare our joint RGB and scene-coordinate (SC) backbone against an RGB-only variant trained from the same initialization for 20K steps, evaluated in the single-view conditioned extrapolation setting.

	Global				Observed		Unobserved
Depth	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
DA3 † [ 22 ]	17.35	0.6169	0.3206	63.69	22.34	0.6736	15.26	0.5242
MapAnything [ 19 ]	17.23	0.6151	0.3236	63.66	22.08	0.6717	15.15	0.5192
VGGT [ 45 ]	16.43	0.5734	0.3478	67.48	20.54	0.6092	14.79	0.4952

Appendix

Table 11: Depth source at inference on RealEstate10K, single-view extrapolation. GenRec is trained on DA3 renders (

\dagger

); the estimator is swapped at test time with no retraining. Evaluated on the same RealEstate10K subset as Table 12 , so values are not comparable to Table 1 . Best results in bold .

		Global				Observed		Unobserved
Corruption	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$
Clean	GenRec	17.35	0.6196	0.3201	63.12	22.36	0.6759	15.25	0.5263
	Gen3C [ 34 ]	15.44	0.5613	0.3627	67.04	20.04	0.6037	13.81	0.4893
	ViewCrafter [ 56 ]	13.01	0.4749	0.4505	85.04	17.27	0.5222	11.43	0.3875
Affine	GenRec	15.08	0.5122	0.4101	67.08	18.55	0.5232	14.09	0.4691
	Gen3C [ 34 ]	14.64	0.5240	0.4071	70.91	18.30	0.5404	13.56	0.4736

Appendix

Table 12: Depth-robustness sweep on RealEstate10K (25-scene subset, so clean values are not comparable to Tables 1 – 3 ). At inference, the source depth is corrupted after the Umeyama alignment that fixes scale; every method builds its own conditioning from the corrupted depth, and all runs are scored against a single frozen observation mask. Each entry averages over the severity levels of the corresponding family: three levels for the affine corruptions, one for boundary, and three for noise. Best result per block in bold .

Explore similar work

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space
Sep 28, 2026Kerui Ren, Tao Lu, Linning Xu +5Multi-View Consistency Novel View Synthesis
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3
MetaView: Monocular Novel View Synthesis with Scale-Aware Implicit Geometry Priors
Jul 13, 2026Yufei Cai, Xuesong Niu, Hao Lu +3Multi-View Consistency Novel View Synthesis
Nanyang Technological University · Kolors Team, Kuaishou Technology · The Hong Kong University of Science and Technology (Guangzhou)
Reflection-aware Generative Novel View Synthesis
Sep 4, 2026GeonU Kim, Shin Dong-Yeon, Tae-Hyun OhNovel View Synthesis Diffusion Models
KAIST

Sep 28, 2026J/K move · Enter open · S save

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

Kerui Ren, Tao Lu, Linning Xu, Changjian Jiang+4

Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong+3

Multi-View Consistency Novel View Synthesis Sparse-View 3D Reconstruction 3D Diffusion Models

Open paper AskPDF arXiv

Novel view synthesis from sparse images must reconcile faithful reconstruction of observed regions with plausible completion of unseen content, while maintaining world consistency across viewpoints. Existing geometry-based methods preserve observed scene structure but often struggle to complete unseen regions, whereas video generative models offer rich appearance priors but accumulate inconsistencies during sequential view generation. We propose GeoVerse, a framework that synthesizes world-consistent novel views by performing generation within the geometric latent space of a pretrained 3D foundation model and injecting appearance priors from a video generative model. Specifically, GeoVerse extracts multilevel features from Wan2.2 VACE and injects them into the geometric latent diffusion model via a ControlNet-style adapter, incorporating video-learned appearance priors to enhance structural completion. To enforce cross-view coherence, a global spatial memory continuously aggregates observed and synthesized content, reprojecting target-aligned guidance to anchor subsequent predictions to a shared scene representation. Extensive experiments across diverse datasets demonstrate improved visual quality and geometric consistency, with a 2.23 dB higher PSNR on DL3DV and 32.4% lower ATE on Mip-NeRF360 compared to GLD.

Abstract

Explore similar work

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space

MetaView: Monocular Novel View Synthesis with Scale-Aware Implicit Geometry Priors

Reflection-aware Generative Novel View Synthesis