ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

Giuseppe Stracquadanio; Kevin Raj; Julia Grabinski; Stefan Roth

cs.CVAug 28, 2026

ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

Authors: Giuseppe Stracquadanio, Kevin Raj, Julia Grabinski, Stefan Roth

Organizations: TU Darmstadt · Zuse School ELIZA

Abstract

We introduce ReconSplat, a feed-forward model for 3D scene reconstruction that aims to address the longstanding trade-off between plausible view generation for unobserved regions and geometric consistency, providing both geometrically aligned novel views and sharp depth estimates. Our approach builds on 3D Gaussian splatting (3DGS) as an intermediate differentiable scene representation and integrates it with a multi-view latent diffusion model (MV-LDM) trained to act simultaneously as a refiner and an inpainter for appearance and scene geometry. We enforce geometric consistency by guiding the diffusion process with variational 3D latent features for appearance and geometry, encoded by the feed-forward 3DGS representation and rasterized to 2D latent space. ReconSplat produces both photorealistic novel views and accurate depth maps on real-world benchmarks, RealEstate10K and DL3DV-10K, outperforming existing methods in challenging extrapolation setups. Notably, ReconSplat allows the extrapolation of unseen and challenging viewpoints jointly with coherent and precise scene geometry.

Figures & tables

Panel of Figure 1 — Figure 1: Qualitative comparison of ReconSplat, our feed-forward 3D scene reconstruction model, to previous work. Previous methods need to trade-off plausible generation in unobserved regions and geometric consistency. While [ 16 , 84 ] produce plausible appearance for unobserved views, they suffer from texture inconsistencies and fail to capture the true scene geometry, resulting in noisy and fragmented depth maps. [ 86 ] produces clean depth estimates, but fails to synthesize appearance and geometry in unobserved regions. In contrast, our ReconSplat achieves both consistent appearance and accurate depth across views by learning a diffusion prior guided by multi-view-consistent variational latents for RGB and depth.

Figure 2: Overview of ReconSplat. From sparse context views $\mathcal{I}^{c}$ , we regress a 3DGS representation $\mathcal{G}$ encoding variational appearance and geometry 3D latent fields. Given target viewpoints $\mathcal{P}^{\odot}$ , we rasterize the variational distributions from 3D into 2D, allowing us to sample preliminary 2D latents $\tilde{\mathcal{Z}}^{\odot}$ , i.e . , $\tilde{\mathcal{Z}}^{\odot}\sim\mathcal{R}(\mathcal{G}\mid\mathcal{P}^{\odot})$ . These are projected via $\phi$ and refined by a multi-view diffusion model into target latents $\mathcal{Z}^{\odot}$ , encoding high-quality novel views and depth maps, conditioning on context views $\mathcal{I}^{c}$ .

Figure 3: U-Net block. We add two camera-conditioned branches: 3D self-attention for cross-view communication and optional 3D convolutions for smooth nearby-view transitions.

Panel of Figure 4 — Figure 4: Qualitative comparison on RE10K [ 100 ] for the challenging extrapolation setting. ReconSplat predicts high-quality novel views and sharp depth estimates for unobserved viewpoints given only two context views, clearly surpassing prior work.

Method	Interpolation					Extrapolation
	FID $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
Regression-based
pixelSplat [ 12 ]	$4.58$	$0.178$	$0.109$	$24.29$	$0.820$	$11.18$	$0.259$	$0.154$	$20.65$	$0.728$
MVSplat [ 15 ]	3.49	$0.161$	$0.097$	$24.12$	0.845	$9.83$	$0.245$	$0.145$	$20.49$	0.752
DepthSplat [ 86 ]	2.70	0.109	0.069	27.65	0.894	$11.81$	0.227	$0.139$	$20.71$	0.776
Generative-based

Table 1: Novel-view synthesis on RE10K. ReconSplat outperforms both regression-based and generative-based prior methods, often substantially, in the challenging extrapolation setting (except in SSIM), demonstrating the generalizability of our approach. ReconSplat also outperforms generative approaches (bottom) in the interpolation setting across all metrics (except SSIM, where all gen. methods are close).

Panel of Figure 5 — Figure 5: Qualitative comparison on DL3DV-10K [ 49 ] . ReconSplat predicts high-quality novel views and sharp depth estimates for unobserved viewpoints compared to prior work for the challenging wide-baseline setup on DL3DV-10K given only four context views.

Regression-based
Method	Two-round ( $n=300$ )					One round ( $n=150$ )
	FID $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
pixelSplat [ 12 ]	$132.79$	$0.472$	$0.287$	17.15	0.494	$108.63$	$0.424$	$0.255$	18.46	0.551
MVSplat [ 15 ]	$97.73$	$0.476$	$0.235$	$15.59$	$0.358$	$70.23$	$0.429$	$0.191$	$16.14$	$0.356$
DepthSplat* [ 86 ]	55.84	0.297	0.155	19.41	0.633	43.67	0.221	0.115	21.51	0.716
Generative-based

Table 2: Novel-view synthesis on DL3DV-10K. ReconSplat outperforms both regression-based and generative-based methods on reconstruction, perceptual, and FID metrics, demonstrating its generalizability. Note : DepthSplat* is trained on the full DL3DV-10K dataset (

\sim

\times

more data), making direct comparison less meaningful.

Table 8

Figure 6: Dense point clouds. Given a set of $N$ context views ( cyan ), ReconSplat predicts $M$ novel views and their geometry ( red ), for both observed and unobserved regions, as demonstrated by unprojecting depth and appearance into dense point clouds for DL3DV-10K (left, center) and for RE10K (right) .

Method	6 Context Views					8 Context Views
Method	FID $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
latentSplat [ 84 ]	$63.87$	0.341	0.158	18.05	0.571	(OOM)	(OOM)	(OOM)	(OOM)	(OOM)
MVSplat360 [ 16 ]	41.80	$0.403$	$0.162$	$14.98$	$0.348$	$41.11$	$0.399$	$0.161$	$14.98$	$0.348$
ReconSplat (Ours)	30.13	0.218	0.095	21.55	0.671	27.40	0.188	0.087	22.71	0.715

Table 5: Increased number of context views. On DL3DV-10K, ReconSplat consistently outperforms generative baselines when given additional inputs (

N_{\uparrow}=6,8

), despite being trained with

N_{\text{DL3DV}}=

4$$ . OOM denotes “out of memory”.

Table 11

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Qualitative comparison with LVSM [ 107 ] .

Figure 8: Influence of classifier-free guidance. Metrics over different choices of

\gamma_{\text{CFG}}

for DL3DV-10K [ 49 ] (

n=150

) and RealEstate10K [ 100 ] (Extrapolation). Fixing

\gamma_{\text{CFG}}=3

yields improvements across all metrics and datasets.

Figure 9: Influence of sampling budget. Distortion and perceptual metrics given different sampling budgets for DL3DV-10K.

Data and views
Setting	RE10K	DL3DV
Init. from	MVSplat [ 15 ]	RE10K ckpt.
Resolution	$256^{2}$	$256\times 448^{\dagger}$
Context views ( $N$ )	2	4
Target views ( $M$ )	4	4
Effective batch size ( $B_{\text{eff}}$ )	24	8

Appendix

Table 8: Stage 1 training configuration. (a) Training setup for RE10K and DL3DV, including initialization, output resolution, number of context and target views, batch size, and optimization settings. (b) Loss terms and corresponding weights for the auxiliary Gaussian rendering

\mathcal{L}_{\text{aux}}

, appearance

\mathcal{L}_{\text{a}}

, geometry

\mathcal{L}_{\text{g}}

, and latent-regularization objectives.

\dagger

The encoder’s patch shim requires both image dimensions to be multiples of

64

(patch size

16

\times

downscale

4

). DL3DV inputs are therefore center-cropped from

256\times 480

256\times 448

, with intrinsics rescaled accordingly.

\ddagger

We set the KL weight to

10^{-6}

, following Stable Diffusion / LDM

f=8

autoencoder [ 60 ] .

Optimization
Setting	Value
Precision	fp32
Learning rate	$5\!\times\!10^{-5}$
Learning rate schedule	constant
Warm-up	$1\text{K}$ steps, linear
Gradient clipping	$0.5$

Appendix

Table 9: Stage 2 configuration. (a) Shared optimization, diffusion-training, and inference settings. (b) Checkpoint-specific configurations for RE10K and DL3DV.

\dagger

The encoder’s patch shim requires both image dimensions to be multiples of

64

(patch size

16

\times

downscale

4

). DL3DV inputs are therefore center-cropped from

256\times 480

256\times 448

, with intrinsics rescaled accordingly.

\ddagger

“Only

\mathcal{T}

” denotes fine-tuning only the 3D convolution blocks, with all remaining parameters frozen.

Panel of Figure 10 — Figure 10: Extensive qualitative comparisons on RE10K. Given $N=$ 2$$ context views (leftmost column) , our ReconSplat predicts extrapolated novel views and consistent depth (third and fourth columns) for unobserved viewpoints. We show the corresponding ground-truth image for each target viewpoint (second column) . We compare our predictions with MVSplat360 [ 16 ] and latentSplat [ 84 ] (fifth to eighth columns) .

Panel of Figure 11 — Figure 11: Extensive qualitative comparisons on DL3DV-10K. Given $N=$ 4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.

Panel of Figure 12 — Figure 12: Extensive qualitative comparisons on DL3DV-10K. Given $N=$ 4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.

Panel of Figure 13 — Figure 13: Extensive qualitative comparisons on DL3DV-10K. Given $N=$ 4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.

Panel of Figure 14 — Figure 14: Extensive qualitative comparisons on DL3DV-10K. Given $N=$ 4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.

Panel of Figure 15 — Figure 15: Extensive qualitative comparisons on DL3DV-10K. Given $N=$ 4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.

Explore similar work

Generalizable Sparse-View 3D Reconstruction from Unconstrained Images
Apr 30, 2026Vinayak Gupta, Chih-Hao Lin, Shenlong Wang +23D Gaussian Splatting 3D Reconstruction
University of Maryland, College Park · University of Illinois Urbana-Champaign · Johns Hopkins University
VidSplat: Gaussian Splatting Reconstruction with Geometry-Guided Video Diffusion Priors
May 12, 2026Jimin Tang, Wenyuan Zhang, Junsheng Zhou +53D Gaussian Splatting Sparse-View 3D Reconstruction
School of Software, Tsinghua University, China · Kuaishou Technology, China · Department of Computer Science, Wayne State University, USA
GlobalSplat: Efficient Feed-Forward 3D Gaussian Splatting via Global Scene Tokens
Apr 16, 2026Roni Itkin, Noam Issachar, Yehonatan Keypur +33D Gaussian Splatting 3DGS Acceleration
The Hebrew University of Jerusalem · Westlake University

Apr 30, 2026J/K move · Enter open · S save

Generalizable Sparse-View 3D Reconstruction from Unconstrained Images

Vinayak Gupta, Chih-Hao Lin, Shenlong Wang, Anand Bhattad+1

University of Maryland, College Park · University of Illinois Urbana-Champaign · Johns Hopkins University

3D Gaussian Splatting 3D Reconstruction Sparse-View 3D Reconstruction Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Reconstructing 3D scenes from sparse, unposed images remains challenging under real-world conditions with varying illumination and transient occlusions. Existing methods rely on scene-specific optimization using appearance embeddings or dynamic masks, which requires extensive per-scene training and fails under sparse views. Moreover, evaluations on limited scenes raise questions about generalization. We present GenWildSplat, a feed-forward framework for sparse-view outdoor reconstruction that requires no per-scene optimization. Given unposed internet images, GenWildSplat predicts depth, camera parameters, and 3D Gaussians in a canonical space using learned geometric priors. An appearance adapter modulates appearance for target lighting conditions, while semantic segmentation handles transient objects. Through curriculum learning on synthetic and real data, GenWildSplat generalizes across diverse illumination and occlusion patterns. Evaluations on PhotoTourism and MegaScenes benchmark demonstrate state-of-the-art feed-forward rendering quality, achieving real-time inference without test-time optimization

Abstract

Explore similar work

Generalizable Sparse-View 3D Reconstruction from Unconstrained Images

VidSplat: Gaussian Splatting Reconstruction with Geometry-Guided Video Diffusion Priors

GlobalSplat: Efficient Feed-Forward 3D Gaussian Splatting via Global Scene Tokens