cs.CVAug 28, 2026

ReconSplat: Generalizable 3D Scene Reconstruction Beyond Observed Views

Authors: Giuseppe Stracquadanio, Kevin Raj, Julia Grabinski, Stefan Roth

Organizations: TU Darmstadt · Zuse School ELIZA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce ReconSplat, a feed-forward model for 3D scene reconstruction that aims to address the longstanding trade-off between plausible view generation for unobserved regions and geometric consistency, providing both geometrically aligned novel views and sharp depth estimates. Our approach builds on 3D Gaussian splatting (3DGS) as an intermediate differentiable scene representation and integrates it with a multi-view latent diffusion model (MV-LDM) trained to act simultaneously as a refiner and an inpainter for appearance and scene geometry. We enforce geometric consistency by guiding the diffusion process with variational 3D latent features for appearance and geometry, encoded by the feed-forward 3DGS representation and rasterized to 2D latent space. ReconSplat produces both photorealistic novel views and accurate depth maps on real-world benchmarks, RealEstate10K and DL3DV-10K, outperforming existing methods in challenging extrapolation setups. Notably, ReconSplat allows the extrapolation of unseen and challenging viewpoints jointly with coherent and precise scene geometry.

Figures & tables

Figure 1: Qualitative comparison of ReconSplat, our feed-forward 3D scene reconstruction model, to previous work. Previous methods need to trade-off plausible generation in unobserved regions and geometric consistency. While [ 16 , 84 ] produce plausible appearance for unobserved views, they suffer from texture inconsistencies and fail to capture the true scene geometry, resulting in noisy and fragmented depth maps. [ 86 ] produces clean depth estimates, but fails to synthesize appearance and geometry in unobserved regions. In contrast, our ReconSplat achieves both consistent appearance and accurate depth across views by learning a diffusion prior guided by multi-view-consistent variational latents for RGB and depth.
Figure 2: Overview of ReconSplat. From sparse context views Ic\mathcal{I}^{c} , we regress a 3DGS representation G\mathcal{G} encoding variational appearance and geometry 3D latent fields. Given target viewpoints P⊙\mathcal{P}^{\odot} , we rasterize the variational distributions from 3D into 2D, allowing us to sample preliminary 2D latents Z~⊙\tilde{\mathcal{Z}}^{\odot} , i.e . , Z~⊙∼R(G∣P⊙)\tilde{\mathcal{Z}}^{\odot}\sim\mathcal{R}(\mathcal{G}\mid\mathcal{P}^{\odot}) . These are projected via ϕ\phi and refined by a multi-view diffusion model into target latents Z⊙\mathcal{Z}^{\odot} , encoding high-quality novel views and depth maps, conditioning on context views Ic\mathcal{I}^{c} .
Figure 3: U-Net block. We add two camera-conditioned branches: 3D self-attention for cross-view communication and optional 3D convolutions for smooth nearby-view transitions.
Figure 4: Qualitative comparison on RE10K [ 100 ] for the challenging extrapolation setting. ReconSplat predicts high-quality novel views and sharp depth estimates for unobserved viewpoints given only two context views, clearly surpassing prior work.
MethodInterpolation Extrapolation
 FID ↓\downarrowLPIPS ↓\downarrowDISTS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrow FID ↓\downarrowLPIPS ↓\downarrowDISTS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrow
Regression-based
pixelSplat [ 12 ]4.584.580.1780.1780.1090.10924.2924.290.8200.820 11.1811.180.2590.2590.1540.15420.6520.650.7280.728
MVSplat [ 15 ]3.490.1610.1610.0970.09724.1224.120.845 9.839.830.2450.2450.1450.14520.4920.490.752
DepthSplat [ 86 ]2.700.1090.06927.650.894 11.8111.810.2270.1390.13920.7120.710.776
Generative-based
Table 1: Novel-view synthesis on RE10K. ReconSplat outperforms both regression-based and generative-based prior methods, often substantially, in the challenging extrapolation setting (except in SSIM), demonstrating the generalizability of our approach. ReconSplat also outperforms generative approaches (bottom) in the interpolation setting across all metrics (except SSIM, where all gen. methods are close).
Figure 5: Qualitative comparison on DL3DV-10K [ 49 ] . ReconSplat predicts high-quality novel views and sharp depth estimates for unobserved viewpoints compared to prior work for the challenging wide-baseline setup on DL3DV-10K given only four context views.
MethodTwo-round ( n=300n=300 ) One round ( n=150n=150 )
 FID ↓\downarrowLPIPS ↓\downarrowDISTS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrow FID ↓\downarrowLPIPS ↓\downarrowDISTS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrow
Regression-based
pixelSplat [ 12 ]132.79132.790.4720.4720.2870.28717.150.494 108.63108.630.4240.4240.2550.25518.460.551
MVSplat [ 15 ]97.7397.730.4760.4760.2350.23515.5915.590.3580.358 70.2370.230.4290.4290.1910.19116.1416.140.3560.356
DepthSplat* [ 86 ]55.840.2970.15519.410.633 43.670.2210.11521.510.716
Generative-based
Table 2: Novel-view synthesis on DL3DV-10K. ReconSplat outperforms both regression-based and generative-based methods on reconstruction, perceptual, and FID metrics, demonstrating its generalizability. Note : DepthSplat* is trained on the full DL3DV-10K dataset ( ∼\sim 5 ×\times more data), making direct comparison less meaningful.
Table 8
Figure 6: Dense point clouds. Given a set of NN context views ( cyan ), ReconSplat predicts MM novel views and their geometry ( red ), for both observed and unobserved regions, as demonstrated by unprojecting depth and appearance into dense point clouds for DL3DV-10K (left, center) and for RE10K (right) .
Method6 Context Views 8 Context Views
FID ↓\downarrowLPIPS ↓\downarrowDISTS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrow FID ↓\downarrowLPIPS ↓\downarrowDISTS ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrow
latentSplat [ 84 ]63.8763.870.3410.15818.050.571 (OOM)(OOM)(OOM)(OOM)(OOM)
MVSplat360 [ 16 ]41.800.4030.4030.1620.16214.9814.980.3480.348 41.1141.110.3990.3990.1610.16114.9814.980.3480.348
ReconSplat (Ours)30.130.2180.09521.550.671 27.400.1880.08722.710.715
Table 5: Increased number of context views. On DL3DV-10K, ReconSplat consistently outperforms generative baselines when given additional inputs ( N↑=6,8N_{\uparrow}=6,8 ), despite being trained with NDL3DV=N_{\text{DL3DV}}=4$$ . OOM denotes “out of memory”.
Table 11
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Qualitative comparison with LVSM [ 107 ] .
Figure 8: Influence of classifier-free guidance. Metrics over different choices of γCFG\gamma_{\text{CFG}} for DL3DV-10K [ 49 ] ( n=150n=150 ) and RealEstate10K [ 100 ] (Extrapolation). Fixing γCFG=3\gamma_{\text{CFG}}=3 yields improvements across all metrics and datasets.
Figure 9: Influence of sampling budget. Distortion and perceptual metrics given different sampling budgets for DL3DV-10K.
SettingRE10KDL3DV
Data and views
Init. fromMVSplat [ 15 ]RE10K ckpt.
Resolution2562256^{2}256×448†256\times 448^{\dagger}
Context views ( NN )24
Target views ( MM )44
Effective batch size ( BeffB_{\text{eff}} )248
Appendix
Table 8: Stage 1 training configuration. (a) Training setup for RE10K and DL3DV, including initialization, output resolution, number of context and target views, batch size, and optimization settings. (b) Loss terms and corresponding weights for the auxiliary Gaussian rendering Laux\mathcal{L}_{\text{aux}} , appearance La\mathcal{L}_{\text{a}} , geometry Lg\mathcal{L}_{\text{g}} , and latent-regularization objectives. †\dagger The encoder’s patch shim requires both image dimensions to be multiples of 6464 (patch size 1616 ×\times downscale 44 ). DL3DV inputs are therefore center-cropped from 256×480256\times 480 to 256×448256\times 448 , with intrinsics rescaled accordingly. ‡\ddagger We set the KL weight to 10−610^{-6} , following Stable Diffusion / LDM f=8f=8 autoencoder [ 60 ] .
SettingValue
Optimization
Precisionfp32
Learning rate5 ⁣× ⁣10−55\!\times\!10^{-5}
Learning rate scheduleconstant
Warm-up1K1\text{K} steps, linear
Gradient clipping0.50.5
Appendix
Table 9: Stage 2 configuration. (a) Shared optimization, diffusion-training, and inference settings. (b) Checkpoint-specific configurations for RE10K and DL3DV. †\dagger The encoder’s patch shim requires both image dimensions to be multiples of 6464 (patch size 1616 ×\times downscale 44 ). DL3DV inputs are therefore center-cropped from 256×480256\times 480 to 256×448256\times 448 , with intrinsics rescaled accordingly. ‡\ddagger “Only T\mathcal{T} ” denotes fine-tuning only the 3D convolution blocks, with all remaining parameters frozen.
Figure 10: Extensive qualitative comparisons on RE10K. Given N=N=2$$ context views (leftmost column) , our ReconSplat predicts extrapolated novel views and consistent depth (third and fourth columns) for unobserved viewpoints. We show the corresponding ground-truth image for each target viewpoint (second column) . We compare our predictions with MVSplat360 [ 16 ] and latentSplat [ 84 ] (fifth to eighth columns) .
Figure 11: Extensive qualitative comparisons on DL3DV-10K. Given N=N=4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.
Figure 12: Extensive qualitative comparisons on DL3DV-10K. Given N=N=4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.
Figure 13: Extensive qualitative comparisons on DL3DV-10K. Given N=N=4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.
Figure 14: Extensive qualitative comparisons on DL3DV-10K. Given N=N=4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.
Figure 15: Extensive qualitative comparisons on DL3DV-10K. Given N=N=4$$ context views (top row) , ReconSplat predicts high-quality novel views (odd rows, second column) and accurate depth predictions (even rows, second column) for unobserved viewpoints compared to prior work for the challenging wide-baseline setup of DL3DV-10K. For each target viewpoint, we show the corresponding ground-truth image (first column) , followed by predictions from ReconSplat and other generative baselines.

Explore similar work

CardsList
Apr 30, 2026

Generalizable Sparse-View 3D Reconstruction from Unconstrained Images

Reconstructing 3D scenes from sparse, unposed images remains challenging under real-world conditions with varying illumination and transient occlusions. Existing methods rely on scene-specific optimization using appearance embeddings or dynamic masks, which requires extensive per-scene training and fails under sparse views. Moreover, evaluations on limited scenes raise questions about generalization. We present GenWildSplat, a feed-forward framework for sparse-view outdoor reconstruction that requires no per-scene optimization. Given unposed internet images, GenWildSplat predicts depth, camera parameters, and 3D Gaussians in a canonical space using learned geometric priors. An appearance adapter modulates appearance for target lighting conditions, while semantic segmentation handles transient objects. Through curriculum learning on synthetic and real data, GenWildSplat generalizes across diverse illumination and occlusion patterns. Evaluations on PhotoTourism and MegaScenes benchmark demonstrate state-of-the-art feed-forward rendering quality, achieving real-time inference without test-time optimization

Vinayak Gupta, Chih-Hao Lin, Shenlong Wang +2

University of Maryland, College Park · University of Illinois Urbana-Champaign · Johns Hopkins University

3D Gaussian Splatting3D ReconstructionSparse-View 3D ReconstructionFeed-Forward 3D Reconstruction
AskPDF
May 12, 2026

VidSplat: Gaussian Splatting Reconstruction with Geometry-Guided Video Diffusion Priors

Gaussian Splatting has achieved remarkable progress in multi-view surface reconstruction, yet it exhibits notable degradation when only few views are available. Although recent efforts alleviate this issue by enhancing multi-view consistency to produce plausible surfaces, they struggle to infer unseen, occluded, or weakly constrained regions beyond the input coverage. To address this limitation, we present VidSplat, a training-free generative reconstruction framework that leverages powerful video diffusion priors to iteratively synthesize novel views that compensate for missing input coverage, and thereby recover complete 3D scenes from sparse inputs. Specifically, we tackle two key challenges that enable the effective integration of generation and reconstruction. First, for 3D consistent generation, we elaborate a training-free, stage-wise denoising strategy that adaptively guides the denoising direction toward the underlying geometry using the rendered RGB and mask images. Second, to enhance the reconstruction, we develop an iterative mechanism that samples camera trajectories, explores unobserved regions, synthesizes novel views, and supplements training through confidence weighted refinement. VidSplat performs robustly to sparse input and even a single image. Extensive experiments on widely used benchmarks demonstrate our superior performance in sparse-view scene reconstruction.

Jimin Tang, Wenyuan Zhang, Junsheng Zhou +5

School of Software, Tsinghua University, China · Kuaishou Technology, China · Department of Computer Science, Wayne State University, USA

3D Gaussian SplattingSparse-View 3D ReconstructionSingle-View 3D Reconstruction
AskPDF
Apr 16, 2026

GlobalSplat: Efficient Feed-Forward 3D Gaussian Splatting via Global Scene Tokens

The efficient spatial allocation of primitives serves as the foundation of 3D Gaussian Splatting, as it directly dictates the synergy between representation compactness, reconstruction speed, and rendering fidelity. Previous solutions, whether based on iterative optimization or feed-forward inference, suffer from significant trade-offs between these goals, mainly due to the reliance on local, heuristic-driven allocation strategies that lack global scene awareness. Specifically, current feed-forward methods are largely pixel-aligned or voxel-aligned. By unprojecting pixels into dense, view-aligned primitives, they bake redundancy into the 3D asset. As more input views are added, the representation size increases and global consistency becomes fragile. To this end, we introduce GlobalSplat, a framework built on the principle of align first, decode later. Our approach learns a compact, global, latent scene representation that encodes multi-view input and resolves cross-view correspondences before decoding any explicit 3D geometry. Crucially, this formulation enables compact, globally consistent reconstructions without relying on pretrained pixel-prediction backbones or reusing latent features from dense baselines. Utilizing a coarse-to-fine training curriculum that gradually increases decoded capacity, GlobalSplat natively prevents representation bloat. On RealEstate10K and ACID, our model achieves competitive novel-view synthesis performance while utilizing as few as 16K Gaussians, significantly less than required by dense pipelines, obtaining a light 4MB footprint. Further, GlobalSplat enables significantly faster inference than the baselines, operating under 78 milliseconds in a single forward pass. Project page is available at https://r-itk.github.io/globalsplat/

Roni Itkin, Noam Issachar, Yehonatan Keypur +3

The Hebrew University of Jerusalem · Westlake University

3D Gaussian Splatting3DGS AccelerationNovel View Synthesis3D Scene RepresentationFeed-Forward 3D Reconstruction
AskPDF