Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow

Quan Meng; Yujin Chen; Lei Li; Matthias Nießner; Angela Dai

cs.CVMar 30, 2026

Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow

Authors: Quan Meng, Yujin Chen, Lei Li, Matthias Nießner, Angela Dai

Organizations: Technical University of Munich · University of Virginia

Abstract

We present Seen2Scene, the first flow matching-based approach that trains directly on incomplete, real-world 3D scans for scene completion and generation. Unlike prior methods that rely on complete and hence synthetic 3D data, our approach introduces visibility-guided flow matching, which explicitly masks out unknown regions in real scans, enabling effective learning from real-world, partial observations. We represent 3D scenes using truncated signed distance field (TSDF) volumes encoded in sparse grids and employ a sparse transformer to efficiently model complex scene structures while masking unknown regions. We employ 3D layout boxes as an input conditioning signal, and our approach is flexibly adapted to various other inputs such as text or partial scans. By learning directly from real-world, incomplete 3D scans, Seen2Scene enables realistic 3D scene completion for complex, cluttered real environments. Experiments demonstrate that our model produces coherent, complete, and realistic 3D scenes, outperforming baselines in completion accuracy and generation quality.

Figures & tables

Figure 1: Seen2Scene is the first visibility-guided flow matching approach for 3D scene completion and generation, trained directly on incomplete, real-world 3D scan data. Our approach predicts high-fidelity, geometrically complete, and structurally coherent scene geometry in unobserved regions, conditioned on observed areas and 3D box-based layouts.

Figure 2: Overview of Seen2Scene. We introduce visibility-guided flow matching for modeling the distribution of TSDF partial scans. (a) Partial scan TSDF patches $\mathbf{v}$ are encoded by a masked sparse VAE ( $\mathcal{E}_{\tau},\mathcal{D}_{\tau}$ ) into latent representations $\mathbf{z}$ , masking out unknown regions unseen by the camera. (b) A sparse transformer $\mathcal{G}_{\psi}$ conditioned on 3D layout boxes $\mathcal{B}$ is trained with masked flow matching on surface and empty region tokens. (c) We fine-tune $\mathcal{G}_{\psi}$ for scan completion by injecting partial scan inputs $\mathbf{v}^{\mathrm{p}}$ via ControlNet [ 70 ] . (d) $\mathcal{G}_{\psi}$ can also be flexibly adapted for text or layout-conditioned 3D scene generation from scratch.

Figure 3: Qualitative comparison on 3D scan completion. Given real-world partial depth scans from ScanNet++ [ 67 ] and ARKitScenes [ 3 ] , Seen2Scene learns realistic, high-fidelity priors from incomplete real-scan data to produce much more complete, detailed geometry than baselines.

Method	CD $\times 10^{-2}$ $\downarrow$	L2 $\times 10^{-4}$ $\downarrow$	TMD $\times 10^{-2}$ $\uparrow$	U3D-FPD $\times 10^{-2}$ $\downarrow$
SG-NN [ 13 ]	10.77	1.90	0.00	15.41
NKSR [ 27 ]	5.22	2.55	0.00	21.78
Seen2Scene (w/o bbox)	1.92	0.53	1.69	9.57
Seen2Scene	2.05	0.51	3.33	7.79

Table 1: Quantitative evaluation of scan completion. We compare Seen2Scene with SG-NN [ 13 ] and NKSR [ 27 ] : CD on 3D-FRONT [ 20 ] , and L2, TMD, and U3D-FPD on ScanNet++ [ 67 ] , ARKitScenes [ 3 ] , and 3D-FRONT [ 20 ] .

Figure 4: 3D scene completion. Seen2Scene can complete large-scale scenes by generating geometry for unobserved regions across multiple chunks, guided by partial scan conditions injected into the ControlNet branch. Results are shown on ScanNet++ [ 67 ] and ARKitScenes [ 3 ] .

Figure 5: Qualitative comparison on layout-conditioned 3D scene generation. Our method produces more geometrically detailed and semantically coherent scenes compared to BlockFusion [ 61 ] , LT3SD [ 42 ] , and WorldGrow [ 34 ] . Layouts are from ScanNet++ [ 67 ] and ARKitScenes [ 3 ] .

Method	DINOv2-FID $\times 10^{2}$ $\downarrow$	U3D-FPD $\times 10^{2}$ $\downarrow$	VLM Score $\uparrow$
BlockFusion [ 61 ]	12.92	28.30	3.51
LT3SD [ 42 ]	3.51	28.55	5.20
WorldGrow [ 34 ]	4.44	50.83	4.69
Seen2Scene (Synthetic only)	2.73	11.55	5.61

Table 2: Quantitative comparison on 3D scene generation. We report DINOv2 Fréchet Inception Distance (FID), Uni3D Fréchet Point Cloud Distance (FPD), and VLM Quality Score assessed by Qwen3-VL on a 0–10 scale 3D-FRONT [ 20 ] only.

Figure 6: Text to scene. Given a text description, Seen2Scene generates a 3D object layout first via an LLM and then synthesizes high-fidelity scene geometry from scratch conditioned on the layout.

Configuration	Reconstruction			Generation
Configuration	L1 $\times 10^{-4}$ $\downarrow$	L2 $\times 10^{-6}$ $\downarrow$	CD $\times 10^{-3}$ $\downarrow$	U3D-FPD $\times 10^{2}$ $\downarrow$	VLM Score $\uparrow$
w/o masked training.	3.24	7.1	10.64	20.28	5.66
Seen2Scene (full)	2.01	2.0	9.04	14.56	5.70

Table 3: Masked training ablation. Reconstruction metrics (L1, L2, CD) assess the autoencoder and generation metrics (U3D-FPD, VLM) assess the flow matching model; CD is computed on 3D-FRONT only, while others are evaluated across datasets.

Figure 7: Qualitative ablation on masked training. Comparison of generated scenes with and without masked flow matching during training. Without two stages of masked training, the model tends to generate holes (under the table) following the training incomplete scans.

Configuration	DINOv2-FID $\times 10^{2}$ $\downarrow$	U3D-FPD $\times 10^{2}$ $\downarrow$	VLM Score $\uparrow$
Synthetic only	2.90	21.49	5.61
Discrete Category	3.17	21.46	5.64
Without Label Synonym	2.14	18.77	5.62
Seen2Scene (full)	1.90	14.56	5.70

Table 4: Ablation study on generation. We evaluate the contribution of each design choice by removing it from the full model on ScanNet++ [ 67 ] , ARKitScenes [ 3 ] , and 3D-FRONT [ 20 ] .

Figure 8: Qualitative ablation on real data. Training the model only on 3D-FRONT [ 20 ] , or with limited categories, or without label synonym augmentation, reduces the model’s generalization ability to diverse object layouts, like “clothes on an office chair.”

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: More 3D scene completion results. Seen2Scene can complete large-scale scenes by generating geometry for unobserved regions across multiple chunks, guided by partial scan conditions injected into the ControlNet branch.

Figure 9: Qualitative ablation on object bounding boxes. Seen2Scene achieves comparable completion quality without object bounding box conditioning.

Dataset	# Samples	Synthetic/Real	Object Annot.	Mesh Source	Semantic Seg.	# Categories
3D-FRONT	6,802	Synthetic	✓	Artist	✓	99
ScanNet++	1,006	Real	✓	LiDAR	✓	2,877
ARKitScenes	928	Real	✓	LiDAR	–	2,466
Total	8,736	–	–	–	–	4,489 (unique)

Appendix

Table 5: Dataset statistics. Overview of the training datasets used, covering both synthetic (3D-FRONT) and real-world (ScanNet++, ARKitScenes) indoor scenes.

Figure 10: Dataset analysis. Left: object-name semantic distribution (PCA of CLIP embeddings). Top right: vocabulary diversity. Bottom right: object density per scene across 3D-FRONT, ScanNet++, and ARKitScenes.

Panel of Figure 11 — Figure 11: Prompt template for LLM-based label augmentation. The placeholders {n} , {label} , and {n_per_type} are filled at runtime for each category.

Figure 12: Architecture of Seen2Scene. (a) A masked sparse VAE encodes partial TSDF patches into compact geometry latents via sparse convolutional downsampling, with category and TSDF reconstruction heads in the decoder. (b) A sparse DiT generates geometry latents from Gaussian noise via flow matching, conditioned on CLIP-encoded layout embeddings and timestep information.

Figure 13: Architecture of scene completion. A ControlNet branch injects partial scan geometry encoded by the frozen VAE encoder into the locked pretrained generator, guiding completion of unobserved regions while the layout condition provides semantic structure.

Method	Geom.	Struc.	Sem.	Cont.	Avg.
Method	Qual.	Logic	Cohere.	Dens.	Avg.
BlockFusion	3.82	3.92	3.43	2.85	3.51
LT3SD	5.22	5.48	5.14	4.97	5.20
WorldGrow	4.80	4.77	4.53	4.66	4.69
Seen2Scene (syn.)	5.27	5.79	5.74	5.62	5.61
Seen2Scene (full)	5.37	5.84	5.84	5.75	5.70

Appendix

Table 6: VLM-based Perceptual Evaluation. Comparison of Seen2Scene with state-of-the-art baselines on 3D scene generation. Scores are provided by Qwen3-VL-8B-Instruct (0-10 scale), evaluating geometric fidelity, structural logic, semantic coherence, and content density.

Panel of Figure 14 — Figure 14: Detailed VLM prompts for evaluating perceptual quality of generated 3D scene chunks.

3D Scene Completion

3D scene completion infers missing geometry, occupancy, semantics, and appearance from partial images, point clouds, or depth scans.

See all 41 papers

Sparse auto-regressive modeling for scene generation from multi-view images
Sep 3, 2026Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel +4Sparse-View 3D Reconstruction 3D Scene Generation
1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA
VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching
May 29, 2026Tuan Duc Ngo, Chuang Gan, Evangelos Kalogerakis3D Reconstruction 3D Scene Representation
UMass Amherst · 2TU Crete
ScaleBlind: Point Cloud Completion under Unknown Scale
Sep 20, 2026Shenghui Wu, Chen Wang, Yuan Feng +3Point Cloud Completion
Shandong University, China · University of Edinburgh, United Kingdom

Sep 3, 2026J/K move · Enter open · S save

Sparse auto-regressive modeling for scene generation from multi-view images

Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel, Wonjune Cho+3

1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA

Sparse-View 3D Reconstruction 3D Scene Generation 3D Scene Representation Autoregressive Generation 3D Scene Completion Autoregressive 3D Scene Generation

Open paper AskPDF arXiv

Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.

Abstract

3D Scene Completion

Explore similar work

Sparse auto-regressive modeling for scene generation from multi-view images

VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching

ScaleBlind: Point Cloud Completion under Unknown Scale