Humans exhibit an innate capacity to rapidly perceive and segment objects from video observations, and even mentally assemble them into structured 3D scenes. Replicating such capability, termed compositional 3D reconstruction, is pivotal for the advancement of Spatial Intelligence and Embodied AI. However, existing methods struggle to achieve practical deployment due to the insufficient integration of cross-modal information, leaving them dependent on manual object prompting, reliant on auxiliary visual inputs, and restricted to overly simplistic scenes by training biases. To address these limitations, we propose ReplicateAnyScene, a framework capable of fully automated and zero-shot transformation of casually captured videos into compositional 3D scenes. Specifically, our pipeline incorporates a five-stage cascade to extract and structurally align generic priors from vision foundation models across textual, visual, and spatial dimensions, grounding them into structured 3D representations and ensuring semantic coherence and physical plausibility of the constructed scenes. To facilitate a more comprehensive evaluation of this task, we further introduce the C3DR benchmark to assess reconstruction quality from diverse aspects. Extensive experiments demonstrate the superiority of our method over existing baselines in generating high-quality compositional 3D scenes.
Figures & tables
Figure 1: We propose ReplicateAnyScene , a framework for fully automated and zero-shot compositional 3D reconstruction from casually captured videos. Our method extracts and aligns cross-modal priors from vision foundation models to generate semantically coherent and physically plausible 3D scenes.
Figure 2: Framework Overview. Our pipeline consists of a five-stage cascade where each stage is specifically designed to resolve targeted alignment gaps among our three core modalities including textual (green), visual (orange), and spatial (blue) . The gradient backgrounds and multi-colored dashed borders within each module explicitly illustrate the specific cross-modal alignment process occurring at that step.
Figure 3: Overview of the proposed C3DR benchmark. This figure illustrates the ground-truth construction pipeline. We first manually select the semantic text and corresponding video frames for each object. We then independently condition a diverse suite of generation models on multiple individual input views for the final selection of high-quality 3D assets. Finally, professional modelers manually place these assets against reference scene meshes to ensure accurate spatial layouts.
Figure 4: Qualitative comparison on the C3DR benchmark. We present representative qualitative visualizations of the final reconstructed scenes alongside results from MetaScenes [ 70 ] and SimRecon [ 62 ] .
Method
Textual
Visual
Geometric
Rec ↑
F1 ↑
PSNR ↑
SSIM ↑
LPIPS ↓
MUSIQ ↑
CD ↓
F ↑
NC ↑
MetaScenes
64.29
75.10
11.63
0.746
0.243
68.18
0.643
3.95
41.04
SimRecon
-
80.88
15.77
0.828
0.206
69.49
0.778
6.03
46.32
Ours
91.43
90.86
19.45
0.854
0.154
71.03
0.614
12.91
48.30
Table 1: Quantitative comparisons on the C3DR benchmark. We evaluate against MetaScenes [ 70 ] and SimRecon [ 62 ] across textual completeness (Rec, F1), visual quality (PSNR, SSIM [ 57 ] , LPIPS [ 73 ] , MUSIQ [ 24 ] ), and geometric accuracy (CD, F, NC). Our method outperforms all baselines across every evaluation dimension.
Method
Rec ↑
SRR ↓
DGR ↑
Time ↓
Single Frame Merging
91.27
0.44
1.712
77s
Global Video Inference
77.01
0
1.214
18s
Ours
91.43
0
1.715
82s
Table 2: Quantitative ablation study on progressive object discovery. We compare our approach against Single Frame Merging and Global Video Inference. Reported metrics include recall (Rec) on benchmark-annotated categories, semantic redundancy rate (SRR), the discovery gain ratio (DGR), and total runtime.
Figure 5: Qualitative ablation study on stages 2 to 5 of our pipeline. Each subfigure shows the scene reconstruction when a specific stage is ablated alongside results from our full method and the ground-truth scene for comparison. Our full pipeline effectively resolves various visual and spatial artifacts.
Method
Visual
Geometric
PSNR ↑
SSIM ↑
LPIPS ↓
MUSIQ ↑
CD ↓
F ↑
NC ↑
w/o Deduplication
18.90
0.832
0.176
69.76
0.730
12.25
45.38
Max Pixel-Area View
17.26
0.803
0.194
71.87
0.729
9.67
44.01
w/o Alignment
16.85
0.781
0.229
70.33
0.790
12.22
43.26
ICP Alignment
16.53
0.770
0.236
68.62
0.922
11.82
42.59
w/o Refinement
16.58
0.798
0.222
68.59
0.799
9.54
44.36
Table 3: Quantitative ablation study on stages 2 to 5 of our pipeline. We evaluate across visual quality (PSNR, SSIM [ 57 ] , LPIPS [ 73 ] , MUSIQ [ 24 ] ) and geometric accuracy (CD, F, NC). The results clearly indicate that our full five-stage framework achieves the optimal performance across all evaluated metrics.
The growing demand for Embodied AI and VR applications has highlighted the need for synthesizing high-quality 3D indoor scenes from sparse inputs. However, existing approaches struggle to infer massive amounts of missing geometry in large unseen areas while maintaining global consistency, often producing locally plausible but globally inconsistent reconstructions. We present Rein3D, a framework that reconstructs full 360-degree indoor environments by coupling explicit 3D Gaussian Splatting (3DGS) with temporally coherent priors from video diffusion models. Our approach follows a "restore-and-refine" paradigm: we employ a radial exploration strategy to render imperfect panoramic videos along trajectories starting from the origin, effectively uncovering occluded regions from a coarse 3DGS initialization. These sequences are restored by a panoramic video-to-video diffusion model and further enhanced via video super-resolution to synthesize high-fidelity geometry and textures. Finally, these refined videos serve as pseudo-ground truths to update the global 3D Gaussian field. To support this task, we construct PanoV2V-15K, a dataset of over 15K paired clean and degraded panoramic videos for diffusion-based scene restoration. Experiments demonstrate that Rein3D produces photorealistic and globally consistent 3D scenes and significantly improves long-range camera exploration compared with existing baselines.
Dehui Wang, Rong Wei, Yue Shi +9
Shanghai Jiao Tong University · Manycore Tech Inc. · The University of Hong Kong +1
We present DistScene, a framework for single-image compositional 3D scene generation by jointly modeling the environment and individual objects. Unlike existing methods that represent scenes primarily as collections of objects, we model the environment as an explicit scene component to provide geometric context for object placement. Specifically, we introduce Scene-Frame Generation, which jointly generates separate environment and object components in a shared coordinate frame, allowing their geometry and relative placement to be learned together. Then we introduce Object-Centric Refinement to refine each object in a local frame with scene context. Finally, we develop Object-to-Scene Distillation to transfer pretrained object-generation priors to scene generation through automatically composed and rendered synthetic scenes. Evaluations on indoor and outdoor benchmarks demonstrate improved scene-level spatial coherence over the evaluated baselines. Project page: https://coolbeam.github.io/DistScene/
Kunming Luo, Hongyu Yan, Ken Deng +6
The Hong Kong University of Science and Technology · TeleAI, China Telecom
Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at https://github.com/ROUJINN/SceneAssistant