Diverse and simulation-ready indoor scenes are essential for interactive entertainment and embodied AI, yet their scalable generation remains challenging. Recent agentic text-to-3D scene pipelines that rely on vision-language models (VLMs) can generate scenes of high fidelity but require costly iterative object placement and refinement. Another mainstream paradigm, parametric image-to-3D scene models, produces scenes efficiently from strong priors learned from 2D images but often leads to imprecise and physically invalid scenes. More importantly, both paradigms struggle to output diverse scenes for a single input, making it hard for them to reflect the dynamically changing nature of real scenes. In this paper we propose SceneMosaic, a framework that combines the merits of both paradigms. It obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity. Within the evolution process, SceneMosaic exploits the locality of natural scenes and decomposes a scene into independent local units, allowing separate evolution within each unit before composing the global scene via Cartesian product. On SceneEval-100, SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Our code is publicly available at https://github.com/rxjfighting/SceneMosaic.
Figures & tables
Figure 2: Overview of our proposed framework. Given a single reference input signal, we first reconstruct and structure the scene into object-centric local units. We then perform physics-based layout stabilization and Critic-Actor agentic evolution to generate diverse local layout variants. Finally, the variants are combinatorially assembled and selected using a novelty-aware distance metric and dynamic greedy search to produce diverse, physically plausible scene layouts.
Figure 3: Qualitative comparison with existing text-to-3D and image-to-3D baseline methods. Our approach generates realistic, high-quality 3D layouts and supports fast variant scene generation while avoiding physically implausible configurations.
Method
POS ↑
ROT ↑
NAV ↑
COL ↓
OOB ↓
Time ↓
Agentic Text-to-3D
HoloDeck
49.8 ± 6.2
47.2 ± 7.6
99.5 ± 0.3
12.1 ± 3.4
0.9 ± 0.3
0.05h
SceneWeaver
55.2 ± 6.8
51.8 ± 6.5
97.3 ± 1.1
10.7 ± 3.2
0.0 ± 0.0
1.07h
SAGE †
59.8 ± 7.9
53.3 ± 5.5
90.1 ± 1.2
1.8 ± 2.5
0.8 ± 0.5
7.56h
SceneSmith
83.5 ± 2.7
80.9 ± 3.1
96.9 ± 0.8
0.7 ± 1.4
0.3 ± 0.3
3.43h
Image-to-3D
Table 1: Quantitative comparison on SceneEval-100. POS and ROT evaluate semantic layout quality, while NAV, COL, and OOB measure physical validity. Time indicates the average generation time per scene.
Method
Sem. ↑
Phy. ↑
SceneSmith
4.01 ± 0.99
4.16 ± 0.97
SAGE
3.08 ± 1.13
2.81 ± 1.31
SceneGen
2.12 ± 1.21
2.08 ± 1.33
SAM3D
3.22 ± 1.21
2.43 ± 1.45
Ours
4.33 ± 0.89
4.47 ± 0.82
Kendall’s W
0.648
0.633
Table 2: User study results. We report the mean and standard deviation on semantic plausibility (Sem.) and physical plausibility (Phy.), along with Kendall’s coefficient.
POS ↑
ROT ↑
NAV ↑
COL ↓
OOB ↓
Time ↓
Base Scenes
Full SceneMosaic
84.3 ± 2.0
78.4 ± 3.4
99.2 ± 0.4
0.0 ± 0.0
0.0 ± 0.0
0.14h
(1) w/o Image Init (Text-only Init)
80.7 ± 3.2
74.2 ± 3.5
98.8 ± 0.6
0.0 ± 0.0
0.0 ± 0.0
1.25h
(2) w/o Physics
81.5 ± 2.4
75.6 ± 3.7
96.8 ± 1.6
18.9 ± 4.8
8.5 ± 3.3
0.13h
(3) Perspective 3D Ops + Local Decomposition
78.1 ± 4.1
71.8 ± 4.4
98.7 ± 0.7
2.6 ± 1.3
2.4 ± 1.1
0.31h
(4) Global Evolution + Perspective 3D Ops
72.3 ± 4.8
64.9 ± 5.6
97.2 ± 1.3
12.1 ± 3.7
6.2 ± 2.0
0.68h
Table 3: Ablation study on SceneEval-100. We evaluate the impact of key component choices on both base and variant scene generation.
Figure 4: Additional qualitative results of our method. We randomly sample variant scenes generated by our framework to demonstrate the diversity of the resulting layouts. The examples show that our method can produce diverse yet physically plausible scene configurations while preserving the semantic structure of the input scene.
Method
Div-IoU ↓
Div-N ↑
Time ↓
Full SceneMosaic
0.42 ± 0.08
0.37 ± 0.06
0.03h
w/o Novelty Selection
0.58 ± 0.09
0.19 ± 0.05
0.03h
Repeated Sampling
0.71 ± 0.07
0.12 ± 0.04
0.14h
Table 4: Diversity against degraded variants of our own pipeline. All settings share the same reconstructed assets and produce m=5 layouts per input. w/o Novelty Selection draws m scenes uniformly at random from C , and Repeated Sampling runs the full agentic evolution m times from the same initialization with different random seeds, without local unit decomposition or Cartesian assembly. Time is amortized per variant.
Method
Div-IoU ↓
Div-N ↑
POS ↑
ROT ↑
RelKeep ↑
Full SceneMosaic
0.42 ± 0.08
0.37 ± 0.06
84.5 ± 2.4
79.9 ± 3.6
99.1 ± 1.2
Perturbation, σ=0.10
0.63 ± 0.08
0.15 ± 0.05
79.2 ± 3.5
71.4 ± 4.6
88.3 ± 4.2
Perturbation, σ=0.25
0.43 ± 0.09
0.30 ± 0.07
71.6 ± 4.4
62.1 ± 5.3
71.5 ± 6.1
Perturbation, σ=0.50
0.24 ± 0.10
0.52 ± 0.09
60.3 ± 5.6
51.7 ± 6.2
52.8 ± 7.4
Table 5: Diversity versus semantic quality under a random perturbation sweep. RelKeep denotes the fraction of attach/contain relations of the base scene that remain satisfied in the variant. The σ=0.25 setting is matched to the diversity of Full SceneMosaic, enabling a like-for-like comparison at equal diversity.