SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

Xingjian Ran; Xiaoye Mo; Sihao Liu; Jianyu Zhang; Li Luo; Bo Dai

cs.CVSep 4, 2026

SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

Authors: Xingjian Ran, Xiaoye Mo, Sihao Liu, Jianyu Zhang, Li Luo, Bo Dai

Organizations: The University of Hong Kong · University of Electronic Science and Technology of China

Abstract

Diverse and simulation-ready indoor scenes are essential for interactive entertainment and embodied AI, yet their scalable generation remains challenging. Recent agentic text-to-3D scene pipelines that rely on vision-language models (VLMs) can generate scenes of high fidelity but require costly iterative object placement and refinement. Another mainstream paradigm, parametric image-to-3D scene models, produces scenes efficiently from strong priors learned from 2D images but often leads to imprecise and physically invalid scenes. More importantly, both paradigms struggle to output diverse scenes for a single input, making it hard for them to reflect the dynamically changing nature of real scenes. In this paper we propose SceneMosaic, a framework that combines the merits of both paradigms. It obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity. Within the evolution process, SceneMosaic exploits the locality of natural scenes and decomposes a scene into independent local units, allowing separate evolution within each unit before composing the global scene via Cartesian product. On SceneEval-100, SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Our code is publicly available at https://github.com/rxjfighting/SceneMosaic.

Figures & tables

Figure 2: Overview of our proposed framework. Given a single reference input signal, we first reconstruct and structure the scene into object-centric local units. We then perform physics-based layout stabilization and Critic-Actor agentic evolution to generate diverse local layout variants. Finally, the variants are combinatorially assembled and selected using a novelty-aware distance metric and dynamic greedy search to produce diverse, physically plausible scene layouts.

Figure 3: Qualitative comparison with existing text-to-3D and image-to-3D baseline methods. Our approach generates realistic, high-quality 3D layouts and supports fast variant scene generation while avoiding physically implausible configurations.

Agentic Text-to-3D
Method	POS $\uparrow$	ROT $\uparrow$	NAV $\uparrow$	COL $\downarrow$	OOB $\downarrow$	Time $\downarrow$
HoloDeck	49.8 $\pm$ 6.2	47.2 $\pm$ 7.6	99.5 $\pm$ 0.3	12.1 $\pm$ 3.4	0.9 $\pm$ 0.3	0.05h
SceneWeaver	55.2 $\pm$ 6.8	51.8 $\pm$ 6.5	97.3 $\pm$ 1.1	10.7 $\pm$ 3.2	0.0 $\pm$ 0.0	1.07h
SAGE $\dagger$	59.8 $\pm$ 7.9	53.3 $\pm$ 5.5	90.1 $\pm$ 1.2	1.8 $\pm$ 2.5	0.8 $\pm$ 0.5	7.56h
SceneSmith	83.5 $\pm$ 2.7	80.9 $\pm$ 3.1	96.9 $\pm$ 0.8	0.7 $\pm$ 1.4	0.3 $\pm$ 0.3	3.43h
Image-to-3D

Table 1: Quantitative comparison on SceneEval-100. POS and ROT evaluate semantic layout quality, while NAV, COL, and OOB measure physical validity. Time indicates the average generation time per scene.

Method	Sem. $\uparrow$	Phy. $\uparrow$
SceneSmith	4.01 $\pm$ 0.99	4.16 $\pm$ 0.97
SAGE	3.08 $\pm$ 1.13	2.81 $\pm$ 1.31
SceneGen	2.12 $\pm$ 1.21	2.08 $\pm$ 1.33
SAM3D	3.22 $\pm$ 1.21	2.43 $\pm$ 1.45
Ours	4.33 $\pm$ 0.89	4.47 $\pm$ 0.82
Kendall’s $W$	0.648	0.633

Table 2: User study results. We report the mean and standard deviation on semantic plausibility (Sem.) and physical plausibility (Phy.), along with Kendall’s coefficient.

Base Scenes
	POS $\uparrow$	ROT $\uparrow$	NAV $\uparrow$	COL $\downarrow$	OOB $\downarrow$	Time $\downarrow$
Full SceneMosaic	84.3 $\pm$ 2.0	78.4 $\pm$ 3.4	99.2 $\pm$ 0.4	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.14h
(1) w/o Image Init (Text-only Init)	80.7 $\pm$ 3.2	74.2 $\pm$ 3.5	98.8 $\pm$ 0.6	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	1.25h
(2) w/o Physics	81.5 $\pm$ 2.4	75.6 $\pm$ 3.7	96.8 $\pm$ 1.6	18.9 $\pm$ 4.8	8.5 $\pm$ 3.3	0.13h
(3) Perspective 3D Ops + Local Decomposition	78.1 $\pm$ 4.1	71.8 $\pm$ 4.4	98.7 $\pm$ 0.7	2.6 $\pm$ 1.3	2.4 $\pm$ 1.1	0.31h
(4) Global Evolution + Perspective 3D Ops	72.3 $\pm$ 4.8	64.9 $\pm$ 5.6	97.2 $\pm$ 1.3	12.1 $\pm$ 3.7	6.2 $\pm$ 2.0	0.68h

Table 3: Ablation study on SceneEval-100. We evaluate the impact of key component choices on both base and variant scene generation.

Figure 4: Additional qualitative results of our method. We randomly sample variant scenes generated by our framework to demonstrate the diversity of the resulting layouts. The examples show that our method can produce diverse yet physically plausible scene configurations while preserving the semantic structure of the input scene.

Method	Div-IoU $\downarrow$	Div-N $\uparrow$	Time $\downarrow$
Full SceneMosaic	0.42 $\pm$ 0.08	0.37 $\pm$ 0.06	0.03h
w/o Novelty Selection	0.58 $\pm$ 0.09	0.19 $\pm$ 0.05	0.03h
Repeated Sampling	0.71 $\pm$ 0.07	0.12 $\pm$ 0.04	0.14h

Table 4: Diversity against degraded variants of our own pipeline. All settings share the same reconstructed assets and produce

m=5

layouts per input. w/o Novelty Selection draws

m

scenes uniformly at random from

\mathcal{C}

, and Repeated Sampling runs the full agentic evolution

m

times from the same initialization with different random seeds, without local unit decomposition or Cartesian assembly. Time is amortized per variant.

Method	Div-IoU $\downarrow$	Div-N $\uparrow$	POS $\uparrow$	ROT $\uparrow$	RelKeep $\uparrow$
Full SceneMosaic	0.42 $\pm$ 0.08	0.37 $\pm$ 0.06	84.5 $\pm$ 2.4	79.9 $\pm$ 3.6	99.1 $\pm$ 1.2
Perturbation, $\sigma=0.10$	0.63 $\pm$ 0.08	0.15 $\pm$ 0.05	79.2 $\pm$ 3.5	71.4 $\pm$ 4.6	88.3 $\pm$ 4.2
Perturbation, $\sigma=0.25$	0.43 $\pm$ 0.09	0.30 $\pm$ 0.07	71.6 $\pm$ 4.4	62.1 $\pm$ 5.3	71.5 $\pm$ 6.1
Perturbation, $\sigma=0.50$	0.24 $\pm$ 0.10	0.52 $\pm$ 0.09	60.3 $\pm$ 5.6	51.7 $\pm$ 6.2	52.8 $\pm$ 7.4

Table 5: Diversity versus semantic quality under a random perturbation sweep. RelKeep denotes the fraction of attach/contain relations of the base scene that remain satisfied in the variant. The

\sigma=0.25

setting is matched to the diversity of Full SceneMosaic, enabling a like-for-like comparison at equal diversity.

3D Scene Generation

3D scene generation creates structured, explorable environments from text, images, maps, or other inputs.

See all 189 papers

Oct 23, 2026

SceneBloom: Agentic Indoor Environment Generation with Realistic and Diverse Layouts for Embodied Learning

Training generalizable embodied agents requires realistic, varied, and simulation-ready indoor environments. Language-guided methods support semantic planning and object-level reasoning, but precise spatial grounding remains difficult. Visual generative models offer rich priors over plausible arrangements, yet their outputs require geometric reasoning to become physically valid 3D scenes. We introduce SceneBloom, an agentic framework that combines visual layout sampling with contract-guided 3D compilation. A hierarchical sampler uses diffusion priors to generate furniture arrangements within functional zones. A compiler then uses relations extracted from each proposal to guide geometric fitting and local pose correction. Across five room types, SceneBloom attains the highest visual quality and physical stability among the compared methods and is the only method with no detected mesh collisions and out-of-bounds objects. Its layouts are also closer to the distribution of real rooms than those of all compared baselines. Object navigation policies trained in these realistic scenes achieve a higher success rate in held-out real scans than those trained on an equal number of real training scans. Code is available at https://github.com/Tinaliu0123/scenebloom.

Yuhan Liu, Lianhui Qin, Shenji Wan

New York University · University of California, San Diego

Indoor Scene Generation 3D Scene Layout Generation Functional 3D Scene Generation

Ask PDF

Apr 12, 2026

HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models

3D layout generation and editing play a crucial role in Embodied AI and immersive VR interaction. However, manual creation requires tedious labor, while data-driven generation often lacks diversity. The emergence of large models introduces new possibilities for 3D scene synthesis. We present HOG-Layout that enables text-driven hierarchical scene generation, optimization and real-time scene editing with large language models (LLMs) and vision-language models (VLMs). HOG-Layout improves scene semantic consistency and plausibility through retrieval-augmented generation (RAG) technology, incorporates an optimization module to enhance physical consistency, and adopts a hierarchical representation to enhance inference and optimization, achieving real-time editing. Experimental results demonstrate that HOG-Layout produces more reasonable environments compared with existing baselines, while supporting fast and intuitive scene editing.

Haiyan Jiang, Deyu Zhang, Dongdong Weng +2

The Hong Kong Polytechnic University · Beijing Institute of Technology

3D Scene Generation 3D Scene Layout Generation 3D Scene Editing Text-Guided 3D Scene Editing 3D VLMs

Ask PDF

Oct 16, 2026

Scene-Designer: Learning Design Intelligence for Agentic 3D Scene Generation

Generating complete, coherent, and usable 3D indoor scenes from natural language requires strong design intelligence: the ability to design high-quality scenes and critique their quality. However, existing approaches either rely on general-purpose models lacking domain-specific design experience or learn from static scenes. In this work, we introduce Scene-Designer, an agentic framework that learns design intelligence through complementary design and critique experiences. We construct two datasets: Scene-OTA-35K, which provides Observation--Thought--Action supervision for learning scene design and efficient error correction, and Scene-OTC-15K, which provides Observation--Thought--Critic supervision for learning scene quality critique across physical validity, functional correctness, and layout coherence. Based on these experiences, we propose Scene-GRPO, a Critic-guided GRPO framework that improves design decisions and reduces unnecessary trial-and-error corrections. Extensive experiments demonstrate that Scene-Designer achieves superior performance in physical validity, functional correctness, and layout coherence, while the learned Scene-Critic significantly outperforms general-purpose models in scene critique. Moreover, test-time Critic guidance further improves scene generation, showing that the Critic can provide useful guidance beyond training. Additional analysis suggests a trend of mutual enhancement between scene design and critique abilities.

Xuanyu Liu, Hongyan Feng, Miao Pan +5

Zhejiang University · Universal Identification Technology (Hangzhou) Co.,Ltd.

3D Scene Generation Indoor Scene Generation Functional 3D Scene Generation

Ask PDF

Abstract

3D Scene Generation

Explore similar work

SceneBloom: Agentic Indoor Environment Generation with Realistic and Diverse Layouts for Embodied Learning

HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models

Scene-Designer: Learning Design Intelligence for Agentic 3D Scene Generation