HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models

Haiyan Jiang; Deyu Zhang; Dongdong Weng; Weitao Song; Henry Been-Lirn Duh

cs.CVApr 12, 2026

HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Models

Authors: Haiyan Jiang, Deyu Zhang, Dongdong Weng, Weitao Song, Henry Been-Lirn Duh

Organizations: The Hong Kong Polytechnic University · Beijing Institute of Technology

Abstract

3D layout generation and editing play a crucial role in Embodied AI and immersive VR interaction. However, manual creation requires tedious labor, while data-driven generation often lacks diversity. The emergence of large models introduces new possibilities for 3D scene synthesis. We present HOG-Layout that enables text-driven hierarchical scene generation, optimization and real-time scene editing with large language models (LLMs) and vision-language models (VLMs). HOG-Layout improves scene semantic consistency and plausibility through retrieval-augmented generation (RAG) technology, incorporates an optimization module to enhance physical consistency, and adopts a hierarchical representation to enhance inference and optimization, achieving real-time editing. Experimental results demonstrate that HOG-Layout produces more reasonable environments compared with existing baselines, while supporting fast and intuitive scene editing.

Figures & tables

Figure 2: The pipeline of HOG-Layout. In the layout generation phase, layouts are generated sequentially according to groups and then optimized. The optimized scene is used as input for generating the layout for the next group. In the optimization phase, iterative optimization is performed according to the parent-child hierarchy, with optimization occurring simultaneously across different layers.

	$\uparrow~$ CNT %	$\uparrow~$ ATR %	$\uparrow~$ OOR %	$\uparrow~$ OAR %	$\downarrow~$ COL ${}_{ob}\%$	$\downarrow~$ COL ${}_{sc}\%$	$\uparrow~$ SUP %	$\uparrow~$ NAV %	$\uparrow~$ ACC %	$\downarrow~$ OOB %	$\downarrow~$ Time $(s)$	$\uparrow~$ SP	$\uparrow~$ CLIPsim
	Fidelity				Plausibility
LayoutGPT	29.02	22.48	6.11	11.48	35.67	49.00	34.39	99.52	73.77	35.81	37.41	35.14	16.22
Holodeck	39.98	45.62	18.99	38.27	12.24	63.00	34.72	99.29	91.21	0.34	271.97	55.45	18.25
LayoutVLM	76.72	61.79	38.09	61.99	29.44	55.00	77.54	96.96	87.67	8.42	321.47	65.54	18.40
HOG-Layout	77.84	60.01	43.09	75.74	5.28	16.00	81.17	97.15	94.74	2.45	70.44	69.69	18.61

Table 1: Evaluation results of different methods.

Figure 3: The editing pipeline of HOG-Layout.

Figure 4: Results of SP Score Distribution (Kernel Density Estimation, KDE) of different methods.

Figure 5: Generation Examples of different methods on the benchmark.

Method	Plausibility $\uparrow$	Semantic Alignment $\uparrow$
LayoutGPT	2.43 ( $\pm$ 0.54)	2.58 ( $\pm$ 0.62)
Holodeck	3.97 ( $\pm$ 0.91)	3.66 ( $\pm$ 0.76)
LayoutVLM	3.69 ( $\pm$ 0.68)	4.61 ( $\pm$ 0.92)
HOG-Layout	5.33 ( $\pm$ 0.88)	5.75 ( $\pm$ 0.87)

Table 2: Human evaluation results.

	$\uparrow~$ OOR %	$\uparrow~$ OAR %	$\downarrow~$ COL ${}_{ob}\%$	$\downarrow~$ COL ${}_{sc}\%$	$\uparrow~$ SUP %	$\uparrow~$ NAV %	$\uparrow~$ ACC %	$\downarrow~$ OOB %	$\downarrow~$ Time $(s)$	$\uparrow~$ SP	$\uparrow~$ CLIPsim
	Fidelity		Plausibility
w/o RAG	37.89	68.98	6.24	20.00	83.34	96.56	93.42	4.84	82.26	67.08	18.46
w/o layout description	40.42	67.14	7.54	16.00	83.48	97.33	94.34	4.55	75.87	67.58	18.95
w/o object data	38.23	65.07	7.80	22.00	82.16	97.83	94.46	6.61	76.09	65.61	18.95
only room planning	39.06	67.63	6.26	19.00	82.52	97.70	94.91	2.70	67.66	68.02	18.43
w/o grid	41.25	67.60	6.53	19.00	83.07	97.84	95.61	5.54	66.82	67.51	18.53

Table 3: Results of ablation study.

Retrieval Methods	$\uparrow~$ CNT %	$\uparrow~$ ATR %
CLIP	75.34	63.05
CLIP+SBERT	76.30	61.13
CLIP+SBERT+SIZE	77.86	64.21

Table 4: Comparison of retrieval methods.

Instruction type	Time simple	TIME complex
Move	$15.64\pm 1.55$	$16.35\pm 3.58$
Delete	$15.55\pm 1.14$	$18.99\pm 11.56$
Add	$18.39\pm 1.38$	$23.93\pm 6.15$

Table 5: Editing time results.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description	Value
$w_{\text{col}}$	Collision weight (Horizontal)	1.134
$w_{\text{vcol}}$	Collision weight (Vertical)	2.850
$w_{\text{bnd}}$	Boundary weight	2.857
$w_{\text{sup}}$	Support weight	0.525
$w_{\text{adj}}$	Adjacent weight	0.833
$w_{\text{wall}}$	Against-wall weight	2.977

Appendix

Table 6: Optimized Parameters for the Hierarchical Force-Directed Optimizer.

Panel of Figure 8 — Figure 8: Optional object acquisition methods. (a) The retrieval-based pipeline matches query text and geometry against a fixed database using semantic and visual similarity. (b) The generative pipeline replaces retrieval with generative objects: generating an image from text (e.g., via DALL-E) and then converting it to a 3D model (e.g., via Hunyuan 3D).

Figure 9: Examples generated by the pipeline with generative 3D objects. The pipeline supports generating assets that do not exist in the asset library, such as anime characters, sports car models, and more.

Instruction Type	ESR ( $\uparrow$ )	COL scene ( $\downarrow$ )	OOB scene ( $\downarrow$ )
Move	80%	10%	10%
Delete	100%	0%	0%
Add	100%	0%	0%

Appendix

Table 7: Quantitative results of scene editing. We report the Editing Success Rate (ESR), Scene-level Collision (

COL_{scene}

), and Scene-level Out-of-Bounds (

OOB_{scene}

Figure 10: Layouts generated by HOG-Layout and the corresponding input instructions.

Figure 11: Edited layouts produced by HOG-Layout using the instructions.

Figure 12: Generation examples of different methods on the benchmark.

Explore similar work

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning
Jul 15, 2026Yuan Xiao, Can Wang, Xiangyu Kong +13D Scene Editing 3D Scene Generation
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University
ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing
Jun 3, 2025Martin JJ. Bucher, Iro Armeni3D Scene Editing 3D Scene Representation
Stanford University
Repurposing 3D Generative Model for Autoregressive Layout Generation
Apr 17, 2026Haoran Feng, Yifan Niu, Zehuan Huang +43D Scene Generation Autoregressive Generation
School of Software, Beihang University · 2Tsinghua University · University of Hong Kong +2

Jul 15, 2026J/K move · Enter open · S save

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

Yuan Xiao, Can Wang, Xiangyu Kong, Jing Liao

Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University

3D Scene Editing 3D Scene Generation 3D Scene Layout Generation Indoor Scene Generation

Open paper AskPDF arXiv

While traditional graphics methods often synthesize 3D indoor scenes autoregressively or hierarchically, recent vision-language model (VLM)-based generators predominantly adopt a one-shot paradigm where the full layout is planned at once. This one-shot approach often requires global re-optimization or complete reconstruction during interactive editing (e.g., inserting or moving objects) and can lead to physically or semantically poorly organized arrangements. To address these challenges, we propose ThinkBLOX, a VLM-based progressive reasoning framework that iteratively designs and refines 3D scenes. ThinkBLOX treats layout generation as a state-conditioned, step-by-step reasoningand-action process. To power this, we construct the ThinkBLOX-Data-200K dataset, containing 224,757 procedural placement pairs annotated with multi-view scene context, explicit Chain-of-Thought (CoT) rationales, and structured JSON layouts. Through supervised fine-tuning (SFT) on this dataset, the VLM learns to bridge the reasoning-action gap under incremental updates. Furthermore, recognizing that scene synthesis is inherently a multisolution task where SFT suffers from reward conflict, we introduce Tier-Decoupled GDPO. This reinforcement learning scheme organizes heterogeneous rewards into distinct tiers, stabilizing policy optimization across physical validity, semantic plausibility, and reasoning-action consistency. Extensive experiments show that ThinkBLOX significantly outperforms recent one-shot and iterative baselines in physical plausibility, semantic alignment, and interactive editability. Additionally, we show that it supports diverse applications, including both global and local generation and rearrangement of 3D scenes.

Abstract

Explore similar work

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editing

Repurposing 3D Generative Model for Autoregressive Layout Generation