Tokenization Allows Multimodal Large Language Models to Understand, Generate and Edit Architectural Floor Plans

Sizhong Qin; Ramon Elias Weber; Xinzheng Lu

cs.CVMar 12, 2026

Tokenization Allows Multimodal Large Language Models to Understand, Generate and Edit Architectural Floor Plans

Authors: Sizhong Qin, Ramon Elias Weber, Xinzheng Lu

Organizations: Tsinghua University · UC Berkeley

Abstract

Architectural floor plan design demands joint reasoning over geometry, semantics, and spatial hierarchy, which remains a major challenge for current AI systems. Although recent diffusion and language models improve visual fidelity, they still struggle with coherent spatial reasoning and controllable generation. We present HouseMind, a multimodal large language model that unifies floor plan understanding, generation, and editing in one framework. We introduce discrete room-instance tokens to construct a unified vocabulary that bridges layouts and symbolic reasoning. With multimodal alignment and instruction tuning, the model synthesizes coherent, controllable layouts from text instructions. Experiments show how the framework achieves superior geometric validity and controllability while remaining efficient and locally deployable.

Figures & tables

Figure 2: Understanding : given a prompt, an outline, and an existing floor plan, the model outputs a textual description, a bubble diagram, and structured JSON capturing spatial semantics. Generation : given a prompt and an outline, the model produces a complete, coherent floor plan. Editing : given a prompt, an outline, and a reference floor plan, the model outputs an updated plan aligned with the editing intent.

Figure 3: Overall framework of HouseMind. The model is trained through a three-stage multimodal alignment and instruction tuning pipeline: (S1) Embedding Initialization establishes cross-modal compatibility between geometric and linguistic tokens; (S2) Multimodal Pre-training aligns text and spatial representations; and (S3) Instruction Tuning (SFT) enables task-aware spatial reasoning.

Method	Success	RMR	LocAcc	AreaDiff $\downarrow$	AdjAcc	RelAcc	Time (s)
LLaVA-v1.6-Mistral-7B-HF	1.000	0.616	0.225	3.649	0.134	0.056	$\sim$ 6
Qwen3-VL-8B-Instruct	1.000	0.698	0.347	5.837	0.382	0.128	$\sim$ 8
InternVL3.5-8B	1.000	0.847	0.546	12.234	0.469	0.157	$\sim$ 13
MiniCPM-V 4.5	0.996	0.904	0.492	13.765	0.597	0.208	$\sim$ 14
HouseMind-U	1.000	0.998	0.969	0.549	0.990	0.808	$\sim$ 3
HouseMind-O	1.000	0.998	0.925	0.655	0.954	0.738	$\sim$ 3

Table 1: Understanding results. Success: success rate; RMR: room match rate; LocAcc: room location accuracy; AreaDiff: room area difference (m 2 ); AdjAcc: room adjacency accuracy; RelAcc: spatial relation accuracy.

Method	Micro IoU	Macro IoU	SSIM	PSNR	FID $\downarrow$	GED $\downarrow$	Node F1	Edge Ovl.	Time (s)
Qwen-Image-Edit-2509	0.161	0.0621	0.721	12.4	156	–	–	–	$\sim$ 240
Tell2Design	0.390	0.307	0.840	13.2	30.5	6.94	0.808	0.197	$\sim$ 15
ChatHouseDiffusion	0.589	0.521	0.866	14.9	11.3	2.36	0.985	0.710	$\sim$ 30
FloorPlanLLaMA*	0.607	0.511	0.874	15.5	49.3	2.68	0.922	0.574	$\sim$ 1
HouseMind-G	0.709	0.653	0.886	16.0	1.91	1.01	0.994	0.880	$\sim$ 2
HouseMind-O	0.710	0.654	0.887	16.1	1.89	1.03	0.994	0.880	$\sim$ 2

Table 2: Generation results. Micro/Macro IoU measure pixel-level overlap; SSIM [ 40 ] and PSNR quantify perceptual similarity; FID [ 6 ] and GED [ 30 ] evaluate distributional realism; Node F1 and Edge Overlap assess graph-level correctness. * denotes methods without released code; results are reproduced.

Method	$\Delta$ IoU	$\Delta$ MSE $\downarrow$	Micro IoU	Macro IoU	GED $\downarrow$	Node F1	Edge Ovl.	Time (s)
Before edit	–	–	0.880	0.821	3.06	0.934	0.740	–
FLUX.1-Kontext-dev	0.053	0.0162	0.289	0.185	8.91	0.765	0.222	$\sim$ 240
Qwen-Image-Edit-2509	0.088	0.0074	0.567	0.429	7.96	0.915	0.426	$\sim$ 240
HouseMind-E	0.608	0.0019	0.855	0.823	0.467	0.998	0.934	$\sim$ 3
HouseMind-O	0.598	0.0022	0.844	0.813	0.653	0.997	0.908	$\sim$ 3

Table 3: Editing results.

\Delta

IoU and

\Delta

MSE measure editing precision (spatial and pixel-level change correctness); Micro/Macro IoU assess final layout quality; GED evaluates distributional realism; Node F1 and Edge Overlap assess graph-level consistency.

Figure 4: Qualitative comparison results of understanding, generation, and editing. For understanding tasks (U), HouseMind accurately identifies the number of rooms and their connections. For generation tasks (G), HouseMind preserves both the room layout and the overall outline consistency; the generation prompts are provided in the supplementary materials. For editing tasks (E), HouseMind accurately executes the specified modifications when the instructions are explicit.

Model	Train Loss $\downarrow$	Eval Loss $\downarrow$
w/o Stage 1 & 2	0.0729	0.0836
w/o Stage 1	0.0659	0.0840
w/o Stage 2	0.0712	0.0831
Full	0.0644	0.0830

Table 4: Loss under different training-stage configurations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : VQ-VAE tokenization framework. The outline branch encodes the global boundary, while the conditional room branch encodes each room with outline context to capture spatial relations.

$\mathbf{N_{\text{{tokens}}}}$	Codebook	PSNR	SSIM
$4\times 4$	256	20.003	0.911
$4\times 4$	512	22.188	0.946
$4\times 4$	1024	23.202	0.957
$8\times 8$	256	35.898	0.998
$8\times 8$	512	36.657	0.998
$8\times 8$	1024	37.053	0.998

Appendix

Table A.1 : Outline branch: PSNR (dB) and SSIM under different token grids and codebook sizes.

$\mathbf{N_{\text{{tokens}}}}$	Codebook	PSNR	SSIM
$4\times 4$	256	31.896	0.994
$4\times 4$	512	33.106	0.995
$4\times 4$	1024	34.506	0.997
$8\times 8$	256	44.624	1.000
$8\times 8$	512	48.467	1.000
$8\times 8$	1024	51.769	1.000

Appendix

Table A.2 : Room branch: PSNR (dB) and SSIM under different token grids and codebook sizes.

Size	Ma. IoU	FID $\downarrow$	GED $\downarrow$	Node F1	Edge Ovl.
256	0.654	1.89	1.03	0.994	0.880
512	0.657	1.95	1.04	0.995	0.877
1024	0.657	1.97	1.00	0.994	0.881

Appendix

Table A.3 : Codebook size ablation study

Metric	w/o Correction	w/ Correction
Micro IoU	0.710	0.710
Macro IoU	0.654	0.654
SSIM	0.887	0.887
PSNR	16.0	16.1
FID $\downarrow$	1.87	1.89
GED $\downarrow$	1.10	1.03

Appendix

Table C.1 : Impact of Structural Correction in HouseMind-O .

Figure D.1 : Result distribution across tasks. HouseMind-O maintains stable, high-performance distributions in understanding and generation, with the bimodal $\Delta\text{IoU}$ in editing mainly caused by unclear spatial instructions.

Figure D.2 : Pixel–structure coupling analysis. Scatter plots show the correlation between Macro IoU and Edge Overlap for three generation paradigms.

Figure F.1 : Results of HouseMind on out-of-distribution cases.

Explore similar work

HypergraphFormer: Learning Hypergraphs from LLMs for Editable Floor Plan Generation
May 18, 2026Nikita Klimenko, Hesam Salehipour, Parham Eftekhar +2Indoor Scene Generation Hypergraph Learning
Autodesk Research · York University · UC Berkeley
Generative Floor Plan Design with LLMs via Reinforcement Learning with Verifiable Rewards
May 13, 2026Luis Lara, Aristides Milios, Zhi Hao Luo +5Constrained Generative Modeling Generative Modeling
Mila – Quebec AI Institute · Université de Montréal · Polytechnique Montréal +1
PlanCraft: Sketch, Refine, and Furnish for Architect-Inspired Progressive 3D Residential Scene Generation
Jul 26, 2026Pengyu Zeng, Yuqin Dai, Jun Yin +73D Scene Generation 3D Scene Layout Generation
Tsinghua University

May 18, 2026J/K move · Enter open · S save

HypergraphFormer: Learning Hypergraphs from LLMs for Editable Floor Plan Generation

Nikita Klimenko, Hesam Salehipour, Parham Eftekhar, Amir Khasahmadi+1

Autodesk Research · York University · UC Berkeley

Indoor Scene Generation Hypergraph Learning

Open paper AskPDF arXiv

In this work, we propose HypergraphFormer, a novel and efficient approach to floor plan generation based on learning hypergraph representations with a large language model (LLM). The model is trained via supervised fine-tuning to generate a hypergraph-based textual representation that encodes spatial relationships and connectivity information within floor plans. We train and evaluate our approach on the RPLAN dataset, and further demonstrate its generalizability on a separate out-of-distribution dataset, which we release in this paper. Our method outperforms state-of-the-art techniques based on rasterized or vectorized representations across a diverse set of metrics. We also show improved data efficiency, particularly under distribution shift. The hypergraph formulation enables the generation of floor plans for arbitrary, irregular, user-specified boundaries by decoupling apartment footprints from their functional and geometric subdivisions. Furthermore, we show that the proposed methodology offers a high degree of editability, making it particularly well suited to design-oriented workflows supported by LLMs.

Abstract

Explore similar work

HypergraphFormer: Learning Hypergraphs from LLMs for Editable Floor Plan Generation

Generative Floor Plan Design with LLMs via Reinforcement Learning with Verifiable Rewards

PlanCraft: Sketch, Refine, and Furnish for Architect-Inspired Progressive 3D Residential Scene Generation