3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation

Shuqing Li; Anson Y. Lam; Yun Peng; Wenxuan Wang; Michael R. Lyu

cs.CVJul 24, 2025

3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation

Authors: Shuqing Li, Anson Y. Lam, Yun Peng, Wenxuan Wang, Michael R. Lyu

Organizations: The Chinese University of Hong Kong Hong Kong, China · Renmin University of China Beijing, China

Abstract

Graphical user interface (UI) software has undergone a fundamental transformation from traditional two-dimensional (2D) desktop/web/mobile interfaces to spatial three-dimensional (3D) environments. While existing work has made remarkable success in automated 2D software generation, such as HTML/CSS and mobile app interface code synthesis, the generation of 3D software still remains under-explored. Current methods for 3D software generation usually generate the 3D environments as a whole and cannot modify or control specific elements in the software. Furthermore, these methods struggle to handle the complex spatial and semantic constraints inherent in the real world. To address the challenges, we present Scenethesis, a novel requirement-sensitive 3D software synthesis approach that maintains formal traceability between user specifications and generated 3D software. Scenethesis is built upon ScenethesisLang, a domain-specific language that serves as a granular constraint-aware intermediate representation (IR) to bridge natural language requirements and executable 3D software. It serves both as a comprehensive scene description language enabling fine-grained modification of 3D software elements and as a formal constraint-expressive specification language capable of expressing complex spatial constraints. By decomposing 3D software synthesis into stages operating on ScenethesisLang, Scenethesis enables independent verification, targeted modification, and systematic constraint satisfaction. Our evaluation demonstrates that Scenethesis accurately captures over 80% of user requirements and satisfies more than 90% of hard constraints while handling over 100 constraints simultaneously. Furthermore, Scenethesis achieves a 42.8% improvement in BLIP-2 visual evaluation scores compared to the state-of-the-art method.

Figures & tables

(1)	$\displaystyle program\in\mathit{Programs}::=$	$\displaystyle\ stmt;\ \|\ stmt;\ program$
(2)	$\displaystyle stmt\in\mathit{Statements}::=$	$\displaystyle\ \mathit{decl}\ \|\ \mathit{const}\ \|\ assign$
(3)	$\displaystyle decl\in\mathit{Declarations}::=$	$\displaystyle\ \textbf{object}\;id\ \|\ \textbf{region}\;id\ \|\ \tau\ id$
(4)	$\displaystyle const\in\mathit{Constraints}::=$	$\displaystyle\ \textbf{assert}\;\phi\ \|\ \textbf{allowCollide}(id,id)\ \|\ \textbf{allowOutside}(id)$
(5)	$\displaystyle assign\in\mathit{Assignments}::=$	$\displaystyle\ id.\alpha\leftarrow e\ \|\ id.\beta\leftarrow e\ \|\ id\leftarrow e$
(6)	$\displaystyle\alpha\in\mathit{Object\ Properties}::=$	$\displaystyle\ \textbf{color}\mid\textbf{material}\mid\textbf{features}$
(7)	$\displaystyle\beta\in\mathit{Region\ Properties}::=$	$\displaystyle\ \textbf{pos}\mid\textbf{rot}\mid\textbf{scale}\mid$

Figure 2. Domain-specific language: ScenethesisLang

Figure 3. Rubik Spatial Constraint Solver

	Object Constraints			Layout Constraints			Overall
Model	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Threshold $\tau_{o}=\tau_{l}=0.7$
GPT-4o	99.2	98.0	98.5	98.3	80.3	86.4	98.7	88.3	92.1
Gemini-2.5-Pro	97.9	98.7	98.2	89.3	99.9	93.8	93.4	99.3	95.9
DeepSeek R1	99.1	95.9	96.7	97.1	97.1	97.1	98.1	96.5	96.7
Threshold $\tau_{o}=\tau_{l}=0.8$

Table 1. Requirement formalization performance (%) across object constraints, layout constraints, and overall (harmonic mean). Best results in each metric are in bold .

Method	BLIP-2	CLIP	Mean
Retrieval only ( $\tau=0.0$ )	51.2	27.1	39.1
Generation only ( $\tau=1.0$ )	42.2	25.9	34.1
R+G ( $\tau=0.652$ )	51.6	27.1	39.3

Table 2. Object synthesis performance (%) comparing pure retrieval, pure generation, and our hybrid approach (R+G). Scores represent object-query coherence.

Model	Iter 0	Iter 1	Iter 2	Iter 3	Iter 4	Iter 5
GPT-4o	47.1	60.4	63.2	65.6	67.3	68.3
Gemini-2.5-Pro	74.1	91.1	92.5	93.8	93.5	93.4
DeepSeek R1	47.8	87.6	90.6	91.7	92.9	93.0

Table 3. Spatial constraint solving performance across iterations. Scores (%) represent the ratio of satisfied constraints (solution correctness).

Method	BLIP-2		CLIP		VQA
Method	O	S	O	S	O	S
Before	74.3	75.1	26.1	25.5	29.5	47.9
After	75.1	71.3	26.8	27.0	49.6	59.4

Table 4. Overall performance (%) before and after targeted modification. “O”

\rightarrow

“Original”, “S”

\rightarrow

“Sentence”.

Method	LLM Backbone	BLIP-2		CLIP		VQA
Method	LLM Backbone	O	S	O	S	O	S
Scenethesis (Ours)	GPT-4o	71.3	69.9	25.6	25.5	28.3	44.8
	Gemini 2.5 Pro	74.3	75.1	26.1	25.5	29.5	47.9
	DeepSeek R1	72.5	74.7	26.2	25.8	29.8	48.6
End-to-end LLM	GPT-4o	61.9	60.0	24.7	24.0	15.1	28.6
	Gemini 2.5 Pro	71.6	73.2	25.6	25.3	27.1	41.1

Table 5. Overall performance (%) comparison against baselines. Best results for each metric are in bold , second-best are underlined . “O”

\rightarrow

“Original”, “S”

\rightarrow

“Sentence”.

Method	Dataset	BLIP-2	CLIP	VQA
Scenethesis (Ours)	MIT-ST	66.4	26.8	44.7
	MIT-IC	78.4	24.0	13.6
	IS	58.8	25.5	36.5
End-to-end LLM	MIT-ST	60.2	25.5	39.3
	MIT-IC	69.5	22.7	8.8
	IS	65.0	23.9	34.7

Table 6. Overall performance (%) on three external datasets.

Method	BLIP-2		CLIP		VQA
Method	O	S	O	S	O	S
Scenethesis	74.3	75.1	26.1	25.5	29.5	47.9
w/o Stage I	57.6	63.8	21.7	21.9	13.7	30.2
w/o Stage II	48.8	48.0	22.5	22.5	18.3	32.7
w/o Stage III	61.3	61.8	23.4	22.4	26.3	41.0

Table 7. Ablation study. Values are percentage scores (%).

Method	Layout	Spatial	Overall
	Coherence	Realism	Consistency
Scenethesis (Ours)	4.12	3.89	4.05
End-to-end LLM	3.45	3.21	3.38
Holodeck	3.68	3.42	3.61

Table 8. User study results (mean scores ± std. dev.) comparing perceived quality of 3D software. All scores on 1-5 scale, higher is better. Best results in bold .

Explore similar work

MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction
Jun 12, 2026Ruijie Xu, Xinnan Zhu, Jiayu Ying +33D Scene Generation Scene Understanding
1East China Normal University · 2Fudan University · 3Shanghai Jiao Tong University
ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning
Jul 15, 2026Yuan Xiao, Can Wang, Xiangyu Kong +13D Scene Generation 3D Scene Understanding
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University
SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation
Apr 30, 2026Song Tang, Kaiyong Zhao, Yuliang Li +53D Scene Generation 3D Layout Generation
The Hong Kong University of Science and Technology (Guangzhou) · XGRIDS · Harbin Institute of Technology (Shenzhen)

Jun 12, 2026, cs.CVJ/K move · Enter open · S save

MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction

Ruijie Xu, Xinnan Zhu, Jiayu Ying, Daoguo Dong+2

1East China Normal University · 2Fudan University · 3Shanghai Jiao Tong University

3D Scene Generation Scene Understanding

Open paper AskPDF arXiv

Text-driven 3D scene generation is a promising technique for digital content creation, embodied AI simulation, and interactive design, yet practical workflows often require refining, extending, or correcting existing scenes while preserving non-target content. Existing methods can produce realistic and structurally plausible scenes, but they generally lack editability with requirement-level state tracking, so part-level failures often lead to full-scene regeneration or manual intervention. To tackle this challenge, we formulate controllable 3D scene authoring as incremental requirement satisfaction, unifying construction and editing. In this paper, we present MUSE, a memory-grounded multi-agent framework in which an Architect compiles instructions into structured requirements, a Sculptor executes local scene operations, and an Inspector verifies each step while updating Working, Scene, and Skill Memory. To evaluate requirement-level controllability and preservation-aware editing, we introduce AuthorBench, offering 145 constrained construction cases and a 1,584-case preservation-aware editing pool paired with external structured checks. On full construction cases, MUSE improves All-Goal success from 37.9 to 80.7 and surface-constraint fulfillment from 35.0 to 92.6 over the strongest baseline. On a stratified 240-case editing test split, MUSE achieves 49.6 All-Goal success, 99.9 preservation rate, and only 0.6 unintended change rate. Beyond automated metrics, human evaluations on compared local-editing baselines support stronger alignment with user intent, and downstream navigation-proxy tests indicate stronger spatial stability. Combined with ablations validating our memory designs, these results establish MUSE as an effective framework for controllable 3D scene authoring.

Abstract

Explore similar work

MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation