3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation

Shuqing Li; Anson Y. Lam; Yun Peng; Wenxuan Wang; Michael R. Lyu

cs.CVJul 24, 2025

3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation

Authors: Shuqing Li, Anson Y. Lam, Yun Peng, Wenxuan Wang, Michael R. Lyu

Organizations: The Chinese University of Hong Kong Hong Kong, China · Renmin University of China Beijing, China

Abstract

Graphical user interface (UI) software has undergone a fundamental transformation from traditional two-dimensional (2D) desktop/web/mobile interfaces to spatial three-dimensional (3D) environments. While existing work has made remarkable success in automated 2D software generation, such as HTML/CSS and mobile app interface code synthesis, the generation of 3D software still remains under-explored. Current methods for 3D software generation usually generate the 3D environments as a whole and cannot modify or control specific elements in the software. Furthermore, these methods struggle to handle the complex spatial and semantic constraints inherent in the real world. To address the challenges, we present Scenethesis, a novel requirement-sensitive 3D software synthesis approach that maintains formal traceability between user specifications and generated 3D software. Scenethesis is built upon ScenethesisLang, a domain-specific language that serves as a granular constraint-aware intermediate representation (IR) to bridge natural language requirements and executable 3D software. It serves both as a comprehensive scene description language enabling fine-grained modification of 3D software elements and as a formal constraint-expressive specification language capable of expressing complex spatial constraints. By decomposing 3D software synthesis into stages operating on ScenethesisLang, Scenethesis enables independent verification, targeted modification, and systematic constraint satisfaction. Our evaluation demonstrates that Scenethesis accurately captures over 80% of user requirements and satisfies more than 90% of hard constraints while handling over 100 constraints simultaneously. Furthermore, Scenethesis achieves a 42.8% improvement in BLIP-2 visual evaluation scores compared to the state-of-the-art method.

Figures & tables

(1)	$\displaystyle program\in\mathit{Programs}::=$	$\displaystyle\ stmt;\ \|\ stmt;\ program$
(2)	$\displaystyle stmt\in\mathit{Statements}::=$	$\displaystyle\ \mathit{decl}\ \|\ \mathit{const}\ \|\ assign$
(3)	$\displaystyle decl\in\mathit{Declarations}::=$	$\displaystyle\ \textbf{object}\;id\ \|\ \textbf{region}\;id\ \|\ \tau\ id$
(4)	$\displaystyle const\in\mathit{Constraints}::=$	$\displaystyle\ \textbf{assert}\;\phi\ \|\ \textbf{allowCollide}(id,id)\ \|\ \textbf{allowOutside}(id)$
(5)	$\displaystyle assign\in\mathit{Assignments}::=$	$\displaystyle\ id.\alpha\leftarrow e\ \|\ id.\beta\leftarrow e\ \|\ id\leftarrow e$
(6)	$\displaystyle\alpha\in\mathit{Object\ Properties}::=$	$\displaystyle\ \textbf{color}\mid\textbf{material}\mid\textbf{features}$
(7)	$\displaystyle\beta\in\mathit{Region\ Properties}::=$	$\displaystyle\ \textbf{pos}\mid\textbf{rot}\mid\textbf{scale}\mid$

Figure 2. Domain-specific language: ScenethesisLang

Figure 3. Rubik Spatial Constraint Solver

	Object Constraints			Layout Constraints			Overall
Model	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Threshold $\tau_{o}=\tau_{l}=0.7$
GPT-4o	99.2	98.0	98.5	98.3	80.3	86.4	98.7	88.3	92.1
Gemini-2.5-Pro	97.9	98.7	98.2	89.3	99.9	93.8	93.4	99.3	95.9
DeepSeek R1	99.1	95.9	96.7	97.1	97.1	97.1	98.1	96.5	96.7
Threshold $\tau_{o}=\tau_{l}=0.8$

Table 1. Requirement formalization performance (%) across object constraints, layout constraints, and overall (harmonic mean). Best results in each metric are in bold .

Method	BLIP-2	CLIP	Mean
Retrieval only ( $\tau=0.0$ )	51.2	27.1	39.1
Generation only ( $\tau=1.0$ )	42.2	25.9	34.1
R+G ( $\tau=0.652$ )	51.6	27.1	39.3

Table 2. Object synthesis performance (%) comparing pure retrieval, pure generation, and our hybrid approach (R+G). Scores represent object-query coherence.

Model	Iter 0	Iter 1	Iter 2	Iter 3	Iter 4	Iter 5
GPT-4o	47.1	60.4	63.2	65.6	67.3	68.3
Gemini-2.5-Pro	74.1	91.1	92.5	93.8	93.5	93.4
DeepSeek R1	47.8	87.6	90.6	91.7	92.9	93.0

Table 3. Spatial constraint solving performance across iterations. Scores (%) represent the ratio of satisfied constraints (solution correctness).

Method	BLIP-2		CLIP		VQA
Method	O	S	O	S	O	S
Before	74.3	75.1	26.1	25.5	29.5	47.9
After	75.1	71.3	26.8	27.0	49.6	59.4

Table 4. Overall performance (%) before and after targeted modification. “O”

\rightarrow

“Original”, “S”

\rightarrow

“Sentence”.

Method	LLM Backbone	BLIP-2		CLIP		VQA
Method	LLM Backbone	O	S	O	S	O	S
Scenethesis (Ours)	GPT-4o	71.3	69.9	25.6	25.5	28.3	44.8
	Gemini 2.5 Pro	74.3	75.1	26.1	25.5	29.5	47.9
	DeepSeek R1	72.5	74.7	26.2	25.8	29.8	48.6
End-to-end LLM	GPT-4o	61.9	60.0	24.7	24.0	15.1	28.6
	Gemini 2.5 Pro	71.6	73.2	25.6	25.3	27.1	41.1

Table 5. Overall performance (%) comparison against baselines. Best results for each metric are in bold , second-best are underlined . “O”

\rightarrow

“Original”, “S”

\rightarrow

“Sentence”.

Method	Dataset	BLIP-2	CLIP	VQA
Scenethesis (Ours)	MIT-ST	66.4	26.8	44.7
	MIT-IC	78.4	24.0	13.6
	IS	58.8	25.5	36.5
End-to-end LLM	MIT-ST	60.2	25.5	39.3
	MIT-IC	69.5	22.7	8.8
	IS	65.0	23.9	34.7

Table 6. Overall performance (%) on three external datasets.

Method	BLIP-2		CLIP		VQA
Method	O	S	O	S	O	S
Scenethesis	74.3	75.1	26.1	25.5	29.5	47.9
w/o Stage I	57.6	63.8	21.7	21.9	13.7	30.2
w/o Stage II	48.8	48.0	22.5	22.5	18.3	32.7
w/o Stage III	61.3	61.8	23.4	22.4	26.3	41.0

Table 7. Ablation study. Values are percentage scores (%).

Method	Layout	Spatial	Overall
	Coherence	Realism	Consistency
Scenethesis (Ours)	4.12	3.89	4.05
End-to-end LLM	3.45	3.21	3.38
Holodeck	3.68	3.42	3.61

Table 8. User study results (mean scores ± std. dev.) comparing perceived quality of 3D software. All scores on 1-5 scale, higher is better. Best results in bold .

Explore similar work

Jun 12, 2026cs.CV

MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction

Text-driven 3D scene generation is a promising technique for digital content creation, embodied AI simulation, and interactive design, yet practical workflows often require refining, extending, or correcting existing scenes while preserving non-target content. Existing methods can produce realistic and structurally plausible scenes, but they generally lack editability with requirement-level state tracking, so part-level failures often lead to full-scene regeneration or manual intervention. To tackle this challenge, we formulate controllable 3D scene authoring as incremental requirement satisfaction, unifying construction and editing. In this paper, we present MUSE, a memory-grounded multi-agent framework in which an Architect compiles instructions into structured requirements, a Sculptor executes local scene operations, and an Inspector verifies each step while updating Working, Scene, and Skill Memory. To evaluate requirement-level controllability and preservation-aware editing, we introduce AuthorBench, offering 145 constrained construction cases and a 1,584-case preservation-aware editing pool paired with external structured checks. On full construction cases, MUSE improves All-Goal success from 37.9 to 80.7 and surface-constraint fulfillment from 35.0 to 92.6 over the strongest baseline. On a stratified 240-case editing test split, MUSE achieves 49.6 All-Goal success, 99.9 preservation rate, and only 0.6 unintended change rate. Beyond automated metrics, human evaluations on compared local-editing baselines support stronger alignment with user intent, and downstream navigation-proxy tests indicate stronger spatial stability. Combined with ablations validating our memory designs, these results establish MUSE as an effective framework for controllable 3D scene authoring.

Ruijie Xu, Xinnan Zhu, Jiayu Ying +3

1East China Normal University · 2Fudan University · 3Shanghai Jiao Tong University

3D Scene Generation Scene Understanding

Ask PDF

Jul 15, 2026cs.CV

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

While traditional graphics methods often synthesize 3D indoor scenes autoregressively or hierarchically, recent vision-language model (VLM)-based generators predominantly adopt a one-shot paradigm where the full layout is planned at once. This one-shot approach often requires global re-optimization or complete reconstruction during interactive editing (e.g., inserting or moving objects) and can lead to physically or semantically poorly organized arrangements. To address these challenges, we propose ThinkBLOX, a VLM-based progressive reasoning framework that iteratively designs and refines 3D scenes. ThinkBLOX treats layout generation as a state-conditioned, step-by-step reasoningand-action process. To power this, we construct the ThinkBLOX-Data-200K dataset, containing 224,757 procedural placement pairs annotated with multi-view scene context, explicit Chain-of-Thought (CoT) rationales, and structured JSON layouts. Through supervised fine-tuning (SFT) on this dataset, the VLM learns to bridge the reasoning-action gap under incremental updates. Furthermore, recognizing that scene synthesis is inherently a multisolution task where SFT suffers from reward conflict, we introduce Tier-Decoupled GDPO. This reinforcement learning scheme organizes heterogeneous rewards into distinct tiers, stabilizing policy optimization across physical validity, semantic plausibility, and reasoning-action consistency. Extensive experiments show that ThinkBLOX significantly outperforms recent one-shot and iterative baselines in physical plausibility, semantic alignment, and interactive editability. Additionally, we show that it supports diverse applications, including both global and local generation and rearrangement of 3D scenes.

Yuan Xiao, Can Wang, Xiangyu Kong +1

Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computing and Data Science, University of Hong Kong · School of Computer Science, Beijing Information Science and Technology University

3D Scene Generation 3D Scene Understanding Motion Generation Recent Vision-Language Models Progressive Reasoning

Ask PDF

Apr 30, 2026cs.AI

SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation

Automatically generating interactive 3D indoor scenes from natural language is crucial for virtual reality, gaming, and embodied AI. However, existing LLM-based approaches often suffer from spatial errors and collisions, in part because common scene representations-raw coordinates or verbose code-are difficult for models to reason about 3D spatial relationships and physical constraints. We propose SpatialGrammar, a domain-specific language that represents gravity-aligned indoor layouts as BEV grid placements with deterministic compilation to valid 3D geometry, enabling verifiable constraint checking. Building on this representation, we develop (1) SG-Agent, a closed-loop system that uses compiler feedback to iteratively refine scenes and enforce collision constraints, and (2) SG-Mini, a 104M-parameter model trained entirely on compiler-validated synthetic data. Across 159 test scenes spanning five scenarios of different complexity, SG-Agent improves spatial fidelity and physical plausibility over prior methods, while SG-Mini performs competitively against larger LLM-based baselines on single-shot generation scenarios.

Song Tang, Kaiyong Zhao, Yuliang Li +5

The Hong Kong University of Science and Technology (Guangzhou) · XGRIDS · Harbin Institute of Technology (Shenzhen)

3D Scene Generation 3D Layout Generation 3D Spatial Reasoning 3D Generation Scenario Generation Domain-Specific Language+1

Ask PDF