Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

Xiaokang Ye; Siddhant Hitesh Mantri; Zimeng Chen; Edward Zhang; Zhaoxu Zheng; Yuanheng Li; Yizhao Chen; Tianyang Huang

cs.AISep 29, 2026

Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

Authors: Xiaokang Ye, Siddhant Hitesh Mantri, Zimeng Chen, Edward Zhang, Zhaoxu Zheng, Yuanheng Li, Yizhao Chen, Tianyang Huang, +1 more

Organizations: UC San Diego · UC Berkeley

Abstract

Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear. The generated 3D scene is a persistent, executable artifact: a convincing render can hide incorrect spatial relations, intersecting objects, or unintended modifications. We introduce Code4Scene, a benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface. Construction tests scene-level spatial reasoning from open-ended language specifications, where many realizations are valid; editing tests precise control of scene state, where the agent must recover the target scene from reference images while preserving everything else. Rather than scoring code or rendered views, Code4Scene evaluates the generated engine-native scene for task fulfillment, artifact integrity, and static physical validity, with edits additionally compared against withheld ground truth. Across 14 coding-agent configurations on the 95-case public set, construction and editing performance are strongly correlated but not interchangeable (Spearman $ρ= 0.78$ ): Claude Fable 5.1 leads construction, Gemini 3.8 Flash leads editing, and GPT-6 Astra narrowly leads overall. Spatial Composition is the weakest construction category for every agent, while editing remains imprecise: the best Repair F1 is only 0.527, and 35.8% of edits that fully recover the target still introduce unintended changes elsewhere in the scene. These results expose a gap between plausible 3D generation and reliable spatial reasoning and state control.

Figures & tables

Figure 1: Code4Scene tasks and leaderboard. (a) Coding agents use Python and tool calls to construct or edit persistent Unreal Engine scenes based on text prompt or image reference. (b) Overall score (Table 2 ) for agent configurations against estimated cost per case on the public set (Appendix D.5 ). The solid line marks the Pareto frontier among the displayed agent configurations.

Benchmark	Artifact	Input		Task Setting		Evaluation
		Text	Images	Open-ended	GT-based	Semantic	Physical	Preservation
BlenderGym ( 2025 )	Script		$\checkmark$		$\checkmark$	$\checkmark$
SceneActBench ( 2026 )	GLB/pose		$\checkmark$		$\checkmark$	$\checkmark$
4DBuildBench ( 2026a )	.blend	$\checkmark$		$\checkmark$		$\checkmark$	$\checkmark$
CutsceneBench ( 2026 )	Sequence	$\checkmark$		$\checkmark$	$\checkmark$	$\checkmark$
WorldCoder-Bench ( 2026 )	HTML	$\checkmark$		$\checkmark$		$\checkmark$	$\checkmark$

Table 1: Comparison of related benchmarks and their reported coverage. Input references specify the desired outcome and exclude initial-scene observations. Open-ended settings admit multiple valid scene realizations; ground-truth-based (GT-based) settings use a hidden target state for evaluation. Full related work discussions appear in Appendix E .

Figure 2: Task construction. Text-to-Scene prompts describe human-assembled scenes. Image-to-Scene inputs are corrupted copies, with the human-assembled scene retained as ground truth. Human review checks task solvability in both settings.

Figure 3: Evaluation pipeline. Both settings share Candidate Integrity, Physical Safety, and Agentic Verification. Text-to-Scene uses semantic alignment, whereas Image-to-Scene measures target recovery and scene preservation against withheld ground truth.

Agent configuration	Text-to-Scene (20 cases)			Image-to-Scene (75 cases)			Overall
	Semantic $\uparrow$	Physical $\uparrow$	$S_{\mathrm{T2S}}\uparrow$	Repair F1 $\uparrow$	Physical $\uparrow$	$S_{\mathrm{I2S}}\uparrow$	$S_{\mathrm{model}}\uparrow$
GPT-6 Astra (max)	0.727	0.712	0.724	0.445	0.796	0.515	0.619
Gemini 3.8 Flash (high)	0.657	0.655	0.657	0.527	0.796	0.581	0.619
Claude Fable 5.1 (max)	0.760	0.898	0.788	0.332	0.795	0.424	0.606
Claude Opus 5 (max)	0.695	0.811	0.718	0.389	0.786	0.468	0.593
GPT-5.6 Sol (high)	0.691	0.772	0.707	0.319	0.689	0.393	0.550

Table 2: Main results on the public and private sets. Scores lie in

[0,1]

(

\uparrow

). Each setting averages its scheduled cases;

S_{\mathrm{model}}

weights the two settings equally. Rows follow

S_{\mathrm{model}}

within each set; dark/light shading marks the best/second-best values. Metrics and zero-score rules are in Appendix C .

Figure 4: Construction and evaluation of a chemical-plant scene generated by Claude Fable 5.1 (max). (a) Scene construction. Given an asset catalog and a task prompt, the coding agent uses code and tools to retrieve assets, plan the layout, and construct the scene. It repeatedly inspects and refines the scene before submission. (b) Evaluation. The generated scene must pass Candidate Integrity checks before undergoing evaluation by the Semantic Verifier and Physical Safety checks. Selected evidence is shown beside the corresponding component scores, which contribute to the final Text-to-Scene score. The bottom inset contrasts physical failures in a separate scene generated by Inkling (high); its fractions summarize eligible evaluated actors, not only the highlighted examples.

Panel of Figure 5 — Figure 5: Text-to-Scene evaluator agreement on the retrospective human-study holdout.

Figure 6: (a) Spatial errors with required objects present. (b) Mean I2S score by repair type for four agents; orange marks the best of the four. (c) Complete recoveries with or without unintended edits.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Tool (arguments)	Result
Asset discovery	search_assets(query ∗ , category, k)	Ranked matching assets
	get_categories()	Asset categories with counts and descriptions
	get_asset_metadata(ids ∗ )	Category, description, tags, dimensions in metres, spawn path
	get_asset_preview(id ∗ )	Rendered preview image, if available
Scene editing	spawn_actor(name ∗ , static_mesh ∗ , location ∗ , rotation, scale)	Spawns a static-mesh actor
	spawn_blueprint_actor(actor_name ∗ , blueprint_id ∗ , location ∗ , rotation, scale)	Spawns a Blueprint actor

Appendix

Table 3: The 25 shared agent tools. Arguments marked ∗ are required; all others are optional.

Cases	T2S/Indoor/ Outdoor	Kept	Total error	Component error	Reversed pairs
61	16/25/20	32.1%	0.016	0.025	8
75	20/25/30	39.5%	0.014	0.019	1
95	20/25/50	50.0%	0.011	0.017	1
128	28/50/50	67.4%	0.007	0.011	0

Appendix

Table 4: Candidate public-set sizes at selection time. Errors are leave-one-model-out mean absolute errors of full-suite means on the

[0,1]

scale, averaged over setting totals or their components. Reversed pairs count agent configuration pairs whose full-suite scores differ by at least

0.02

but are ordered oppositely, over totals and components.

Score	Mean $\|\Delta\|$	Max $\|\Delta\|$	Mean $\Delta$	Kendall $\tau$	Reversed pairs
$S_{\mathrm{T2S}}$	0.009	0.023	$-0.003$	0.956	0
Indoor	0.034	0.115	$+0.028$	1.000	0
Outdoor	0.009	0.016	$0.000$	0.956	1
$S_{\mathrm{I2S}}$	0.013	0.030	$-0.005$	1.000	0
$S_{\mathrm{model}}$	0.008	0.024	$-0.004$	0.956	0

Appendix

Table 5: Public-set versus full-suite scores under current scoring for the ten agent configurations evaluated on both.

\Delta

is the public-set score minus the full-suite score. Kendall’s

\tau

compares rankings. Reversed pairs have full-suite gaps of at least

0.02

but the opposite public-set order.

Panel of Figure 7 — Figure 7: Text-to-Scene case: Nordic Harbour. Fable places 2,872 actors, arranging townhouses around rectangular canals with quays and bridges. Inkling places 24 actors, largely tilted townhouses on a bare plate without visible canals or quays.

Panel of Figure 9 — Figure 9: Image-to-Scene running case: Indoor Atomic v1 054. The successful run restores the world globe and changes nothing else. The unsuccessful run does not restore the globe and deletes nearly the whole room, keeping 31 of 1,325 actors. Rows are brightened identically.

Panel of Figure 10 — Figure 10: Image-to-Scene running case: Comp 31 Old Industrial Pallet Bay Repair. The successful run removes the duplicate and returns the original stack, with no other change. The unsuccessful run keeps the duplicate, restores neither stack, and makes four unintended edits.

Group	Input	Criterion
Candidate Integrity all cases	Generated scene, execution record, and dependencies	Loadability, minimum content, scene-graph validity, dependency resolution, and asset-library manifest agreement
Physical Safety all eligible cases	Scene geometry and targeted engine measurements	Supported actors and collision-free actors
Semantic Verifier Text-to-Scene	Prompt, frozen requirements, scene state, and rendered evidence	Detailed Alignment and Overview Alignment
Ground-Truth-Based Verifier Image-to-Scene	Input scene, generated scene, and withheld human-assembled scene	Actor repair TP/FP/FN, precision, recall and F1
Downstream EUS reported separately	Navigation mesh, eligible targets, and fixed-walker rollouts	Approachability and route completion

Appendix

Table 6: Evaluation groups and their inputs. Ground truth is withheld from agents.

Quantity	Acceptance test
Asset and class	Exact normalized structural descriptor agreement
World position	$\\|p_{c}-p_{g}\\|_{2}\leq 5$ cm
World rotation	$2\arccos(\min(1,\|q_{c}^{\top}q_{g}\|))\leq 5^{\circ}$
Signed scale	$\max_{j}\|s_{c,j}/s_{g,j}-1\|\leq 0.05$
Bounds centre, if recorded in GT	$\\|b_{c}-b_{g}\\|_{2}\leq 5$ cm
Bounds extent, if recorded in GT	$\max_{j}\|e_{c,j}/e_{g,j}-1\|\leq 0.05$

Appendix

Table 7: Nominal acceptance tests for desired-present repair targets. All applicable tests must pass; positions are in cm. These benchmark tolerances are not human-calibrated perceptual equivalences.

Agent configuration	Track	Unresolved dependencies	Empty scene	No submission	Missing result	Total
Muse Spark 1.3 (medium)	Outdoor	–	–	7	–	7
DeepSeek V4.1 Flash (high)	T2S	4	1	–	–	5
GLM-5.3 Flash (max)	T2S	2	–	–	2	4
Claude Opus 5 (max)	T2S	1	–	–	–	1
GPT-6 Astra (max)	T2S	1	–	–	–	1
Grok 4.6 (high)	T2S	1	–	–	–	1

Appendix

Table 8: Public evaluations assigned zero, by configuration and reason (19 of 1,330 pairs). Configurations with no such cases are omitted.

Figure 11: Text-to-Scene scoring flow on Bazaar. Claude Fable 5.1 (max) and Inkling (high) receive the same prompt. Targeted views in panel 3A resolve the colorful-awnings requirement; panel 3B shows support and penetration checks with their respective eligible actor populations (Appendix C.3 ). The dome marker denotes an engine-confirmed collision. Captures are annotated for clarity.

Figure 12: Image-to-Scene scoring flow on roadside billboard reset. GPT-6 Astra (max) restores the target and preserves unrelated actors. Inkling (high) leaves a $55^{\circ}$ rotation error and enlarges two unrelated billboards; these off-target billboards also fail the physical checks and lie outside the displayed view. The reference image is the agent input; comparison images are evaluator base-color renders. Repair F1 uses structured scene states, while Physical Safety uses engine measurements.

Figure 13: EUS scoring illustration on the same Nordic Harbour prompt. For approachability $A$ , most floor-level objects in the left scene are reachable ( $A=0.88$ ), whereas most floor-level objects in the right scene are not ( $A=0.27$ ). For walk validation, three routes lead from a floor-level anchor to targets at the near, mid, and far route-distance quantiles; all three are completed in the left scene and one in the right. The overall EUS is therefore $0.88$ for the left scene and $0.09$ for the right.

Weights and order	Adopted	Individual fit	PL fit
$\mathbf{w}^{D}$ : identity/environment, content/quantity, attributes/materials, spatial composition	$(.25,.40,.15,.20)$	$(.18,.30,.13,.39)$	$(.08,.20,.17,.55)$
$\mathbf{w}^{O}$ : prompt alignment, layout, style/atmosphere, completeness/polish	$(.40,.25,.20,.15)$	$(0,0,1,0)$	$(.10,0,.90,0)$
$\mathbf{w}^{P}$ : supported fraction, collision-free fraction	$(.50,.50)$	$(0,1)$	$(0,1)$
$(\alpha,\beta,\gamma)$ : Detailed, Overview, Physics	$(.20,.60,.20)$	$(.25,.55,.20)$	$(.30,.50,.20)$

Appendix

Table 9: Aggregation weights for the adopted score and two preference fits. Each vector follows the displayed subscore order and sums to one; both fits fix the final Physics share at

\gamma=0.20

. Fitting and the retrospective holdout protocol are defined in Appendix C.9 .

Agent configuration	T2S Easy $\uparrow$	T2S Medium $\uparrow$	T2S Hard $\uparrow$	I2S Easy $\uparrow$	I2S Medium $\uparrow$	I2S Hard $\uparrow$	$S_{\mathrm{model}}\uparrow$
Gemini 3.8 Flash (high)	0.724	0.618	0.660	0.627	0.610	0.553	0.632
GPT-5.6 Sol (high)	0.733	0.713	0.687	0.451	0.427	0.372	0.564
Muse Spark 1.3 (medium)	0.662	0.605	0.602	0.485	0.330	0.311	0.500
GLM-5.3 Flash (max)	0.630	0.462	0.499	0.427	0.338	0.276	0.438
Qwen 3.8 27B (thinking off)	0.598	0.566	0.512	0.258	0.199	0.149	0.380
Qwen 3.8 27B (thinking on)	0.572	0.540	0.442	0.291	0.222	0.146	0.369

Appendix

Table 10: Difficulty-stratified scores for ten agent configurations on all 190 public and private cases.

Figure 14: Task-category performance of 14 agent configurations on the public set: (a) Image-to-Scene scores by operation category and (b) Text-to-Scene Detailed Alignment.

Task	$\Delta$ score [95% CI]	$\Delta$ calls [95% CI]
Text-to-Scene	6.69 [ $-$ 3.23, 13.46]	$-$ 6.60 [ $-$ 19.55, 6.75]
Indoor	6.89 [ $-$ 4.62, 17.22]	8.44 [5.64, 11.65]
Outdoor	$-$ 13.38 [ $-$ 25.78, $-$ 1.55]	1.88 [0.02, 3.54]

Appendix

Table 11: Paired differences between GPT-6 Astra (max) and Gemini 3.8 Flash (high) on public tasks, computed as Astra minus Gemini, with 95% confidence intervals. Scores use 0–100; calls exclude polling. Interval estimation follows Appendix D.1 .

Figure 15: Construction behavior of Astra, Gemini, Fable and Opus, the four highest-ranked public configurations overall, on the same 20 T2S tasks. (a) Mean non-polling calls per task, split into Python and other tools. (b) Percentage of parseable Python scripts containing each operation family shown; families can co-occur. The outlined column shows pose-editing operations. All four configurations make zero named transform calls. Counts and definitions are in Appendices D.4 and D.1 .

Figure 16: Public repair outcomes by configuration (25 Indoor and 50 Outdoor scheduled cases). All scheduled cases enter the denominators. R denotes Recall and P denotes Precision; R=1 means complete target recovery, and P=1 means at least one recovered target and no false positives; the renamed-material export artifact described below is excluded.

Agent configuration	Indoor Pub.	Indoor Priv.	Outdoor Pub.	Outdoor Priv.
GPT-6 Astra (max)	71.7 / 92.0 / 78.1	—	31.0 / 31.5 / 27.6	—
Gemini 3.8 Flash (high)	66.7 / 76.0 / 69.1	53.4 / 72.7 / 58.1	48.7 / 55.5 / 44.6	51.8 / 56.2 / 50.2
Claude Fable 5.1 (max)	46.0 / 48.0 / 46.7	—	31.4 / 26.1 / 26.4	—
Claude Opus 5 (max)	54.7 / 60.0 / 56.5	—	31.3 / 30.7 / 30.0	—
GPT-5.6 Sol (high)	31.4 / 88.0 / 44.2	30.8 / 78.2 / 41.6	26.0 / 41.8 / 25.8	23.5 / 52.8 / 30.1
Muse Spark 1.3 (medium)	56.7 / 60.0 / 57.9	35.2 / 40.0 / 36.7	7.2 / 10.7 / 8.0	10.7 / 14.4 / 11.6

Appendix

Table 12: Actor Precision / Recall / F1 (0–100) on public and private Image-to-Scene tasks. Values are means over cases, so F1 need not equal the harmonic mean of the displayed Precision and Recall.

Change type	Public, 14 configs. $n=212$	Public, 10 configs. $n=134$	Private, 10 configs. $n=191$
Added actors without a target match	27	23	40
Modified non-target actors	62	52	73
Deleted non-target actors	3	3	3
Non-target actors repurposed for a repair	2	2	0
Unrecovered target state	0	0	0

Appendix

Table 13: Unintended changes despite complete target recovery. Complete recovery means all required repairs succeed (Recall=1). Here, false positives (FP) are added actors without a target match or unintended modifications, deletions or repurposing of actors that should be preserved. Counts refer to submissions (one configuration on one task), not actors. A: submissions containing each change type; categories overlap and

n

is the number of complete recoveries. B: all 14 public configurations; threshold triplets give distance (cm), rotation (degrees) and relative scale/bounds-extent (%) tolerances. The last row excludes non-target changes confined to asset/material properties. C: submissions with FP / complete recoveries (percentage), for ten matched public/private configurations. All counts exclude the renamed-material export artifact (Appendix D.5 ).

Agent configuration	Text-to-Scene		Indoor		Outdoor
	Public	Private	Public	Private	Public	Private
GPT-6 Astra (max)	65.6	—	19.2	—	14.2	—
Gemini 3.8 Flash (high)	72.2	73.0	10.8	10.7	12.4	11.9
Claude Fable 5.1 (max)	85.2	—	12.4	—	14.2	—
Claude Opus 5 (max)	116.7	—	18.9	—	20.5	—
GPT-5.6 Sol (high)	55.4	56.4	23.9	28.2	22.3	23.3

Appendix

Table 14: Mean tool calls per recorded run, excluding asynchronous status polling. Public and private tasks are reported separately; a dash marks configurations without private-task evaluation.

Figure 17: Outdoor navigation changes relative to the matched input. Dots show generated-scene-minus-input means on matched navigation tasks, using the conservative withholding rule. Whiskers show paired standard errors for Gemini, DeepSeek and Gemma; the other five marks show means only. Differences use the reported three-decimal means. Generated-scene means, measured-only means, components and coverage appear in Table 15 .

Agent configuration	EUS $\uparrow$	Measured $\uparrow$	Coverage	$A\uparrow$	$W\uparrow$	Scored	Status
Gemini 3.8 Flash (high)	0.479 $\pm$ 0.050	0.513	93%	0.533	0.905	28	0 / 2
GPT-5.6 Sol (high)	0.477 $\pm$ 0.068	0.511	93%	0.519	0.926	27	1 / 2
GLM-5.3 Flash (max)	0.387 $\pm$ 0.066	0.432	90%	0.478	0.778	24	2 / 3
Qwen 3.8 27B (thinking off)	0.373 $\pm$ 0.061	0.399	93%	0.420	0.810	28	0 / 2
Gemma 4 31B (thinking on)	0.369 $\pm$ 0.069	0.410	90%	0.419	0.792	24	3 / 3
Gemma 4 31B (thinking off)	0.347 $\pm$ 0.064	0.372	93%	0.389	0.736	24	4 / 2

Appendix

Table 15: Embodied Utility Score (EUS); uncertainties are standard errors. Conservative means assign zero to withheld records; measured means omit them. Coverage is the measured share; Scored counts measured records; Status lists zero / withheld counts. Panel A has 29 GLM-5.3 Flash (max) records and 30 per other agent; Panel B has 71 per agent. Panel C summarizes indoor rooms.

A. Validation questions
Question	Unit	Result	Scope
Human agreement	Four holdout prompts, 84 consensus pairs	73/84 agreement; 95% CI 76.19–94.05%; $\tau_{b}=0.738$	Internal validation: earlier exploration used all 16 prompts.
Judge repeatability	Nine configurations, six cases, three repeats	Rank $\tau_{b}=0.889$ – $0.944$ ; model-mean SD 0.0010–0.0132	Fixed evidence and routing; not complete adaptive-verifier variability.
Repair-weight sensitivity	Fourteen configurations, 75 public repairs	For tested $\lambda_{\mathrm{rep}}\geq 0.5$ , rank $\rho\geq 0.952$ ; at most three places shifted	Fixed scenes, thresholds and cases; no human labels fitted.

Appendix

Table 16: Evaluator validation and its scope. Panel A separates three validation questions; Panel B reports agreement fractions for the retrospective 12/4-prompt split. Fitted weights are in Table 9 .

Figure 18: Detailed Alignment repeatability on six selected T2S cases. Marks show each repeat’s six-case mean minus the configuration’s three-repeat mean; colors and shapes distinguish repeats. The right columns report the absolute mean and sample standard deviation across the three repeats. Evidence and routing are fixed.

$\lambda_{\mathrm{rep}}$	Spearman $\rho$	Kendall $\tau_{b}$	Max. rank shift
0.00	0.705	0.516	6
0.25	0.916	0.802	3
0.50	0.952	0.868	3
0.60	0.978	0.912	2
0.70	0.991	0.956	1
0.80	1.000	1.000	0

Appendix

Table 17: I2S ranking sensitivity to the Repair F1 weight for 14 configurations on 75 public repairs. Correlations and maximum rank shifts are relative to

\lambda_{\mathrm{rep}}=0.8

\lambda_{\mathrm{rep}}F_{1}+(1-\lambda_{\mathrm{rep}})S_{\mathrm{physics}}

. Scenes, thresholds and case selection remain fixed.

Explore similar work

Jul 24, 2026cs.AI

SceneActBench: Can Agents Act on the 3D Scenes They See?

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under evaluated. We present SceneActBench, a benchmark for visually conditioned action across five 3D tasks under a unified agent-environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6-50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.

Yifei Zhao, Xiangxin Zhou, Wenhao Yang +11

1Tencent Hunyuan · 2THU · 3NJU +3

3D Scene Understanding Scene Understanding 3D Generation Agent Loop

Ask PDF

Sep 28, 2026cs.CV

LEGO-Anything: Coding Agents for 3D Scene Reconstruction

A 3D scene reconstructed from a single image is most useful when represented not as a rendering or a fixed 3D output, but as an explicit scene program whose execution yields a scene that can be inspected, edited, and queried. We present LEGO-Anything, an Image-to-Code framework in which a coding agent iteratively writes and executes Blender code, inspects scenes and renderings, and revises the program. To evaluate end-to-end scene recovery, we introduce LEGO-Bench, a simulator-grounded benchmark with 208 images from 104 diverse indoor and outdoor scenes. LEGO-Bench separately scores artifact validity, visible-surface geometry, and rendered appearance. Its simulator-grounded design enables extensibility and precise automatic evaluation. Among evaluated agents, GPT-6-astra achieves the strongest overall results, with 53.4% indoor and 39.6% outdoor scores, yet substantial gaps remain between delivering valid scene artifacts and faithfully recovering scene geometry and appearance. Analysis of agent construction trajectories reveals three recurring issues: weak scene initialization, regressive edits during iteration, and unreliable self-evaluation. These findings motivate LEGO-Plugin, a training-free harness plugin for more controlled iterative scene construction, which improves all six evaluated models, with relative gains of up to 62.7% in overall score. Finally, we test whether reconstructed scenes can represent natural images and support vision tasks. In LEGO-World, we derive object detections, instance masks, and relative depth as deterministic queries on scenes reconstructed by GPT-6-astra. These readouts show non-trivial performance across all three tasks but fall well short of specialized vision models, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.

Xirui Li, Peng Shi, Mingwen Dong +7

University of Maryland, College Park · AWS · Work done during an internship at AWS

3D Reconstruction Scene Understanding Image-To-Code Generation Coding Agents

Ask PDF

Apr 30, 2026cs.AI

SpatialGrammar: A Domain-Specific Language for LLM-Based 3D Indoor Scene Generation

Automatically generating interactive 3D indoor scenes from natural language is crucial for virtual reality, gaming, and embodied AI. However, existing LLM-based approaches often suffer from spatial errors and collisions, in part because common scene representations-raw coordinates or verbose code-are difficult for models to reason about 3D spatial relationships and physical constraints. We propose SpatialGrammar, a domain-specific language that represents gravity-aligned indoor layouts as BEV grid placements with deterministic compilation to valid 3D geometry, enabling verifiable constraint checking. Building on this representation, we develop (1) SG-Agent, a closed-loop system that uses compiler feedback to iteratively refine scenes and enforce collision constraints, and (2) SG-Mini, a 104M-parameter model trained entirely on compiler-validated synthetic data. Across 159 test scenes spanning five scenarios of different complexity, SG-Agent improves spatial fidelity and physical plausibility over prior methods, while SG-Mini performs competitively against larger LLM-based baselines on single-shot generation scenarios.

Song Tang, Kaiyong Zhao, Yuliang Li +5

The Hong Kong University of Science and Technology (Guangzhou) · XGRIDS · Harbin Institute of Technology (Shenzhen)

3D Scene Generation 3D Layout Generation 3D Spatial Reasoning 3D Generation Scenario Generation Domain-Specific Language+1

Ask PDF