BrickWright: Agentic Creation of Physically Feasible Brick Builds

Ava Pun; Kevin Dai; Sally Chen; Maxwell Jones; Ruixuan Liu; Kangle Deng; Changliu Liu; Maneesh Agrawala

cs.CVOct 22, 2026

BrickWright: Agentic Creation of Physically Feasible Brick Builds

Authors: Ava Pun, Kevin Dai, Sally Chen, Maxwell Jones, Ruixuan Liu, Kangle Deng, Changliu Liu, Maneesh Agrawala,

Organizations: Carnegie Mellon University · Independent Researcher · Stanford University

Abstract

Frontier coding agents are becoming increasingly adept at 3D modeling. Yet, they still face challenges when building objects meant to be assembled in the physical world, often creating invalid assemblies with floating, colliding, or disconnected parts. We argue that providing agents with tools for assembling builds and verifying their physical feasibility is sufficient for them to produce physically valid assemblies that still adhere to input prompts. Focusing on the domain of LEGO brick builds, we present BrickWright, a framework for generating physically feasible brick builds from text or image prompts using pre-trained agents, where a build is physically feasible if it uses only real parts, has no colliding or floating parts, and forms a connected structure if possible. BrickWright guides agents to create builds by writing code with BrickLayer, a Python toolkit with two main components: an ergonomic brick assembly API that halves inference cost compared to working in raw LDraw, and a validator that provides feedback on physical feasibility. Whereas baseline agents achieve physical validity rates ranging from only 9% (GPT-6 Luna) to 69% (GPT-6 Astra), BrickWright-guided agents achieve 100% validity on a sample of 512 prompts, while improving prompt adherence for lightweight agents and maintaining it for stronger agents. Additionally, they outperform specialized models on the text-to-brick task. Code is available on our project page: https://avalovelace1.github.io/brickwright/

Figures & tables

Figure 2: Physical violations in builds by Codex (GPT-6 Astra, high reasoning). Red outlines in insets show mesh intersections.

Figure 3: Method overview. (a) BrickLayer supports catalog discovery, compositional construction with typed connectors, and physical validation. (b) BrickWright prompts a pre-trained LLM agent to write and revise build.py , which contains brick build creation code and agent-defined tests. The validate tool runs fixed collision and floating checks alongside the agent’s tests, while the render tool provides views for visual inspection. The agent calls finalize once all checks pass. The agent may optionally run explore at any time to investigate the BrickLayer API and written brick build. Arrows summarize information flow rather than a fixed tool-call sequence.

Listing 1: BrickLayer usage. Users can add elements, join them via selected connectors, and assess feasibility. The BrickLayer script is easier to understand than its corresponding LDraw output.

Method		Valid $\uparrow$	No col. $\uparrow$	No fl. $\uparrow$	Comp. $\downarrow$	VQA $\uparrow$	PE $\uparrow$	SL2 $\uparrow$	Parts	Unique	Checks $\downarrow$	Time $\downarrow$	Cost $\downarrow$
BrickGPT [ 37 ]		0.990	1.000	0.990	2.67	0.0658	0.1547	0.0514	144.8	7.1	—	247s	—
BrickNet [ 19 ]		0.318	0.320	0.943	1.03	0.6077	0.2846	0.6407	31.8	11.5	—	5s	—
Haiku 5.5	Claude Code	0.328	0.428	0.387	3.59	0.6032	0.2963	0.7286	12.8	5.3	—	168s	$0.04
	Ours w/o assembly API	0.822	0.889	0.828	2.07	0.5744	0.2940	0.7009	16.6	4.8	9.29	279s	$0.13
	Ours w/o validator	0.586	0.744	0.775	2.99	0.6390	0.3012	0.7481	18.3	7.4	—	119s	$0.03
	Ours (full)	1.000	1.000	1.000	1.00	0.6523	0.3056	0.7660	21.5	7.6	2.89	143s	$0.05

Table 1: Quantitative results on BrickNet-512. Valid, No col., and No fl. are valid, collision-free, and floater-free build rates. Comp., Parts, and Unique are mean component, part, and distinct part-ID counts. PE: Perception Encoder. SL2: SigLIP 2. Checks: mean validate calls. Bold/underline mark best/second-best results within each agent group. Costs use October 2026 token pricing, including cache savings.

Figure 4: Qualitative ablation results. Red outlines in insets indicate mesh intersections. See Appendix C for input text prompts.

Figure 5: Extensions. BrickWright naturally supports image-conditioned generation, scenes, articulated builds, editing, and constrained generation. All examples are physically valid and connected. See Appendix C for full input prompts.

Figure 6: Qualitative baseline comparison. BrickWright improves physical feasibility across all agents and visibly boosts output fidelity for lightweight agents (Haiku, Luna), while outperforming specialized brick generation models [ 37 , 19 ] . See Appendix C for text prompts and Fig. 7 for more results. Red outlines in insets indicate mesh intersections.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Haiku 5.5	6 Luna	6.1 Sol	6 Astra
validate calls	1,481	2,364	1,269	1,230
Failed calls	805	1,598	477	393
Syntax error	0.000	0.003	0.000	0.003
Runtime error	0.143	0.281	0.130	0.109
Physical test	0.820	0.635	0.813	0.847
Collision	0.505	0.796	0.899	0.895

Appendix

Table 2: Validation failure types for BrickWright agents on BrickNet-512. Failure types are given as fractions of failed calls. Runtime errors are exceptions while running create_build() or run_tests() ; custom test failures pass all physical checks but fail an agent-written test. No calls failed for other reasons. Indented rows break down physical failures as fractions of physical failures; a call can fail several physical checks, so they may sum to

>1

		valid@ $k$ $\uparrow$					Tokens (K) $\downarrow$
Method		$k{=}1$	$k{=}2$	$k{=}3$	$k{=}4$	$k{=}5$	Total	Input	Output
Haiku 5.5	Claude Code	—	—	—	—	—	603.2	570.0	33.2
	Ours w/o assembly API	0.143	0.324	0.439	0.527	0.613	1,900.7	1,852.2	48.5
	Ours w/o validator	—	—	—	—	—	562.2	540.8	21.3
	Ours (full)	0.320	0.637	0.801	0.908	0.959	846.4	821.7	24.7
6 Luna	Codex	—	—	—	—	—	495.5	482.6	12.9

Appendix

Table 3: Validation success and token usage on BrickNet-512. valid@

k

is the fraction of builds whose first

k

validate calls include at least one passing call (all physical checks and custom tests pass); it applies only to methods with access to validate . Tokens are mean thousands of tokens per prompt; total tokens are the sum of input tokens (including cached input) and output tokens (including reasoning). Bold marks the best valid@

k

within each agent group.

3D Asset Generation

3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.

See all 167 papers

BrickNet: Graph-Backed Generative Brick Assembly
Apr 24, 2026Peter Kulits, Cordelia SchmidConstrained Generative Modeling 3D Asset Generation
Inria, `Ecole Normale Sup`erieure, CNRS, PSL Research University · Max Planck Institute for Intelligent Systems, Tübingen
BrickBench: Evaluating Agentic Brick Design
Oct 8, 2026Peter Kulits, Yiqing Xu, R. Kenny Jones +2AI Agent Benchmarks AI Coding Agents
Stanford University · Max Planck Institute for Intelligent Systems · Inria
Brick-Composer: Using MLLMs for Assembly with Diverse Bricks
Jun 3, 2026Jiateng Liu, Bingxuan Li, Zhenhailong Wang +8Visual Spatial Reasoning Benchmark Construction
UIUC1 · Stevens Institute of Technology2 · Northwestern University3

Apr 24, 2026J/K move · Enter open · S save

BrickNet: Graph-Backed Generative Brick Assembly

Peter Kulits, Cordelia Schmid

Inria, `Ecole Normale Sup`erieure, CNRS, PSL Research University · Max Planck Institute for Intelligent Systems, Tübingen

Constrained Generative Modeling 3D Asset Generation Generative Modeling

Open paper AskPDF arXiv

We train a language model to generate LEGO-brick build sequences. While prior work has been restricted to discrete, voxel-like towers, we consider a much broader set of pieces, encompassing thousands of part types with diverse connection semantics. To enable this, we first collect a large-scale dataset of over 100,000 human-designed LDraw brick objects and scenes. The complexity of our setting makes it challenging to autoregressively assemble structures that satisfy physical constraints. When predicting block pose directly, build sequences quickly become invalid after a small number of steps. Although pieces are placed in 3D space, it is the spatial relationships of the parts which define the whole. With this in mind, we design a graph-based program representation that parametrizes structure through connectivity, improving the physical grounding of generated sequences. To enable future applications, we make our dataset and models available for research purposes. https://kulits.github.io/BrickNet

Abstract

3D Asset Generation

Explore similar work

BrickNet: Graph-Backed Generative Brick Assembly

BrickBench: Evaluating Agentic Brick Design

Brick-Composer: Using MLLMs for Assembly with Diverse Bricks