Given a set of LEGO parts and an image of an assembled object, our task is to infer the pose of each part to assemble the object. In contrast with traditional 3D reconstruction, doing so requires satisfying discrete physical constraints, which provide structure but render the problem combinatorial and highly sensitive to small pose errors. To learn a prior over how parts fit together, we train a set-conditioned SE(3) flow model on procedurally generated assemblies. We then finetune this model to inject image conditioning, using 1% as much data. However, while the learned flow captures global structure, it does not enforce connectivity. To address this, we introduce an analytic connectivity-field guidance to pull predictions toward connected assemblies at test time. Through this, we demonstrate improved build quality with model size, pretraining, and guidance. We make our models and data available for research purposes at https://kulits.github.io/BrickFlow/
Figures & tables
Figure 1: Top: We present BrickFlow, an SE(3) flow-matching model trained to take in a known set of LEGO pieces (pile) with noisy poses (translucent) and denoise them into some valid structure. Bottom: Using a procedural dataset 1% the size, we adapt it for image-conditioned reconstruction.
Figure 2: ProcBrick samples. We provide sample renders from ProcBrick of increasing complexity. The observable differences in shading are due to random HDRI selection done at the time of rendering.
Figure 3: Pipeline. (a) BrickFlow denoises poses given known part types, conditioned on flow time t . (b) Finetuning adds materials and conditioning on frozen DINOv3 features with a Perceiver resampler.
Figure 4: Connectivity field. (a) The energy E of a two-part assembly as a function of where the upper brick is placed on the lower one. It is zero (ink) exactly at valid seats ( ch=0 ): isolated points where the brick is held rigidly, and arcs where it connects to a single stud and can swivel. The upper brick is colored by how far its material must move to reach a valid seat ( Eq. 3 ), with needles showing the direction. (b) E models each assembly as an electrical network: parts are nodes, and each candidate seat h adds a conductance gh between two parts (drawn as wires, brighter for higher gh ). Seated connections join every part of the valid duck (right); the disordered estimate (left) consists of seated groups joined only through weak, unseated candidates, so its effective resistances Ωab , and hence E , are high. (c) A noisy estimate (center) and six nearby valid structures built from the same parts, colored by how far their material moves from the estimate. E=0 on each of them.
Model
Guid.
Valid (%) ↑
Comp. ↓
Max comp. ↑
MM ↓
Coll. ↓
ProcBrick
PT DiT-L
–
10.3 [9.9, 10.8]
7.0
33.5
72.6
18.6
DPS
13.5 [12.9, 14.0]
2.9
43.0
72.5
17.8
IC DiT-S
–
6.5 [6.1, 6.9]
10.5
26.6
53.7
21.1
DPS
8.9 [8.4, 9.4]
6.0
33.9
53.6
18.5
IC DiT-B
–
10.1 [9.6, 10.6]
8.4
29.9
48.4
17.6
Table 1: Evaluation. Procedural (ProcBrick) and human-designed (BrickNet) assemblies, 100 samples per structure. PT DiT-L receives only the inventory. Valid: 95% CI is shown in brackets.
Figure 5: Validity and connectivity across part counts. IC DiT-L on ProcBrick, unguided (solid) and guided (dashed). (a) Validity of one sample and of the best of 10 and 100. (b) Connected components; guidance removes 16% of them for the smallest assemblies and about half beyond 80 parts. (c) Parts in the largest component, which grows with guidance from 51 to 66 for assemblies of 96–100 parts; the dotted line is a fully connected assembly.
Figure 6: BrickNet reconstructions. Outputs are selected from the guided IC DiT-L. As predictions are defined up to a global rigid transform, outputs are aligned to the ground truth as postprocessing.
Figure 7: Real-world reconstructions. Inputs are photographs of physical builds. Outputs are from guided IC DiT-L, and are rendered following a postprocessing global alignment to set the camera.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Pretraining (PT)
Finetuning (IC)
Data
100M 100-part assemblies
1M rendered assemblies
Part counts
prefixes of 2–100
2–100, balanced
Steps, batch
1M, 4,096
200k, 2,048
Muon LR (S/B/L)
2.4/1.2/0.9×10−3
7.1/3.5/2.7×10−4
AdamW LR
4×10−4
1.4×10−4
Flow time t
logit-normal, μ=1.2 , σ=1.5 ( Abramson et al., 2024 )
logit-normal, μ=0 , σ=1.5
Appendix
Table 2: Training settings. EMA widths maximize the flow’s log-likelihood on held-out assemblies.
Model
Guid.
Δ Valid
Δ Coll.
Δ MM
ProcBrick
PT DiT-S
–
1.8 [1.7, 1.9]
− 4.99 [ − 5.07, − 4.93]
0.02 [0.02, 0.02]
DPS
0.8 [0.8, 0.9]
− 5.10 [ − 5.18, − 5.02]
0.01 [0.01, 0.01]
PT DiT-B
–
2.5 [2.3, 2.7]
− 3.63 [ − 3.69, − 3.58]
0.06 [0.06, 0.06]
DPS
1.2 [1.1, 1.3]
− 4.47 [ − 4.54, − 4.40]
0.05 [0.05, 0.05]
PT DiT-L
–
3.0 [2.8, 3.2]
− 2.80 [ − 2.85, − 2.76]
0.07 [0.07, 0.07]
Appendix
Table 3: Effect of snapping. Snapped minus raw, paired by sample, with 95% CIs. Snapping raises validity and removes colliding pairs for every model while moving MM by at most 0.07 LDU, and it adds less to guided samples.
Table 4: Guidance hyperparameters. Validity (%) of guided samples before snapping. Each change moves validity by at most 2.1 points. Top: changes from the initial setting. Bottom: changes from the final setting.
Table 5: Full results. Validity at collision allowances of 1.1 and 1.0 LDU, and MM, with 100 samples per structure. Validity at 1.0 LDU is nearly identical, and guidance changes MM by at most 0.3 LDU.
Seconds per sample
Samples per second
Model
Unguided
DPS
Unguided
DPS
PT DiT-S
0.09
0.30
919
553
PT DiT-B
0.11
0.33
374
286
PT DiT-L
0.25
0.62
125
98
IC DiT-S
0.39
0.68
106
97
IC DiT-B
0.51
0.75
53
47
Appendix
Table 6: Inference time. Seconds to sample one assembly on one H100 GPU.
Variant
Valid
Comp.
MM
ProcBrick, IC DiT-S
IC DiT-S
6.5 [6.1, 6.9]
10.46
53.73
from scratch
1.2 [1.0, 1.4]
32.15
50.43
50% renders
6.3 [5.9, 6.7]
10.62
54.13
25% renders
5.8 [5.4, 6.3]
11.07
55.24
BrickNet, IC DiT-S
Appendix
Table 7: Ablations. Variants of IC DiT-S, without guidance. Training without the pretrained model collapses validity, and less rendered data lowers it moderately. 10 samples per structure (100 for the reference rows).
We train a language model to generate LEGO-brick build sequences. While prior work has been restricted to discrete, voxel-like towers, we consider a much broader set of pieces, encompassing thousands of part types with diverse connection semantics. To enable this, we first collect a large-scale dataset of over 100,000 human-designed LDraw brick objects and scenes. The complexity of our setting makes it challenging to autoregressively assemble structures that satisfy physical constraints. When predicting block pose directly, build sequences quickly become invalid after a small number of steps. Although pieces are placed in 3D space, it is the spatial relationships of the parts which define the whole. With this in mind, we design a graph-based program representation that parametrizes structure through connectivity, improving the physical grounding of generated sequences. To enable future applications, we make our dataset and models available for research purposes. https://kulits.github.io/BrickNet
Peter Kulits, Cordelia Schmid
Inria, `Ecole Normale Sup`erieure, CNRS, PSL Research University · Max Planck Institute for Intelligent Systems, Tübingen
Frontier coding agents are becoming increasingly adept at 3D modeling. Yet, they still face challenges when building objects meant to be assembled in the physical world, often creating invalid assemblies with floating, colliding, or disconnected parts. We argue that providing agents with tools for assembling builds and verifying their physical feasibility is sufficient for them to produce physically valid assemblies that still adhere to input prompts. Focusing on the domain of LEGO brick builds, we present BrickWright, a framework for generating physically feasible brick builds from text or image prompts using pre-trained agents, where a build is physically feasible if it uses only real parts, has no colliding or floating parts, and forms a connected structure if possible. BrickWright guides agents to create builds by writing code with BrickLayer, a Python toolkit with two main components: an ergonomic brick assembly API that halves inference cost compared to working in raw LDraw, and a validator that provides feedback on physical feasibility. Whereas baseline agents achieve physical validity rates ranging from only 9% (GPT-6 Luna) to 69% (GPT-6 Astra), BrickWright-guided agents achieve 100% validity on a sample of 512 prompts, while improving prompt adherence for lightweight agents and maintaining it for stronger agents. Additionally, they outperform specialized models on the text-to-brick task. Code is available on our project page: https://avalovelace1.github.io/brickwright/
Ava Pun, Kevin Dai, Sally Chen +7
Carnegie Mellon University · Independent Researcher · Stanford University
Generating a LEGO model from a user request is more than a plausible 3D shape generation. LEGO models require physical support per brick, and each brick should be placed unblocked. Current methods address this issue by training an expert large language model. We reduce the problem of generating extremely long context on all brick connections to that of writing a program that builds LEGO models like a human, considering each intermediate step could be executed. We introduce DailyBrick , a multi-stage agentic system that produces a LEGO model together with a brick-by-brick assembly instruction for human building and robot assembly. We design a multi-stage harness with roles to help LLMs generate more effectively. A versioned construction SDK compiles multi-level decisions, from semantic subparts to individual bricks, into canonical operations. An external Meta-Reviewer agent replays and inspects the generated package, verifies the model and its construction process, identifies failures, traces them to the responsible stage, and routes them to bounded repairs or to editable candidates for later skill or tool updates. Extensive experiments demonstrate that DailyBrick generates LEGO models based on simple user requests better than recent agentic systems, and show its utility in downstream applications such as robot manipulations on model assembly from instructions and brick sets (https://SpatialAgent.github.io/DailyBrick/).