Given a set of LEGO parts and an image of an assembled object, our task is to infer the pose of each part to assemble the object. In contrast with traditional 3D reconstruction, doing so requires satisfying discrete physical constraints, which provide structure but render the problem combinatorial and highly sensitive to small pose errors. To learn a prior over how parts fit together, we train a set-conditioned SE(3) flow model on procedurally generated assemblies. We then finetune this model to inject image conditioning, using 1% as much data. However, while the learned flow captures global structure, it does not enforce connectivity. To address this, we introduce an analytic connectivity-field guidance to pull predictions toward connected assemblies at test time. Through this, we demonstrate improved build quality with model size, pretraining, and guidance. We make our models and data available for research purposes at https://kulits.github.io/BrickFlow/
Figures & tables
Figure 1: Top: We present BrickFlow, an SE(3) flow-matching model trained to take in a known set of LEGO pieces (pile) with noisy poses (translucent) and denoise them into some valid structure. Bottom: Using a procedural dataset 1% the size, we adapt it for image-conditioned reconstruction.
Figure 2: ProcBrick samples. We provide sample renders from ProcBrick of increasing complexity. The observable differences in shading are due to random HDRI selection done at the time of rendering.
Figure 3: Pipeline. (a) BrickFlow denoises poses given known part types, conditioned on flow time t . (b) Finetuning adds materials and conditioning on frozen DINOv3 features with a Perceiver resampler.
Figure 4: Connectivity field. (a) The energy E of a two-part assembly as a function of where the upper brick is placed on the lower one. It is zero (ink) exactly at valid seats ( ch=0 ): isolated points where the brick is held rigidly, and arcs where it connects to a single stud and can swivel. The upper brick is colored by how far its material must move to reach a valid seat ( Eq. 3 ), with needles showing the direction. (b) E models each assembly as an electrical network: parts are nodes, and each candidate seat h adds a conductance gh between two parts (drawn as wires, brighter for higher gh ). Seated connections join every part of the valid duck (right); the disordered estimate (left) consists of seated groups joined only through weak, unseated candidates, so its effective resistances Ωab , and hence E , are high. (c) A noisy estimate (center) and six nearby valid structures built from the same parts, colored by how far their material moves from the estimate. E=0 on each of them.
Model
Guid.
Valid (%) ↑
Comp. ↓
Max comp. ↑
MM ↓
Coll. ↓
ProcBrick
PT DiT-L
–
10.3 [9.9, 10.8]
7.0
33.5
72.6
18.6
DPS
13.5 [12.9, 14.0]
2.9
43.0
72.5
17.8
IC DiT-S
–
6.5 [6.1, 6.9]
10.5
26.6
53.7
21.1
DPS
8.9 [8.4, 9.4]
6.0
33.9
53.6
18.5
IC DiT-B
–
10.1 [9.6, 10.6]
8.4
29.9
48.4
17.6
Table 1: Evaluation. Procedural (ProcBrick) and human-designed (BrickNet) assemblies, 100 samples per structure. PT DiT-L receives only the inventory. Valid: 95% CI is shown in brackets.
Figure 5: Validity and connectivity across part counts. IC DiT-L on ProcBrick, unguided (solid) and guided (dashed). (a) Validity of one sample and of the best of 10 and 100. (b) Connected components; guidance removes 16% of them for the smallest assemblies and about half beyond 80 parts. (c) Parts in the largest component, which grows with guidance from 51 to 66 for assemblies of 96–100 parts; the dotted line is a fully connected assembly.
Figure 6: BrickNet reconstructions. Outputs are selected from the guided IC DiT-L. As predictions are defined up to a global rigid transform, outputs are aligned to the ground truth as postprocessing.
Figure 7: Real-world reconstructions. Inputs are photographs of physical builds. Outputs are from guided IC DiT-L, and are rendered following a postprocessing global alignment to set the camera.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Pretraining (PT)
Finetuning (IC)
Data
100M 100-part assemblies
1M rendered assemblies
Part counts
prefixes of 2–100
2–100, balanced
Steps, batch
1M, 4,096
200k, 2,048
Muon LR (S/B/L)
2.4/1.2/0.9×10−3
7.1/3.5/2.7×10−4
AdamW LR
4×10−4
1.4×10−4
Flow time t
logit-normal, μ=1.2 , σ=1.5 ( Abramson et al., 2024 )
logit-normal, μ=0 , σ=1.5
Appendix
Table 2: Training settings. EMA widths maximize the flow’s log-likelihood on held-out assemblies.
Model
Guid.
Δ Valid
Δ Coll.
Δ MM
ProcBrick
PT DiT-S
–
1.8 [1.7, 1.9]
− 4.99 [ − 5.07, − 4.93]
0.02 [0.02, 0.02]
DPS
0.8 [0.8, 0.9]
− 5.10 [ − 5.18, − 5.02]
0.01 [0.01, 0.01]
PT DiT-B
–
2.5 [2.3, 2.7]
− 3.63 [ − 3.69, − 3.58]
0.06 [0.06, 0.06]
DPS
1.2 [1.1, 1.3]
− 4.47 [ − 4.54, − 4.40]
0.05 [0.05, 0.05]
PT DiT-L
–
3.0 [2.8, 3.2]
− 2.80 [ − 2.85, − 2.76]
0.07 [0.07, 0.07]
Appendix
Table 3: Effect of snapping. Snapped minus raw, paired by sample, with 95% CIs. Snapping raises validity and removes colliding pairs for every model while moving MM by at most 0.07 LDU, and it adds less to guided samples.
Table 4: Guidance hyperparameters. Validity (%) of guided samples before snapping. Each change moves validity by at most 2.1 points. Top: changes from the initial setting. Bottom: changes from the final setting.
Table 5: Full results. Validity at collision allowances of 1.1 and 1.0 LDU, and MM, with 100 samples per structure. Validity at 1.0 LDU is nearly identical, and guidance changes MM by at most 0.3 LDU.
Seconds per sample
Samples per second
Model
Unguided
DPS
Unguided
DPS
PT DiT-S
0.09
0.30
919
553
PT DiT-B
0.11
0.33
374
286
PT DiT-L
0.25
0.62
125
98
IC DiT-S
0.39
0.68
106
97
IC DiT-B
0.51
0.75
53
47
Appendix
Table 6: Inference time. Seconds to sample one assembly on one H100 GPU.
Variant
Valid
Comp.
MM
ProcBrick, IC DiT-S
IC DiT-S
6.5 [6.1, 6.9]
10.46
53.73
from scratch
1.2 [1.0, 1.4]
32.15
50.43
50% renders
6.3 [5.9, 6.7]
10.62
54.13
25% renders
5.8 [5.4, 6.3]
11.07
55.24
BrickNet, IC DiT-S
Appendix
Table 7: Ablations. Variants of IC DiT-S, without guidance. Training without the pretrained model collapses validity, and less rendered data lowers it moderately. 10 samples per structure (100 for the reference rows).