BrickNet: Graph-Backed Generative Brick Assembly

Peter Kulits; Cordelia Schmid

cs.CVApr 24, 2026

BrickNet: Graph-Backed Generative Brick Assembly

Authors: Peter Kulits, Cordelia Schmid

Organizations: Inria, `Ecole Normale Sup`erieure, CNRS, PSL Research University · Max Planck Institute for Intelligent Systems, Tübingen

Abstract

We train a language model to generate LEGO-brick build sequences. While prior work has been restricted to discrete, voxel-like towers, we consider a much broader set of pieces, encompassing thousands of part types with diverse connection semantics. To enable this, we first collect a large-scale dataset of over 100,000 human-designed LDraw brick objects and scenes. The complexity of our setting makes it challenging to autoregressively assemble structures that satisfy physical constraints. When predicting block pose directly, build sequences quickly become invalid after a small number of steps. Although pieces are placed in 3D space, it is the spatial relationships of the parts which define the whole. With this in mind, we design a graph-based program representation that parametrizes structure through connectivity, improving the physical grounding of generated sequences. To enable future applications, we make our dataset and models available for research purposes. https://kulits.github.io/BrickNet

Explore similar work

BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization
May 25, 2026Zhengyang Ni, Feng Yan, Yu Guo +1Building Blocks Geometric Reconstruction
Brick-Composer: Using MLLMs for Assembly with Diverse Bricks
Jun 3, 2026Jiateng Liu, Bingxuan Li, Zhenhailong Wang +8Building Blocks Mllm-Based
Sample-Efficient Post-Training for LEGO Spatial-Physics Reasoning
May 29, 2026Yuhuan Yuan, Zhouliang Yu, Minghao Liu +2Building Blocks Spatial Reasoning

May 25, 2026 · cs.AIJ/K move · Enter open · S save

BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization

Zhengyang Ni, Feng Yan, Yu Guo, Fei Wang

Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence · Institute of Artificial Intelligence and Robotics

Generating physically buildable brick structures from 3D shapes requires more than geometric reconstruction: the output must also satisfy discrete part constraints and structural stability. Existing brick generation methods either rely on heuristic optimization, which can break down when the target 3D shape does not admit a feasible structure under predefined constraints, or generate brick sequences without explicitly modeling the underlying 3D geometry and assembly relations. In this work, we present BrickAnything, a geometry-conditioned autoregressive framework for generating buildable brick structures from diverse 3D representations. BrickAnything uses point clouds as a unified geometric interface and predicts brick sequences that reconstruct the target shape under assembly constraints. To model structural dependencies among bricks, we introduce a structure-aware tree tokenization, which represents brick structures through local attachment relations. This formulation makes sequence generation more consistent with the physical construction process, and reduces invalid intermediate states. We further introduce preference-based alignment post-training, validity-constrained decoding and adaptive rollback to improve buildability objectives such as stability and geometric fidelity. Extensive experiments demonstrate that BrickAnything produces geometrically faithful and physically realizable brick structures, and that the proposed tokenization effectively reduces rollback and regeneration compared with conventional ordering strategies.

Building Blocks Geometric Reconstruction Craft 3D Representation Tokenization Autoregressive Model

Open paper Ask about it PDF

Abstract

Explore similar work

BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization

Brick-Composer: Using MLLMs for Assembly with Diverse Bricks

Sample-Efficient Post-Training for LEGO Spatial-Physics Reasoning