cs.ROSep 30, 2026

ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning

Authors: Gadiel Sznaier Camps, Chengyang He, Guillaume Sartoretti, Eduardo Montijano, Mac Schwager

Organizations: Multi-robot Systems Lab (MSL), Stanford University, Stanford, CA, USA · Multi-Agent Robotic Motion Lab (MARMot), National University of Singapore, Singapore · Perception-Oriented Control Team (POC), Universidad de Zaragoza, Zaragoza, Spain

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomously constructing physically realizable 3D structures remains a significant challenge due to combinatorial action spaces, interchangeable components, equifinal assembly sequences, and strict stability requirements during construction. State-of-the-art methods fine-tune large language models for text-based generative construction. However, these approaches do not allow for Multimodal (text, image, sketch) conditioning, overlook the practical role of scaffolding for stabilizing intermediate structures, and suffer from slow inference speeds. Therefore, we formulate construction as a probabilistic next-block generation task with multiple potential assembly actions and multiple potential task conditioning modalities. Concurrently, we explicitly consider the utility of scaffolding by introducing an auxiliary scaffold block token. We present Scaffold Multimodal Monte Carlo (ScaffoldM3C), a multimodal, lightweight, auto-regressive model for stable block-based construction, that proposes a set of next-step candidate blocks. Leveraging these candidates, we utilize Sequential Monte Carlo (SMC) to maintain a population of possible assembly sequences, allowing us to consider multiple, potentially different, assembly directions simultaneously. We train our multimodal architecture by extending the StableText2Brick dataset to contain image conditioning prompts and scaffold-stabilized build sequences. ScaffoldM3C is 4x smaller than competing baselines, yielding a 5x to 20x speedup during inference, while achieving comparable construction quality to state-of-the-art methods and higher overall stability. We demonstrate the effectiveness of our approach through simulations and real-world robot assembly demonstrations.

Figures & tables

Fig. 1: We present ScaffoldM3C, a lightweight, open source 225M parameter Vision-Language-Construction (VLC) model, trained on 553K different conditioned assembly sequences. Given a starting build location, ScaffoldM3C auto-regressively generates a condition aligned step-by-step assembly plan. At each step, ScaffoldM3C proposes a set of next step candidate blocks from which to choose, while multiple possible assembly sequences are considered simultaneously through a sequential Monte Carlo algorithm, populated based on the candidate block likelihoods. We demonstrate the success of our architecture through simulated and robotic real world construction.
Fig. 2: Stability constraints for candidate block placements. Each panel shows a target block being evaluated before placement: the centroid cc is projected vertically onto the contact plane at pp , and the support region H\mathcal{H} (blue) is the 2D convex hull of the contact faces (hatched) provided by the blocks underneath. (a) A block resting on the ground is stable by direct ground contact. (b) With a single support, pp falls outside H\mathcal{H} and the placement is considered unstable. (c) With supports at both ends, H\mathcal{H} spans the gap between their contact faces, so p∈Hp\in\mathcal{H} and the placement is stable. (d) An auxiliary scaffold block (gray) restores the missing support, enlarging H\mathcal{H} so the otherwise-unstable placement becomes feasible.
Fig. 3: A. Model input: a partial build sequence of blocks and a conditioning prompt (e.g., text, images or some combination of both). B. Our proposed architecture. We utilize several interwoven layers stacked in series as our transformer backbone. The output of the final interwoven layer is fed to a windowed attention layer that ties relevant local candidate-level features to each other before being passed to three cascading model heads that predict the candidate block positions, classes and likelihoods. C. We select the next block through probabilistic Sampling using either Top- KK or SMC.
Fig. 4: Data generation pipeline extending the StableText2Brick dataset. Build sequences are updated with scaffold blocks to satisfy our stability constraints. Images are generated either by simulating the construction and rendering the final structure specified by a given build plan or by providing a text-based guidance prompt to a Qwen text-to-image model.
  Per StructureAcross Structures
 Methodecollision↓e_{\text{collision}}\downarrowsinter↑s_{\text{inter}}\uparrowsfinal↑s_{\text{final}}\uparrowasim↑a_{\text{sim}}\uparrowagt↑a_{\text{gt}}\uparrowcmin↑c_{\text{min}}\uparrowcsem↑c_{\text{sem}}\uparrowblocks ≈\approxscaffolds ≈\approxsoverall↑s_{\text{overall}}\uparrowerate↑e_{\text{rate}}\uparrowefeasible↑e_{\text{feasible}}\uparrowInference Time (s) ↓\downarrow
TextGround truth$$0.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}N/AN/A$$111.0271\pm{\scriptstyle70.0497}$$28.0313\pm{\scriptstyle22.3448}$$100 %\ \%$$100 %\ \%$$100 %\ \%N/A
BrickGPT w/o rej.$$7.0566\pm{\scriptstyle10.1357}$$0.7638\pm{\scriptstyle0.1107}$$0.7644\pm{\scriptstyle0.1101}$$0.2692\pm{\scriptstyle0.0288}$$0.8234\pm{\scriptstyle0.0775}$$0.9006\pm{\scriptstyle0.9280}$$4.5665\pm{\scriptstyle5.5986}$$106.1500\pm{\scriptstyle72.1203}$$0.0000\pm{\scriptstyle0.0000}$$2.7254 %\ \%$$28.0922 %\ \%$$2.3061 %\ \%$$11.4646\pm{\scriptstyle9.7937}
BrickGPT w/ rej.$$0.0000\pm{\scriptstyle0.0000}$$0.7567\pm{\scriptstyle0.1118}$$0.7568\pm{\scriptstyle0.1117}$$0.2689\pm{\scriptstyle0.0285}$$0.8203\pm{\scriptstyle0.0813}$$0.9175\pm{\scriptstyle1.1091}$$4.6517\pm{\scriptstyle5.6934}$$110.9102\pm{\scriptstyle83.2579}$$0.0000\pm{\scriptstyle0.0000}$$3.1315 %\ \%$$100.0000 %\ \%$$3.1315 %\ \%$$13.8635\pm{\scriptstyle16.8635}
BrickGPT-Scaffold$$0.4280\pm{\scriptstyle2.4044}$$0.9183\pm{\scriptstyle0.0557}$$0.9188\pm{\scriptstyle0.0552}$$0.2671\pm{\scriptstyle0.0285}$$0.8326\pm{\scriptstyle0.0728}$$0.8231\pm{\scriptstyle0.6533}$$4.2404\pm{\scriptstyle5.2198}$$195.1023\pm{\scriptstyle158.9575}$$42.4029\pm{\scriptstyle43.6773}$$11.4823 %\ \%$$90.1879 %\ \%$$11.2735 %\ \%$$64.5053\pm{\scriptstyle215.9353}
ours (Top- KK )$$0.0000\pm{\scriptstyle0.0000}$$0.8149\pm{\scriptstyle0.1619}$$0.8192\pm{\scriptstyle0.1580}$$0.2695\pm{\scriptstyle0.0269}$$0.8145\pm{\scriptstyle0.0885}$$1.1479\pm{\scriptstyle1.1855}$$5.5036\pm{\scriptstyle6.2879}$$89.5908\pm{\scriptstyle55.2851}$$21.3132\pm{\scriptstyle17.8939}$$20.0418 %\ \%$$100.0000 %\ \%$$20.0418 %\ \%$$0.5827\pm{\scriptstyle0.3574}
TABLE I: Average quantitative results without rollbacks across 479479 text and 288288 image test prompts. Arrows indicate direction of improvement, while ≈\approx specifies the desired target. Performance is highlighted as best , second best , and worst .
Fig. 5: Qualitative results of our method using both text and image prompts taken from the test dataset. We compare against our baseline using text prompts. Our baseline cannot interpret image prompts, therefore we leave “Not Applicable” for its results.
Fig. 6: Hardware experiments using a 6-DOF XFactory robotic arm to build three different structures. During assembly, nine different blocks were used. Overhanging blocks that would normally be unstable are supported by scaffolding.
Fig. 7: Effect of Sequential Monte Carlo (SMC) population size on generated structures. Each row shows results for a different target object as the number of candidate build sequences increases from 1 to 320, with the ground-truth (GT) structure shown on the right. Larger populations generally yield more complete and coherent structures that better match the target geometry, with diminishing improvements beyond 160 sequences.
Populationsoverall↑s_{\text{overall}}\uparrowsfinal↑s_{\text{final}}\uparrowInference Time (s) ↓\downarrow
1$$20.8333 %\ \%$$0.7977\pm{\scriptstyle0.1890}$$0.4216\pm{\scriptstyle0.2961}
4$$20.8333 %\ \%$$0.8120\pm{\scriptstyle0.1645}$$0.5560\pm{\scriptstyle0.3301}
10$$20.8333 %\ \%$$0.8531\pm{\scriptstyle0.1351}$$0.8866\pm{\scriptstyle0.5078}
20$$25.0000 %\ \%$$0.8773\pm{\scriptstyle0.1215}$$1.5963\pm{\scriptstyle1.2019}
40$$20.8333 %\ \%$$0.8785\pm{\scriptstyle0.1051}$$2.6385\pm{\scriptstyle2.2327}
80$$33.3333 %\ \%$$0.9104\pm{\scriptstyle0.0924}$$4.9324\pm{\scriptstyle4.7820}
TABLE II: Ablation Study on Sequential Monte Carlo Population Size using 24 different test samples.
Fig. 8: Heatmaps of intermediate stability, final stability, standard block count, and scaffold block count under varying candidate and class distribution temperatures (evaluated across 24 test samples). Structural stability is particularly sensitive to the class distribution temperature, with higher temperatures potentially favoring implausible block-type selections and disrupting the assembly sequence. In contrast, increasing the candidate distribution temperature generally improves stability and increases both standard and scaffold block counts, suggesting that broader exploration of lower-probability placement locations allows SMC to identify more successful build sequences.
Fig. 9: Belief state of Sequential Monte Carlo after 12 steps when asked to generate a table. A population size of 4 was used. Each belief state considers a separate and independent set of candidate blocks, conditioned on previously placed blocks of the associated assembly sequence. Multiple particles allow for alternative block placement decisions to be considered simultaneously, allowing the model to potentially escape local minima.
Fig. 10: Possible failure modes of sequence generation using Sequential Monte Carlo sampling. (1) Positional errors lead to assembly plan degradation or out-of-distribution (OOD) states. (2) The model misses prompt details, generating a similar but irrelevant structure. (3) The model prematurely outputs an EOS token. (4) Errors in predicted positions result in block shifting.

Explore similar work

CardsList
Jun 3, 2026cs.AI

Brick-Composer: Using MLLMs for Assembly with Diverse Bricks

We dream of AI agents that can read arbitrary designs and construct real-world objects from reusable building blocks. As a first step toward this vision, we study whether multimodal large language models (MLLMs) possess the visual grounding and spatial reasoning capabilities required for brick assembly. We formulate brick assembly as a sequential decision-making problem, where each step involves two subtasks: brick selection, identifying the target brick from candidate components, and brick pose estimation, predicting where and how the selected brick should be placed. To support this study, we introduce BC-Bench (Brick Construction Benchmark), the first benchmark for evaluating MLLMs on assembly with diverse bricks. Experiments show that current state-of-the-art MLLMs remain far from reliable builders, struggling with fine-grained brick selection and failing at precise pose estimation. To bridge this gap, we propose Brick-Composer, a learning framework that equips MLLMs with assembly skills through three complementary signals: Human Design Sparks, which provide affordance-rich construction demonstrations; World Feedback, which grounds predicted actions in visual and physical consequences; and Synthetic Experience, which scales learning beyond existing object designs. Brick-Composer improves brick selection accuracy by over three times, substantially reduces pose estimation errors, and raises strict step-level assembly success from less than 1% to around 15%. After training, a Qwen-3-8B can correctly compose up to 42% of the steps for a complete object, suggesting that MLLMs can acquire assembly capabilities through targeted, physically grounded learning.

Jiateng Liu, Bingxuan Li, Zhenhailong Wang +8

UIUC1 · Stevens Institute of Technology2 · Northwestern University3

AssemblyMultimodal Large Language Models
AskPDF
May 7, 2026cs.LG

Rollback-Free Stable Brick Structures Generation

While autoregressive models have advanced 3D generation, creating physically stable brick structures remains a challenge due to the strict requirements of gravity and interconnectivity. Existing approaches rely on external physical simulators during inference to perform rejection sampling and brick-by-brick rollbacks, which severely bottlenecks efficiency. To address this, we propose a reinforcement learning paradigm that shifts physical validity enforcement from test-time correction to training-time policy optimization. By utilizing assembly-level rewards, the model optimizes for collision avoidance, global connectivity, structural interlocking, and shape conformity. This paradigm allows the model to internalize physical priors, enabling the first rollback-free generation of stable brick structures. Experimental results demonstrate that our approach achieves state-of-the-art generation quality while accelerating inference speed by orders of magnitude. Our code and dataset are available at https://github.com/miniHuiHui/STABLE. Our models are available at https://huggingface.co/miniHui/STABLE.

Chenhui Xu, Ziyue Bai, Fuxun Yu +2

University of Buffalo · University of Maryland, College Park · Microsoft +1

RollbackOffline Reinforcement Learning
AskPDF
May 25, 2026cs.AI

BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization

Generating physically buildable brick structures from 3D shapes requires more than geometric reconstruction: the output must also satisfy discrete part constraints and structural stability. Existing brick generation methods either rely on heuristic optimization, which can break down when the target 3D shape does not admit a feasible structure under predefined constraints, or generate brick sequences without explicitly modeling the underlying 3D geometry and assembly relations. In this work, we present BrickAnything, a geometry-conditioned autoregressive framework for generating buildable brick structures from diverse 3D representations. BrickAnything uses point clouds as a unified geometric interface and predicts brick sequences that reconstruct the target shape under assembly constraints. To model structural dependencies among bricks, we introduce a structure-aware tree tokenization, which represents brick structures through local attachment relations. This formulation makes sequence generation more consistent with the physical construction process, and reduces invalid intermediate states. We further introduce preference-based alignment post-training, validity-constrained decoding and adaptive rollback to improve buildability objectives such as stability and geometric fidelity. Extensive experiments demonstrate that BrickAnything produces geometrically faithful and physically realizable brick structures, and that the proposed tokenization effectively reduces rollback and regeneration compared with conventional ordering strategies.

Zhengyang Ni, Feng Yan, Yu Guo +1

Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence · Institute of Artificial Intelligence and Robotics

Geometric Reconstruction3D RepresentationCraftAutoregressive Model
AskPDF