cs.ROSep 30, 2026

ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning

Authors: Gadiel Sznaier Camps, Chengyang He, Guillaume Sartoretti, Eduardo Montijano, Mac Schwager

Organizations: Multi-robot Systems Lab (MSL), Stanford University, Stanford, CA, USA · Multi-Agent Robotic Motion Lab (MARMot), National University of Singapore, Singapore · Perception-Oriented Control Team (POC), Universidad de Zaragoza, Zaragoza, Spain

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomously constructing physically realizable 3D structures remains a significant challenge due to combinatorial action spaces, interchangeable components, equifinal assembly sequences, and strict stability requirements during construction. State-of-the-art methods fine-tune large language models for text-based generative construction. However, these approaches do not allow for Multimodal (text, image, sketch) conditioning, overlook the practical role of scaffolding for stabilizing intermediate structures, and suffer from slow inference speeds. Therefore, we formulate construction as a probabilistic next-block generation task with multiple potential assembly actions and multiple potential task conditioning modalities. Concurrently, we explicitly consider the utility of scaffolding by introducing an auxiliary scaffold block token. We present Scaffold Multimodal Monte Carlo (ScaffoldM3C), a multimodal, lightweight, auto-regressive model for stable block-based construction, that proposes a set of next-step candidate blocks. Leveraging these candidates, we utilize Sequential Monte Carlo (SMC) to maintain a population of possible assembly sequences, allowing us to consider multiple, potentially different, assembly directions simultaneously. We train our multimodal architecture by extending the StableText2Brick dataset to contain image conditioning prompts and scaffold-stabilized build sequences. ScaffoldM3C is 4x smaller than competing baselines, yielding a 5x to 20x speedup during inference, while achieving comparable construction quality to state-of-the-art methods and higher overall stability. We demonstrate the effectiveness of our approach through simulations and real-world robot assembly demonstrations.

Figures & tables

Fig. 1: We present ScaffoldM3C, a lightweight, open source 225M parameter Vision-Language-Construction (VLC) model, trained on 553K different conditioned assembly sequences. Given a starting build location, ScaffoldM3C auto-regressively generates a condition aligned step-by-step assembly plan. At each step, ScaffoldM3C proposes a set of next step candidate blocks from which to choose, while multiple possible assembly sequences are considered simultaneously through a sequential Monte Carlo algorithm, populated based on the candidate block likelihoods. We demonstrate the success of our architecture through simulated and robotic real world construction.
Fig. 2: Stability constraints for candidate block placements. Each panel shows a target block being evaluated before placement: the centroid cc is projected vertically onto the contact plane at pp , and the support region H\mathcal{H} (blue) is the 2D convex hull of the contact faces (hatched) provided by the blocks underneath. (a) A block resting on the ground is stable by direct ground contact. (b) With a single support, pp falls outside H\mathcal{H} and the placement is considered unstable. (c) With supports at both ends, H\mathcal{H} spans the gap between their contact faces, so p∈Hp\in\mathcal{H} and the placement is stable. (d) An auxiliary scaffold block (gray) restores the missing support, enlarging H\mathcal{H} so the otherwise-unstable placement becomes feasible.
Fig. 3: A. Model input: a partial build sequence of blocks and a conditioning prompt (e.g., text, images or some combination of both). B. Our proposed architecture. We utilize several interwoven layers stacked in series as our transformer backbone. The output of the final interwoven layer is fed to a windowed attention layer that ties relevant local candidate-level features to each other before being passed to three cascading model heads that predict the candidate block positions, classes and likelihoods. C. We select the next block through probabilistic Sampling using either Top- KK or SMC.
Fig. 4: Data generation pipeline extending the StableText2Brick dataset. Build sequences are updated with scaffold blocks to satisfy our stability constraints. Images are generated either by simulating the construction and rendering the final structure specified by a given build plan or by providing a text-based guidance prompt to a Qwen text-to-image model.
  Per StructureAcross Structures
 Methodecollision↓e_{\text{collision}}\downarrowsinter↑s_{\text{inter}}\uparrowsfinal↑s_{\text{final}}\uparrowasim↑a_{\text{sim}}\uparrowagt↑a_{\text{gt}}\uparrowcmin↑c_{\text{min}}\uparrowcsem↑c_{\text{sem}}\uparrowblocks ≈\approxscaffolds ≈\approxsoverall↑s_{\text{overall}}\uparrowerate↑e_{\text{rate}}\uparrowefeasible↑e_{\text{feasible}}\uparrowInference Time (s) ↓\downarrow
TextGround truth$$0.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}$$1.000\pm{\scriptstyle0.000}N/AN/A$$111.0271\pm{\scriptstyle70.0497}$$28.0313\pm{\scriptstyle22.3448}$$100 %\ \%$$100 %\ \%$$100 %\ \%N/A
BrickGPT w/o rej.$$7.0566\pm{\scriptstyle10.1357}$$0.7638\pm{\scriptstyle0.1107}$$0.7644\pm{\scriptstyle0.1101}$$0.2692\pm{\scriptstyle0.0288}$$0.8234\pm{\scriptstyle0.0775}$$0.9006\pm{\scriptstyle0.9280}$$4.5665\pm{\scriptstyle5.5986}$$106.1500\pm{\scriptstyle72.1203}$$0.0000\pm{\scriptstyle0.0000}$$2.7254 %\ \%$$28.0922 %\ \%$$2.3061 %\ \%$$11.4646\pm{\scriptstyle9.7937}
BrickGPT w/ rej.$$0.0000\pm{\scriptstyle0.0000}$$0.7567\pm{\scriptstyle0.1118}$$0.7568\pm{\scriptstyle0.1117}$$0.2689\pm{\scriptstyle0.0285}$$0.8203\pm{\scriptstyle0.0813}$$0.9175\pm{\scriptstyle1.1091}$$4.6517\pm{\scriptstyle5.6934}$$110.9102\pm{\scriptstyle83.2579}$$0.0000\pm{\scriptstyle0.0000}$$3.1315 %\ \%$$100.0000 %\ \%$$3.1315 %\ \%$$13.8635\pm{\scriptstyle16.8635}
BrickGPT-Scaffold$$0.4280\pm{\scriptstyle2.4044}$$0.9183\pm{\scriptstyle0.0557}$$0.9188\pm{\scriptstyle0.0552}$$0.2671\pm{\scriptstyle0.0285}$$0.8326\pm{\scriptstyle0.0728}$$0.8231\pm{\scriptstyle0.6533}$$4.2404\pm{\scriptstyle5.2198}$$195.1023\pm{\scriptstyle158.9575}$$42.4029\pm{\scriptstyle43.6773}$$11.4823 %\ \%$$90.1879 %\ \%$$11.2735 %\ \%$$64.5053\pm{\scriptstyle215.9353}
ours (Top- KK )$$0.0000\pm{\scriptstyle0.0000}$$0.8149\pm{\scriptstyle0.1619}$$0.8192\pm{\scriptstyle0.1580}$$0.2695\pm{\scriptstyle0.0269}$$0.8145\pm{\scriptstyle0.0885}$$1.1479\pm{\scriptstyle1.1855}$$5.5036\pm{\scriptstyle6.2879}$$89.5908\pm{\scriptstyle55.2851}$$21.3132\pm{\scriptstyle17.8939}$$20.0418 %\ \%$$100.0000 %\ \%$$20.0418 %\ \%$$0.5827\pm{\scriptstyle0.3574}
TABLE I: Average quantitative results without rollbacks across 479479 text and 288288 image test prompts. Arrows indicate direction of improvement, while ≈\approx specifies the desired target. Performance is highlighted as best , second best , and worst .
Fig. 5: Qualitative results of our method using both text and image prompts taken from the test dataset. We compare against our baseline using text prompts. Our baseline cannot interpret image prompts, therefore we leave “Not Applicable” for its results.
Fig. 6: Hardware experiments using a 6-DOF XFactory robotic arm to build three different structures. During assembly, nine different blocks were used. Overhanging blocks that would normally be unstable are supported by scaffolding.
Fig. 7: Effect of Sequential Monte Carlo (SMC) population size on generated structures. Each row shows results for a different target object as the number of candidate build sequences increases from 1 to 320, with the ground-truth (GT) structure shown on the right. Larger populations generally yield more complete and coherent structures that better match the target geometry, with diminishing improvements beyond 160 sequences.
Populationsoverall↑s_{\text{overall}}\uparrowsfinal↑s_{\text{final}}\uparrowInference Time (s) ↓\downarrow
1$$20.8333 %\ \%$$0.7977\pm{\scriptstyle0.1890}$$0.4216\pm{\scriptstyle0.2961}
4$$20.8333 %\ \%$$0.8120\pm{\scriptstyle0.1645}$$0.5560\pm{\scriptstyle0.3301}
10$$20.8333 %\ \%$$0.8531\pm{\scriptstyle0.1351}$$0.8866\pm{\scriptstyle0.5078}
20$$25.0000 %\ \%$$0.8773\pm{\scriptstyle0.1215}$$1.5963\pm{\scriptstyle1.2019}
40$$20.8333 %\ \%$$0.8785\pm{\scriptstyle0.1051}$$2.6385\pm{\scriptstyle2.2327}
80$$33.3333 %\ \%$$0.9104\pm{\scriptstyle0.0924}$$4.9324\pm{\scriptstyle4.7820}
TABLE II: Ablation Study on Sequential Monte Carlo Population Size using 24 different test samples.
Fig. 8: Heatmaps of intermediate stability, final stability, standard block count, and scaffold block count under varying candidate and class distribution temperatures (evaluated across 24 test samples). Structural stability is particularly sensitive to the class distribution temperature, with higher temperatures potentially favoring implausible block-type selections and disrupting the assembly sequence. In contrast, increasing the candidate distribution temperature generally improves stability and increases both standard and scaffold block counts, suggesting that broader exploration of lower-probability placement locations allows SMC to identify more successful build sequences.
Fig. 9: Belief state of Sequential Monte Carlo after 12 steps when asked to generate a table. A population size of 4 was used. Each belief state considers a separate and independent set of candidate blocks, conditioned on previously placed blocks of the associated assembly sequence. Multiple particles allow for alternative block placement decisions to be considered simultaneously, allowing the model to potentially escape local minima.
Fig. 10: Possible failure modes of sequence generation using Sequential Monte Carlo sampling. (1) Positional errors lead to assembly plan degradation or out-of-distribution (OOD) states. (2) The model misses prompt details, generating a similar but irrelevant structure. (3) The model prematurely outputs an EOS token. (4) Errors in predicted positions result in block shifting.

Explore similar work

CardsList

  1. Brick-Composer: Using MLLMs for Assembly with Diverse Bricks

    Jun 3, 2026Jiateng Liu, Bingxuan Li, Zhenhailong Wang +8AssemblyMultimodal Large Language Models

    UIUC1 · Stevens Institute of Technology2 · Northwestern University3

  2. Rollback-Free Stable Brick Structures Generation

    May 7, 2026Chenhui Xu, Ziyue Bai, Fuxun Yu +2RollbackOffline Reinforcement Learning

    University of Buffalo · University of Maryland, College Park · Microsoft +1

  3. BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization

    May 25, 2026Zhengyang Ni, Feng Yan, Yu Guo +1Geometric Reconstruction3D Representation

    Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence · Institute of Artificial Intelligence and Robotics