Organizations: Fudan University† · Institute of Artificial Intelligence (TeleAI), China Telecom† · Tianjin University · Northwestern Polytechnical University · City University of Hong Kong · Tsinghua University
Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. Recent methods based on vision-language models (VLMs) largely rely on coarse 2D perception and struggle to perform accurate reasoning over complex 3D geometry. To address this limitation, we propose AssemLM, a spatial multimodal large language model for robotic assembly that integrates assembly manuals, point clouds, and textual instructions to predict task-critical 6D assembly poses with explicit geometric understanding. To bridge raw 3D perception and high-level linguistic reasoning, AssemLM employs a specialized point cloud encoder to extract fine-grained geometric and rotational features for accurate 3D spatial reasoning in assembly tasks. In addition, we introduce AssemBench, a large-scale benchmark for assembly-oriented spatial reasoning with over 900K multimodal samples and precise 6D pose annotations, extending evaluation from 2D grounding to full 3D geometric inference. Extensive experiments and real-robot evaluations demonstrate that AssemLM achieves state-of-the-art 6D pose reasoning performance and effectively supports fine-grained, multi-step assembly tasks in real-world settings. Code, models, and the AssemBench dataset will be made publicly available.
Figures & tables
Figure 1: AssemLM and AssemBench: Scaling Spatial Reasoning for Robotic Assembly. (Left & Right) We introduce AssemBench , a large-scale multimodal dataset and its generation pipeline, providing rich object categories, manuals, text instructions, and 6D pose annotations. (Middle) The proposed AssemLM architecture (top) processes these multimodal inputs by coupling SO(3) -equivariant geometric features with a Qwen3-VL backbone to predict 6D assembly poses, facilitating real-world execution of fine-grained tasks on a Flexiv Rizon 4s arm (bottom).
Figure 2: Overview of the AssemLM architecture. AssemLM integrates visual manuals, 3D point clouds, and text instructions into a Qwen3-VL backbone. Visual inputs are projected into the language space, incorporating a DeepStack mechanism for intermediate feature injection. Concurrently, point clouds of fixed and moving parts are processed by an SO(3) -equivariant encoder, projecting geometric features ( F and G ) into point cloud tokens. These multimodal embeddings are jointly processed to autoregressively predict discrete pose tokens, subsequently decoded into precise 6D assembly poses.
Figure 3: Overview of the AssemBench dataset. (Top) Statistics demonstrating the vast scale and diverse category distributions of AssemBench. (Middle) Multimodal data examples for sequential assembly, including multi-style visual manuals, point clouds, text instructions, and precise 6D poses. (Bottom) Our automated data generation pipeline, which transforms web-sourced image resources into high-fidelity multimodal data to provide large-scale supervision for 6D pose reasoning.
Category
TwoByTwo [ 33 ]
GPT-5.2 [ 37 ]
DeepSeek-V3.2 [ 25 ]
AssemLM (Ours)
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
Bottle
0.1953
0.1692
16.7%
0.2052
0.1728
19.2%
0.1826
0.2564
0.0%
0.0226
0.0102
88.9%
Plug
0.0483
0.0285
66.7%
0.2833
0.3340
0.0%
0.3396
0.6262
0.0%
0.0340
0.0053
100.0%
Postbox
0.2710
0.3005
0.0%
0.1908
0.1823
0.0%
0.3050
0.5379
0.0%
0.0373
0.0170
78.6%
Nut
0.0841
0.0879
12.5%
0.1519
0.0830
41.7%
0.1964
0.2006
16.7%
0.0032
0.0035
100.0%
Coin
0.1366
0.0975
50.0%
0.1925
0.2541
12.5%
0.2500
0.4180
0.0%
0.0417
0.0109
87.5%
Table 1: Quantitative Results on Multi-Category Assembly. AssemLM is compared against baselines (TwoByTwo, GPT-5.2, DeepSeek-V3.2). Metrics: Translation RMSE, Symmetric Chamfer Distance, and Success Rate. The “All" indicates the weighted average across three macro-categories. Due to space constraints, full individual sub-categories results are provided in Appx. C .
Category
TwoByTwo
GPT-5.2
DeepSeek-V3.2
AssemLM (Ours)
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
Bench
0.1826
0.1983
9.9%
0.2213
0.3435
1.4%
0.3304
0.5922
0.0%
0.0330
0.0250
81.4%
Chair
0.1810
0.1695
7.1%
0.2189
0.2661
5.4%
0.2627
0.4273
1.7%
0.0205
0.0164
79.5%
Desk
0.2401
0.2973
2.5%
0.1957
0.2585
7.5%
0.2989
0.5274
5.1%
0.0255
0.0225
82.1%
All
0.2012
0.2217
6.5%
0.2120
0.2893
4.8%
0.2973
0.5156
2.3%
0.0263
0.0213
81.0%
Table 2: Zero-Shot Generalization on the IKEA Dataset. Quantitative results for out-of-distribution (OOD) evaluation. Bench and Desk represent unseen categories. Metrics: Translation RMSE, Symmetric Chamfer Distance, and Success Rate. The “All" indicates unweighted average.
Figure 4: Real-world setup. We evaluate four challenging tasks using a Flexiv Rizon 4s arm.
Table 7
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Visualization of real-world asset processing. We illustrate the data pipeline for four manipulation tasks: Insert Plug , Store Cans , Insert Flower , and Build Blocks . For each task, the figure displays the physical setup, the individual physical objects, the reconstructed high-fidelity 3D assets, and the final sampled point clouds used for model inference.
Figure 6: Canonical Coordinate System.
Figure 7: Real-world assembly steps and corresponding instruction manuals.
AssemLM w/o Manual
AssemLM w/o Instruction
AssemLM (Ours)
R-T ↓
SCD(R) ↓
SCD ↓
SR ↑
R-T ↓
SCD(R) ↓
SCD ↓
SR ↑
R-T ↓
SCD(R) ↓
SCD ↓
SR ↑
DO
0.0510
0.0066
0.0335
76.3%
0.0381
0.0065
0.0135
82.6%
0.0317
0.0060
0.0096
86.9%
Fur.
0.1569
0.0119
0.2115
33.0%
0.0249
0.0113
0.0233
74.0%
0.0194
0.0029
0.0139
91.5%
Frag.
0.0770
0.0081
0.1058
67.4%
0.0127
0.0083
0.0142
86.4%
0.0097
0.0062
0.0092
89.8%
Fur.*
0.1853
0.0245
0.2429
14.8%
0.0570
0.0206
0.0727
56.1%
0.0557
0.0097
0.0613
69.6%
AssemLM w/o DeepStack
AssemLM w/o Equivariant
AssemLM w/o Warm-up Stage
Appendix
Table 5: Ablation studies on model modalities and architectural modules.
ManualPA [ 49 ]
SE(3)-Assembly [ 47 ]
R-T ↓
SCD(R) ↓
SCD ↓
SR ↑
R-T ↓
SCD(R) ↓
SCD ↓
SR ↑
DO
0.0787
0.0075
0.0122
83.4%
0.1525
0.0186
0.0532
15.1%
Fur.
0.1073
0.0071
0.0170
71.5%
0.1583
0.0386
0.0975
10.5%
Frag.
0.0918
0.0052
0.0101
89.1%
0.16
0.0162
0.0497
11.3%
Fur.*
0.1041
0.0129
0.0216
54.9%
0.1551
0.0451
0.0839
2.1%
Appendix
Table 6: Additional baseline methods.
Category
TwoByTwo [ 33 ]
GPT-5.2 [ 37 ]
DeepSeek-V3.2 [ 25 ]
AssemLM (Ours)
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
RMSE(T) ↓
SCD ↓
SR ↑
KitchenPort
0.1544
0.1131
16.7%
0.1224
0.0896
0.0%
0.2181
0.2252
0.0%
0.0374
0.0153
71.4%
Bottle
0.1953
0.1692
16.7%
0.2052
0.1728
19.2%
0.1826
0.2564
0.0%
0.0226
0.0102
88.9%
Kettle
0.1148
0.0631
20.0%
0.2066
0.1665
0.0%
0.2773
0.3789
0.0%
0.0306
0.0086
66.7%
Coffeemachine
0.2833
0.3311
0.0%
0.2231
0.1525
0.0%
0.3412
0.3044
0.0%
0.0304
0.0051
100.0%
Cup
0.1583
0.0777
0.0%
0.1318
0.0856
0.0%
0.1792
0.2585
0.0%
0.0066
0.0014
100.0%
Appendix
Table 7: Quantitative Results across Multi-category Assembly Tasks. We evaluate AssemLM against a specialized SE(3) -equivariant model (TwoByTwo) and state-of-the-art foundation models (GPT-5.2 and DeepSeek-V3.2). Performance is reported via Translation RMSE (RMSE(T)), Symmetric Chamfer Distance (SCD), and Success Rate (SR, defined as SCD<0.02 ). For cross-dataset consistency, related sub-categories (e.g., Bottle variants) are consolidated. The "All" row reflects the weighted average across Daily Objects, Furniture, and Fragments. Bold indicates the best performance in each category.
Figure 8: Ablation study on dataset scale, rotation range, and tokenizer choice. We evaluate the impact of different design components using Translation RMSE (left) and Symmetric Chamfer Distance (right) across diverse daily objects. The All bars represent the average across all categories.
Figure 9: Visualization of Different Types of Manuals.
Figure 10: Representative examples from AssemBench.
Figure 11: Visualization of assembly pose predictions by AssemLM.
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.
Recent vision-language models (VLMs) show strong capabilities in robotic perception and spatial reasoning, yet their ability to reason about complex mechanical assemblies remains underexplored. We introduce OmniCAD, a large-scale benchmark for assembly-aware 3D spatial reasoning across diverse industrial systems, including robotic mechanisms, automotive components, aerospace structures, and agricultural machinery. OmniCAD contains 25k mechanical assemblies, with an average of 12 parts per assembly and 21 types of mate relationships. Each assembly includes a human-verified ground-truth 3D model and renderings from 20 viewpoints. The benchmark evaluates three capabilities: (1) component-level 3D spatial reasoning, requiring prediction of part positions and orientations; (2) part-to-part relational reasoning, requiring identification of mating relationships and assembly constraints; and (3) tool-augmented agentic reasoning, where models iteratively select viewpoints, inspect visual evidence, and refine predictions. Experiments show that current VLMs struggle with industrial assembly reasoning, often producing inaccurate poses, invalid mating relationships, part interpenetration, and degraded performance as assembly complexity increases. We will open-source the benchmark, evaluation code, and tool interfaces to support research on accurate, physically valid, and scalable 3D assembly reasoning.
Mingjia Wang, Taiting Lu, Ziwei Dong +25
Shanghai Jiao Tong University · Pennsylvania State University · Independent Researcher +2
Multimodal large language models (MLLMs) have made significant progress in visual understanding, but precise 3D spatial reasoning integrated with physical environment remains difficult. Furniture assembly requires not only recovering step-level operations from diagrammatic manuals, but also translating semantic attachment relations into 6D pose updates that enable parts to physically interact with the environment and previously assembled components. To study this problem, we propose AssemState, a zero-shot framework for manual and physical-state-guided furniture assembly. It firstly employs anchor-guided boundary assembly states to decompose manual pages into single-part operations and recover an assembly-tree. Then, it uses iterative after-state feedback refinement to guide successive (SE(3)) updates and corrections, and validates their physical plausibility through simulation-based release tests. Experiments show that compared with the strongest prior baseline, AssemState improves F1 from 38.58% to 62.80% and Tree Exact Match from 28.24% to 53.92% for assembly-tree recovery. On 243 independently evaluated part-level operations, our proposed iterative refinement improves judge-accepted operations from 0 to 5.3% and reduces mean Chamfer distance from 5.4111 to 1.7744. However, visually plausible candidate poses may still suffer from collision, floating, mirror-orientation errors, incomplete seating, and wrong-side attachment. These results show that AssemState improves operation-structure recovery and selected local pose metrics, while MLLMs remain limited for spatial relationship reasoning.
Zhiyuan Qi, Jierui Li, Yifan Shen +2
Tsinghua University · University of Illinois Urbana-Champaign · Xidian University
Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified scenarios, overlooking shape complexities and assembly trajectories in industrial assemblies. We introduce AssemblyBench, a synthetic dataset of 2,789 industrial objects with multimodal instruction manuals, corresponding 3D part models, and part assembly trajectories. We also propose a transformer-based model, AssemblyDyno, which uses the instructional manual and the 3D shape of each part to jointly predict assembly order and part assembly trajectories. AssemblyDyno outperforms prior works in both assembly pose estimation and trajectory feasibility, where the latter is evaluated by our physics-based simulations.
Danrui Li, Jiahao Zhang, Bernhard Egger +4
Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1