AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly

Zhi Jing; Jinbin Qiao; Ouyang Lu; Jicong Ao; Shuang Qiu; Huazhe Xu; Yu-Gang Jiang; Chenjia Bai

cs.ROApr 10, 2026

AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly

Authors: Zhi Jing, Jinbin Qiao, Ouyang Lu, Jicong Ao, Shuang Qiu, Huazhe Xu, Yu-Gang Jiang, Chenjia Bai

Organizations: Fudan University† · Institute of Artificial Intelligence (TeleAI), China Telecom† · Tianjin University · Northwestern Polytechnical University ·

Abstract

Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. Recent methods based on vision-language models (VLMs) largely rely on coarse 2D perception and struggle to perform accurate reasoning over complex 3D geometry. To address this limitation, we propose AssemLM, a spatial multimodal large language model for robotic assembly that integrates assembly manuals, point clouds, and textual instructions to predict task-critical 6D assembly poses with explicit geometric understanding. To bridge raw 3D perception and high-level linguistic reasoning, AssemLM employs a specialized point cloud encoder to extract fine-grained geometric and rotational features for accurate 3D spatial reasoning in assembly tasks. In addition, we introduce AssemBench, a large-scale benchmark for assembly-oriented spatial reasoning with over 900K multimodal samples and precise 6D pose annotations, extending evaluation from 2D grounding to full 3D geometric inference. Extensive experiments and real-robot evaluations demonstrate that AssemLM achieves state-of-the-art 6D pose reasoning performance and effectively supports fine-grained, multi-step assembly tasks in real-world settings. Code, models, and the AssemBench dataset will be made publicly available.

Figures & tables

Figure 1: AssemLM and AssemBench: Scaling Spatial Reasoning for Robotic Assembly. (Left & Right) We introduce AssemBench , a large-scale multimodal dataset and its generation pipeline, providing rich object categories, manuals, text instructions, and 6D pose annotations. (Middle) The proposed AssemLM architecture (top) processes these multimodal inputs by coupling $SO(3)$ -equivariant geometric features with a Qwen3-VL backbone to predict 6D assembly poses, facilitating real-world execution of fine-grained tasks on a Flexiv Rizon 4s arm (bottom).

Figure 2: Overview of the AssemLM architecture. AssemLM integrates visual manuals, 3D point clouds, and text instructions into a Qwen3-VL backbone. Visual inputs are projected into the language space, incorporating a DeepStack mechanism for intermediate feature injection. Concurrently, point clouds of fixed and moving parts are processed by an $SO(3)$ -equivariant encoder, projecting geometric features ( $F$ and $G$ ) into point cloud tokens. These multimodal embeddings are jointly processed to autoregressively predict discrete pose tokens, subsequently decoded into precise 6D assembly poses.

Figure 3: Overview of the AssemBench dataset. (Top) Statistics demonstrating the vast scale and diverse category distributions of AssemBench. (Middle) Multimodal data examples for sequential assembly, including multi-style visual manuals, point clouds, text instructions, and precise 6D poses. (Bottom) Our automated data generation pipeline, which transforms web-sourced image resources into high-fidelity multimodal data to provide large-scale supervision for 6D pose reasoning.

Category	TwoByTwo [ 33 ]			GPT-5.2 [ 37 ]			DeepSeek-V3.2 [ 25 ]			AssemLM (Ours)
Category	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$
Bottle	0.1953	0.1692	16.7%	0.2052	0.1728	19.2%	0.1826	0.2564	0.0%	0.0226	0.0102	88.9%
Plug	0.0483	0.0285	66.7%	0.2833	0.3340	0.0%	0.3396	0.6262	0.0%	0.0340	0.0053	100.0%
Postbox	0.2710	0.3005	0.0%	0.1908	0.1823	0.0%	0.3050	0.5379	0.0%	0.0373	0.0170	78.6%
Nut	0.0841	0.0879	12.5%	0.1519	0.0830	41.7%	0.1964	0.2006	16.7%	0.0032	0.0035	100.0%
Coin	0.1366	0.0975	50.0%	0.1925	0.2541	12.5%	0.2500	0.4180	0.0%	0.0417	0.0109	87.5%

Table 1: Quantitative Results on Multi-Category Assembly. AssemLM is compared against baselines (TwoByTwo, GPT-5.2, DeepSeek-V3.2). Metrics: Translation RMSE, Symmetric Chamfer Distance, and Success Rate. The “All" indicates the weighted average across three macro-categories. Due to space constraints, full individual sub-categories results are provided in Appx. C .

Category	TwoByTwo			GPT-5.2			DeepSeek-V3.2			AssemLM (Ours)
Category	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$
Bench	0.1826	0.1983	9.9%	0.2213	0.3435	1.4%	0.3304	0.5922	0.0%	0.0330	0.0250	81.4%
Chair	0.1810	0.1695	7.1%	0.2189	0.2661	5.4%	0.2627	0.4273	1.7%	0.0205	0.0164	79.5%
Desk	0.2401	0.2973	2.5%	0.1957	0.2585	7.5%	0.2989	0.5274	5.1%	0.0255	0.0225	82.1%
All	0.2012	0.2217	6.5%	0.2120	0.2893	4.8%	0.2973	0.5156	2.3%	0.0263	0.0213	81.0%

Table 2: Zero-Shot Generalization on the IKEA Dataset. Quantitative results for out-of-distribution (OOD) evaluation. Bench and Desk represent unseen categories. Metrics: Translation RMSE, Symmetric Chamfer Distance, and Success Rate. The “All" indicates unweighted average.

Figure 4: Real-world setup. We evaluate four challenging tasks using a Flexiv Rizon 4s arm.

Table 7

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualization of real-world asset processing. We illustrate the data pipeline for four manipulation tasks: Insert Plug , Store Cans , Insert Flower , and Build Blocks . For each task, the figure displays the physical setup, the individual physical objects, the reconstructed high-fidelity 3D assets, and the final sampled point clouds used for model inference.

Figure 7: Real-world assembly steps and corresponding instruction manuals.

	AssemLM w/o Manual				AssemLM w/o Instruction				AssemLM (Ours)
	R-T $\downarrow$	SCD(R) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	R-T $\downarrow$	SCD(R) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	R-T $\downarrow$	SCD(R) $\downarrow$	SCD $\downarrow$	SR $\uparrow$
DO	0.0510	0.0066	0.0335	76.3%	0.0381	0.0065	0.0135	82.6%	0.0317	0.0060	0.0096	86.9%
Fur.	0.1569	0.0119	0.2115	33.0%	0.0249	0.0113	0.0233	74.0%	0.0194	0.0029	0.0139	91.5%
Frag.	0.0770	0.0081	0.1058	67.4%	0.0127	0.0083	0.0142	86.4%	0.0097	0.0062	0.0092	89.8%
Fur.*	0.1853	0.0245	0.2429	14.8%	0.0570	0.0206	0.0727	56.1%	0.0557	0.0097	0.0613	69.6%
	AssemLM w/o DeepStack				AssemLM w/o Equivariant				AssemLM w/o Warm-up Stage

Appendix

Table 5: Ablation studies on model modalities and architectural modules.

	ManualPA [ 49 ]				SE(3)-Assembly [ 47 ]
	R-T $\downarrow$	SCD(R) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	R-T $\downarrow$	SCD(R) $\downarrow$	SCD $\downarrow$	SR $\uparrow$
DO	0.0787	0.0075	0.0122	83.4%	0.1525	0.0186	0.0532	15.1%
Fur.	0.1073	0.0071	0.0170	71.5%	0.1583	0.0386	0.0975	10.5%
Frag.	0.0918	0.0052	0.0101	89.1%	0.16	0.0162	0.0497	11.3%
Fur.*	0.1041	0.0129	0.0216	54.9%	0.1551	0.0451	0.0839	2.1%

Appendix

Table 6: Additional baseline methods.

Category	TwoByTwo [ 33 ]			GPT-5.2 [ 37 ]			DeepSeek-V3.2 [ 25 ]			AssemLM (Ours)
Category	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$	RMSE(T) $\downarrow$	SCD $\downarrow$	SR $\uparrow$
KitchenPort	0.1544	0.1131	16.7%	0.1224	0.0896	0.0%	0.2181	0.2252	0.0%	0.0374	0.0153	71.4%
Bottle	0.1953	0.1692	16.7%	0.2052	0.1728	19.2%	0.1826	0.2564	0.0%	0.0226	0.0102	88.9%
Kettle	0.1148	0.0631	20.0%	0.2066	0.1665	0.0%	0.2773	0.3789	0.0%	0.0306	0.0086	66.7%
Coffeemachine	0.2833	0.3311	0.0%	0.2231	0.1525	0.0%	0.3412	0.3044	0.0%	0.0304	0.0051	100.0%
Cup	0.1583	0.0777	0.0%	0.1318	0.0856	0.0%	0.1792	0.2585	0.0%	0.0066	0.0014	100.0%

Appendix

Table 7: Quantitative Results across Multi-category Assembly Tasks. We evaluate AssemLM against a specialized

SE(3)

-equivariant model (TwoByTwo) and state-of-the-art foundation models (GPT-5.2 and DeepSeek-V3.2). Performance is reported via Translation RMSE (RMSE(T)), Symmetric Chamfer Distance (SCD), and Success Rate (SR, defined as

SCD<0.02

). For cross-dataset consistency, related sub-categories (e.g., Bottle variants) are consolidated. The "All" row reflects the weighted average across Daily Objects, Furniture, and Fragments. Bold indicates the best performance in each category.

Figure 8: Ablation study on dataset scale, rotation range, and tokenizer choice. We evaluate the impact of different design components using Translation RMSE (left) and Symmetric Chamfer Distance (right) across diverse daily objects. The All bars represent the average across all categories.

Figure 9: Visualization of Different Types of Manuals.

Figure 10: Representative examples from AssemBench.

Figure 11: Visualization of assembly pose predictions by AssemLM.

3D Spatial Reasoning

3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.

See all 202 papers

OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies
Aug 23, 2026Mingjia Wang, Taiting Lu, Ziwei Dong +25Spatial Reasoning Benchmarks 3D Spatial Reasoning
Shanghai Jiao Tong University · Pennsylvania State University · Independent Researcher +2
AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly
Oct 6, 2026Zhiyuan Qi, Jierui Li, Yifan Shen +2Robotic Assembly 3D Spatial Reasoning
Tsinghua University · University of Illinois Urbana-Champaign · Xidian University
AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects
May 13, 2026Danrui Li, Jiahao Zhang, Bernhard Egger +4Object Pose Estimation Robot Manipulation Benchmarks
Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1

Aug 23, 2026J/K move · Enter open · S save

OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies

Mingjia Wang, Taiting Lu, Ziwei Dong, Sisong Bei+24

Shanghai Jiao Tong University · Pennsylvania State University · Independent Researcher+2

Spatial Reasoning Benchmarks 3D Spatial Reasoning Robotic Assembly Industrial Robotics

Open paper AskPDF arXiv

Recent vision-language models (VLMs) show strong capabilities in robotic perception and spatial reasoning, yet their ability to reason about complex mechanical assemblies remains underexplored. We introduce OmniCAD, a large-scale benchmark for assembly-aware 3D spatial reasoning across diverse industrial systems, including robotic mechanisms, automotive components, aerospace structures, and agricultural machinery. OmniCAD contains 25k mechanical assemblies, with an average of 12 parts per assembly and 21 types of mate relationships. Each assembly includes a human-verified ground-truth 3D model and renderings from 20 viewpoints. The benchmark evaluates three capabilities: (1) component-level 3D spatial reasoning, requiring prediction of part positions and orientations; (2) part-to-part relational reasoning, requiring identification of mating relationships and assembly constraints; and (3) tool-augmented agentic reasoning, where models iteratively select viewpoints, inspect visual evidence, and refine predictions. Experiments show that current VLMs struggle with industrial assembly reasoning, often producing inaccurate poses, invalid mating relationships, part interpenetration, and degraded performance as assembly complexity increases. We will open-source the benchmark, evaluation code, and tool interfaces to support research on accurate, physically valid, and scalable 3D assembly reasoning.

Abstract

3D Spatial Reasoning

Explore similar work

OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies

AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly

AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects