Recent vision-language models (VLMs) show strong capabilities in robotic perception and spatial reasoning, yet their ability to reason about complex mechanical assemblies remains underexplored. We introduce OmniCAD, a large-scale benchmark for assembly-aware 3D spatial reasoning across diverse industrial systems, including robotic mechanisms, automotive components, aerospace structures, and agricultural machinery. OmniCAD contains 25k mechanical assemblies, with an average of 12 parts per assembly and 21 types of mate relationships. Each assembly includes a human-verified ground-truth 3D model and renderings from 20 viewpoints. The benchmark evaluates three capabilities: (1) component-level 3D spatial reasoning, requiring prediction of part positions and orientations; (2) part-to-part relational reasoning, requiring identification of mating relationships and assembly constraints; and (3) tool-augmented agentic reasoning, where models iteratively select viewpoints, inspect visual evidence, and refine predictions. Experiments show that current VLMs struggle with industrial assembly reasoning, often producing inaccurate poses, invalid mating relationships, part interpenetration, and degraded performance as assembly complexity increases. We will open-source the benchmark, evaluation code, and tool interfaces to support research on accurate, physically valid, and scalable 3D assembly reasoning.
Figures & tables
Figure 1: VLMs exhibit limited capability in assembling 3D mechanical components into physically valid configurations. Frequent failures, including incorrect component poses, severe part interpenetration, missing or duplicated component instances, and inconsistent mating relationships, reveal substantial shortcomings in assembly-aware spatial and relational reasoning.
Figure 2: Statistical overview of the OmniCAD benchmark. From left to right, we show (a) a word cloud summarizing the application domains, (b) the distribution of component instances per assembly, (c) the distribution of unique component-library entries per assembly, and (d) the distribution of mating constraints per assembly.
Dataset
#Asm.
Avg. Comp.
#View
#Mate Type
Mate Graph
3D Object
B-Rep
Mesh
3D CAD Assembly ( Lupinetti et al. 2019 )
137
93.31
✗
5
✓
✓
✗
AutoMate ( Jones et al. 2021 )
92.53K
N/R
✗
8
✓
✓
✗
JoinABLe ( Willis et al. 2022 )
8.25K
18.72
✗
7
✓
✓
✓
Assemble Them All ( Tian et al. 2022 )
12.97K
N/R
✗
✗
✗
✗
✓
GRAPE ( Gajek, Schiendorfer, and Reif 2023 )
35.66K †
10.34 †
✗
✗
✗
✗
✗
Table 1: Comparison of publicly available 3D assembly datasets. #Asm. denotes the total number of assemblies. Avg. Comp. denotes the average number of component instances per assembly. #View denotes the number of reference views. A green check mark indicates available, while a red cross indicates unavailable. N/R means not reported. For GRAPE, † denotes statistics aggregated from its three independently reported component catalogs. For AssemblyBench, ‡ denotes the selected reference view, while the number of candidate isometric views is not reported.
Figure 3: Overview of the OmniCAD benchmark with a representative example. From left to right, the figure shows the assembly library and its corresponding mate graph, the ground-truth 3D assembled model, and multi-view renders generated from the vertices of the surrounding view cube.
Figure 4: Agentic assembly framework for evaluating LMM tool use in mechanical assembly reconstruction.
Model
Size
Experiment
Component
Pose
Mate Graph
Physical / Shape
Efficiency / Success
ID-NR (%) ↑
ID-R (%) ↑
Pos@ τ (%) ↑
Rot@ τ (%) ↑
CD i ↓
PA (%) ↑
PairF1 (%) ↑
TypeAcc (%) ↑
GSim (%) ↑
C-Free (%) ↑
SCD ↓
PosErr ↓
RT ↓
PR (%) ↑
Commercial Chatbot Systems
GPT-5.5
-
SameColor
100.00
60.00
15.30
48.40
245.6
32.00
36.40
44.10
12.10
3.10
144.0
368.9
111.0
37.00
RefOBJ
100.00
71.40
0.00
84.70
240.8
14.70
46.10
59.70
23.30
11.80
152.4
316.3
99.5
21.00
ColorView
92.90
57.10
15.70
43.30
349.3
25.00
35.70
44.00
8.40
10.00
232.2
481.9
171.3
35.00
MG+Color
–
–
27.90
63.20
279.7
46.70
–
–
–
0.00
186.5
328.6
83.1
20.00
Table 2: One-shot evaluation of LMMs on component-level CAD assembly generation. Entries marked ‘–’ denote metrics that are not applicable by design: ID-NR/ID-R are omitted when the input mate graph already specifies the component-instance inventory; PairF1/TypeAcc/GSim are omitted for pose-only mate-graph-conditioned settings that do not require predicted mate edges. PR is computed per setting as successful final outputs divided by total attempts.
Model
Size
Agent Setting
Component
Pose
Mate Graph
Physical / Shape
Efficiency / Success
ID-NR (%) ↑
ID-R (%) ↑
Pos@ τ (%) ↑
Rot@ τ (%) ↑
CD i ↓
PA (%) ↑
PairF1 (%) ↑
TypeAcc (%) ↑
GSim (%) ↑
C-Free (%) ↑
SCD ↓
PosErr ↓
Step ↓
RT ↓
PR (%) ↑
Commercial Chatbot Systems
GPT-5.5
-
2DVis
98.40
54.50
18.70
46.70
182.5
30.30
32.70
43.40
15.50
16.00
120.6
224.7
1.41
148.0
62.00
MG+2DVis
–
–
21.20
50.10
189.5
33.30
–
–
–
20.00
135.6
246.0
1.14
130.9
53.74
Conflict+2DVis
100.00
46.30
17.50
42.70
227.9
28.20
30.50
33.40
14.60
20.70
169.1
291.6
2.18
218.9
70.96
GPT-5-mini
-
2DVis
98.10
13.90
18.20
44.90
209.5
27.20
28.60
11.90
13.00
27.30
124.6
230.3
1.28
22.0
50.22
Table 3: Agentic evaluation of LMMs on component-level CAD assembly generation. Entries marked ‘–’ denote metrics that are not applicable by design: ID-NR/ID-R are omitted when the input mate graph already specifies the component-instance inventory; PairF1/TypeAcc/GSim are omitted for pose-only mate-graph-conditioned settings that do not require predicted mate edges. PR is computed per setting as successful final outputs divided by total round attempts.
Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. Recent methods based on vision-language models (VLMs) largely rely on coarse 2D perception and struggle to perform accurate reasoning over complex 3D geometry. To address this limitation, we propose AssemLM, a spatial multimodal large language model for robotic assembly that integrates assembly manuals, point clouds, and textual instructions to predict task-critical 6D assembly poses with explicit geometric understanding. To bridge raw 3D perception and high-level linguistic reasoning, AssemLM employs a specialized point cloud encoder to extract fine-grained geometric and rotational features for accurate 3D spatial reasoning in assembly tasks. In addition, we introduce AssemBench, a large-scale benchmark for assembly-oriented spatial reasoning with over 900K multimodal samples and precise 6D pose annotations, extending evaluation from 2D grounding to full 3D geometric inference. Extensive experiments and real-robot evaluations demonstrate that AssemLM achieves state-of-the-art 6D pose reasoning performance and effectively supports fine-grained, multi-step assembly tasks in real-world settings. Code, models, and the AssemBench dataset will be made publicly available.
Zhi Jing, Jinbin Qiao, Ouyang Lu +5
Fudan University† · Institute of Artificial Intelligence (TeleAI), China Telecom† · Tianjin University +3
Recent vision-language models (VLMs) can generate executable CAD programs from images, but existing methods mainly target coarse, general-purpose 3D objects and rarely address the fine-grained geometry and millimeter-level tolerances required in industrial mechanical design. We introduce OmniMech, the first million-scale benchmark for evaluating VLMs on executable CAD generation from industrial manufacturing data. OmniMech contains more than 251,000 fully dimensioned and toleranced 2D orthographic drawings, paired with native CAD models, multi-view renderings, mesh, STEP and B-rep representations, and rich semantic annotations. The benchmark includes four tasks: (1) parametric CAD program synthesis from engineering drawings; (2) diagram-to-3D reasoning for geometrically and structurally consistent reconstruction; (3) annotation-grounded reasoning over dimensions, symbols, feature callouts, and manufacturing constraints; and (4) tool-augmented agentic reasoning using visualization, measurement, CAD execution, and verification tools. Experiments show that current VLMs and CAD-specialized models still struggle with executable program synthesis, fine-grained 3D reconstruction, and reliable enforcement of dimensions and tolerances. We will release the benchmark data, evaluation code, and tool interfaces to support future research.
Taiting Lu, Runze Liu, Ziwei Dong +18
Pennsylvania State University · Independent Researcher · Shanghai Jiao Tong University +2
We introduceWorkBenchMark, a LEGO Duplo-based robotic assembly benchmark motivated by the RoboCup Smart Manufacturing League. Robotic assembly couples low-level manipulation with task-level symbolic reasoning under physical constraints, a combination that current end-to-end learning methods do not yet solve reliably. The benchmark provides 400 tasks across four complexity tiers. We provide an open-vocabulary perception, Assembly-by-Disassembly baseline solution. Our planning-based pipeline outperforms a modern vision-language-action approach across all tiers. The benchmark, simulation environment, and baseline implementation will be released openly to support the broader robotic assembly community.
Wenbo Ma, Daniel Swoboda, Matteo Tschesche +1
Chair of Machine Learning and Reasoning (i6), RWTH Aachen University, Aachen, Germany · MASCOR Institute, FH Aachen University of Applied Sciences, Aachen, Germany