OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies

Mingjia Wang; Taiting Lu; Ziwei Dong; Sisong Bei; Jingying Zeng; Runze Liu; Kaiyuan Lin; Hongxing Pan

cs.CVAug 23, 2026

OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies

Authors: Mingjia Wang, Taiting Lu, Ziwei Dong, Sisong Bei, Jingying Zeng, Runze Liu, Kaiyuan Lin, Hongxing Pan,

Organizations: Shanghai Jiao Tong University · Pennsylvania State University · Independent Researcher · Microsoft Research ·

Abstract

Recent vision-language models (VLMs) show strong capabilities in robotic perception and spatial reasoning, yet their ability to reason about complex mechanical assemblies remains underexplored. We introduce OmniCAD, a large-scale benchmark for assembly-aware 3D spatial reasoning across diverse industrial systems, including robotic mechanisms, automotive components, aerospace structures, and agricultural machinery. OmniCAD contains 25k mechanical assemblies, with an average of 12 parts per assembly and 21 types of mate relationships. Each assembly includes a human-verified ground-truth 3D model and renderings from 20 viewpoints. The benchmark evaluates three capabilities: (1) component-level 3D spatial reasoning, requiring prediction of part positions and orientations; (2) part-to-part relational reasoning, requiring identification of mating relationships and assembly constraints; and (3) tool-augmented agentic reasoning, where models iteratively select viewpoints, inspect visual evidence, and refine predictions. Experiments show that current VLMs struggle with industrial assembly reasoning, often producing inaccurate poses, invalid mating relationships, part interpenetration, and degraded performance as assembly complexity increases. We will open-source the benchmark, evaluation code, and tool interfaces to support research on accurate, physically valid, and scalable 3D assembly reasoning.

Figures & tables

Figure 1: VLMs exhibit limited capability in assembling 3D mechanical components into physically valid configurations. Frequent failures, including incorrect component poses, severe part interpenetration, missing or duplicated component instances, and inconsistent mating relationships, reveal substantial shortcomings in assembly-aware spatial and relational reasoning.

Figure 2: Statistical overview of the OmniCAD benchmark. From left to right, we show (a) a word cloud summarizing the application domains, (b) the distribution of component instances per assembly, (c) the distribution of unique component-library entries per assembly, and (d) the distribution of mating constraints per assembly.

Dataset	#Asm.	Avg. Comp.	#View	#Mate Type	Mate Graph	3D Object
						B-Rep	Mesh
3D CAD Assembly ( Lupinetti et al. 2019 )	137	93.31	✗	5	✓	✓	✗
AutoMate ( Jones et al. 2021 )	92.53K	N/R	✗	8	✓	✓	✗
JoinABLe ( Willis et al. 2022 )	8.25K	18.72	✗	7	✓	✓	✓
Assemble Them All ( Tian et al. 2022 )	12.97K	N/R	✗	✗	✗	✗	✓
GRAPE ( Gajek, Schiendorfer, and Reif 2023 )	35.66K $\dagger$	10.34 $\dagger$	✗	✗	✗	✗	✗

Table 1: Comparison of publicly available 3D assembly datasets. #Asm. denotes the total number of assemblies. Avg. Comp. denotes the average number of component instances per assembly. #View denotes the number of reference views. A green check mark indicates available, while a red cross indicates unavailable. N/R means not reported. For GRAPE,

\dagger

denotes statistics aggregated from its three independently reported component catalogs. For AssemblyBench,

\ddagger

denotes the selected reference view, while the number of candidate isometric views is not reported.

Figure 3: Overview of the OmniCAD benchmark with a representative example. From left to right, the figure shows the assembly library and its corresponding mate graph, the ground-truth 3D assembled model, and multi-view renders generated from the vertices of the surrounding view cube.

Figure 4: Agentic assembly framework for evaluating LMM tool use in mechanical assembly reconstruction.

Model	Size	Experiment	Component		Pose				Mate Graph			Physical / Shape			Efficiency / Success
Model	Size	Experiment	ID-NR (%) $\uparrow$	ID-R (%) $\uparrow$	Pos@ $\tau$ (%) $\uparrow$	Rot@ $\tau$ (%) $\uparrow$	CD i $\downarrow$	PA (%) $\uparrow$	PairF1 (%) $\uparrow$	TypeAcc (%) $\uparrow$	GSim (%) $\uparrow$	C-Free (%) $\uparrow$	SCD $\downarrow$	PosErr $\downarrow$	RT $\downarrow$	PR (%) $\uparrow$
Commercial Chatbot Systems
GPT-5.5	-	SameColor	100.00	60.00	15.30	48.40	245.6	32.00	36.40	44.10	12.10	3.10	144.0	368.9	111.0	37.00
		RefOBJ	100.00	71.40	0.00	84.70	240.8	14.70	46.10	59.70	23.30	11.80	152.4	316.3	99.5	21.00
		ColorView	92.90	57.10	15.70	43.30	349.3	25.00	35.70	44.00	8.40	10.00	232.2	481.9	171.3	35.00
		MG+Color	–	–	27.90	63.20	279.7	46.70	–	–	–	0.00	186.5	328.6	83.1	20.00

Table 2: One-shot evaluation of LMMs on component-level CAD assembly generation. Entries marked ‘–’ denote metrics that are not applicable by design: ID-NR/ID-R are omitted when the input mate graph already specifies the component-instance inventory; PairF1/TypeAcc/GSim are omitted for pose-only mate-graph-conditioned settings that do not require predicted mate edges. PR is computed per setting as successful final outputs divided by total attempts.

Model	Size	Agent Setting	Component		Pose				Mate Graph			Physical / Shape			Efficiency / Success
Model	Size	Agent Setting	ID-NR (%) $\uparrow$	ID-R (%) $\uparrow$	Pos@ $\tau$ (%) $\uparrow$	Rot@ $\tau$ (%) $\uparrow$	CD i $\downarrow$	PA (%) $\uparrow$	PairF1 (%) $\uparrow$	TypeAcc (%) $\uparrow$	GSim (%) $\uparrow$	C-Free (%) $\uparrow$	SCD $\downarrow$	PosErr $\downarrow$	Step $\downarrow$	RT $\downarrow$	PR (%) $\uparrow$
Commercial Chatbot Systems
GPT-5.5	-	2DVis	98.40	54.50	18.70	46.70	182.5	30.30	32.70	43.40	15.50	16.00	120.6	224.7	1.41	148.0	62.00
		MG+2DVis	–	–	21.20	50.10	189.5	33.30	–	–	–	20.00	135.6	246.0	1.14	130.9	53.74
		Conflict+2DVis	100.00	46.30	17.50	42.70	227.9	28.20	30.50	33.40	14.60	20.70	169.1	291.6	2.18	218.9	70.96
GPT-5-mini	-	2DVis	98.10	13.90	18.20	44.90	209.5	27.20	28.60	11.90	13.00	27.30	124.6	230.3	1.28	22.0	50.22

Table 3: Agentic evaluation of LMMs on component-level CAD assembly generation. Entries marked ‘–’ denote metrics that are not applicable by design: ID-NR/ID-R are omitted when the input mate graph already specifies the component-instance inventory; PairF1/TypeAcc/GSim are omitted for pose-only mate-graph-conditioned settings that do not require predicted mate edges. PR is computed per setting as successful final outputs divided by total round attempts.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly
Apr 10, 2026Zhi Jing, Jinbin Qiao, Ouyang Lu +53D Spatial Reasoning Robotic Assembly
Fudan University† · Institute of Artificial Intelligence (TeleAI), China Telecom† · Tianjin University +3
OmniMech: All-in-one Multimodal Mechanical Benchmark for 3D Reconstruction
Aug 6, 2026Taiting Lu, Runze Liu, Ziwei Dong +18Manufacturing 3D Reconstruction
Pennsylvania State University · Independent Researcher · Shanghai Jiao Tong University +2
WorkBenchMark: A LEGO-Based Assembly Benchmark with an Assembly-by-Disassembly Baseline for the Smart Manufacturing League
Jun 2, 2026Wenbo Ma, Daniel Swoboda, Matteo Tschesche +1Robotic Manipulation Robot Task Planning
Chair of Machine Learning and Reasoning (i6), RWTH Aachen University, Aachen, Germany · MASCOR Institute, FH Aachen University of Applied Sciences, Aachen, Germany

Apr 10, 2026J/K move · Enter open · S save

AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly

Zhi Jing, Jinbin Qiao, Ouyang Lu, Jicong Ao+4

Fudan University† · Institute of Artificial Intelligence (TeleAI), China Telecom† · Tianjin University+3

3D Spatial Reasoning Robotic Assembly 3D VLMs VLMs for Robotics Object Pose Estimation

Open paper AskPDF arXiv

Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. Recent methods based on vision-language models (VLMs) largely rely on coarse 2D perception and struggle to perform accurate reasoning over complex 3D geometry. To address this limitation, we propose AssemLM, a spatial multimodal large language model for robotic assembly that integrates assembly manuals, point clouds, and textual instructions to predict task-critical 6D assembly poses with explicit geometric understanding. To bridge raw 3D perception and high-level linguistic reasoning, AssemLM employs a specialized point cloud encoder to extract fine-grained geometric and rotational features for accurate 3D spatial reasoning in assembly tasks. In addition, we introduce AssemBench, a large-scale benchmark for assembly-oriented spatial reasoning with over 900K multimodal samples and precise 6D pose annotations, extending evaluation from 2D grounding to full 3D geometric inference. Extensive experiments and real-robot evaluations demonstrate that AssemLM achieves state-of-the-art 6D pose reasoning performance and effectively supports fine-grained, multi-step assembly tasks in real-world settings. Code, models, and the AssemBench dataset will be made publicly available.

Abstract

Spatial Reasoning Benchmarks

Explore similar work

AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly

OmniMech: All-in-one Multimodal Mechanical Benchmark for 3D Reconstruction

WorkBenchMark: A LEGO-Based Assembly Benchmark with an Assembly-by-Disassembly Baseline for the Smart Manufacturing League