MechReason: Benchmarking Multi-Image Multi-Hop Reasoning in Mechanical Engineering

Tengyue Wang; Kang An; Chenxu Du; Zhongyu Yang; Yuanchi Zhu; Xinqi Yang; Hebao Zhu; Ziliang Wang

cs.CVAug 21, 2026

MechReason: Benchmarking Multi-Image Multi-Hop Reasoning in Mechanical Engineering

Authors: Tengyue Wang, Kang An, Chenxu Du, Zhongyu Yang, Yuanchi Zhu, Xinqi Yang, Hebao Zhu, Ziliang Wang,

Organizations: South China University of Technology · Shanghai Jiao Tong University · Southwest Jiaotong University · ModelBest ·

Abstract

Despite significant progress in general visual question answering and cross-modal understanding, multimodal large language models still face a pronounced gap in evaluation for complex reasoning within the mechanical engineering domain. Existing benchmarks predominantly focus on rudimentary tasks such as drawing recognition, CAD interpretation, or single-chart querying, falling short of assessing whether models can integrate multiple images, textual conditions, physical principles, and engineering constraints to perform multi-step reasoning when confronted with authentic, intricate mechanical problems. To address this, we introduce MechReason, a benchmark derived from real mechanical engineering papers, comprising 12k question-answer pairs with explicit reasoning-chain annotations and 21k visual materials spanning nine evidence types, including statistical charts, parameter tables, engineering drawings, microscopic images, simulation images, system architectures, real mechanical scene photos, CAD model images and manufacturing flowcharts. MechReason covers eight task types across four reasoning dimensions: explanation, prediction, design, and diagnosis. We devise a four-stage construction pipeline: we first extract core engineering claims and decompose their supporting evidence into premises, reasoning processes, conclusions, and corroborative evidence; we then generate shortcut-preventing questions by masking posterior verification information; finally, we apply multimodal quality validation to ensure task quality and multi-hop nature. Extensive experimental results demonstrate that MechReason is highly challenging, with even the most advanced models achieving only 62.89% accuracy.

Figures & tables

Benchmark	Scale	Hetero. Visuals	Multi-Image	Multi-Hop	Paper Based	CoT Sup.	Features
TechMB	947 QA	–	–	$\triangle$	–	–	Manufacturability
DesignQA	1.45K QA	✓	$\triangle$	$\triangle$	–	✓	Engineering Compliance
TriView2CAD	203K samples	–	✓	$\triangle$	–	$\triangle$	CAD Projection QA
BenchCAD	17.9K parts	–	✓	$\triangle$	–	–	CAD Code
CADBench	18K samples	–	$\triangle$	–	–	–	CAD generation
MechVQA	21K QA	–	$\triangle$	$\triangle$	–	✓	Engineering drawing

Table 1: Comparison with representative mechanical and industrial multimodal benchmarks. “✓” denotes full support, “

\triangle

” denotes partial support, and “–” denotes not supported or not applicable.

Figure 1: A MechReason case requiring multi-image reasoning over process mechanism, rheology, force regulation, and particle-network evolution.

Panel of Figure 2 — Figure 2: Overview of MechReason. (a) Distribution of four reasoning dimensions and eight fine-grained subcategories. (b) Distribution of visual evidence types in MechReason.

Figure 3: Overview of the four-stage MechReason construction pipeline: core claim extraction, reasoning-chain decomposition, anti-shortcut question generation, and quality verification. The pipeline separates logical reasoning from posterior corroboration evidence to construct multi-image, multi-hop mechanical reasoning questions.

Model	Explanatory		Predictive		Design		Diagnostic		Overall
Model	Mech.	Dev.	Perf.	Proc.	Entity	Workflow	Limit	Failure	Overall
Closed Source Models
GPT-5.5	60.03	63.16	87.36	77.44	58.49	54.78	48.82	54.10	62.89
Claude Opus-4.8	52.41	48.77	85.60	71.76	58.32	51.30	46.83	62.30	58.07
Gemini-3.1-Pro	43.26	48.99	80.46	69.92	49.80	49.35	43.31	50.82	51.99
Gemini-3.5-Flash	44.52	47.37	80.84	72.93	50.00	44.16	40.94	44.26	51.64

Table 2: Main results on MechReason. Accuracy (%) is reported for each reasoning subcategory and overall performance.

Model	N	Grounding	Chain	Assumption	Hallucination	Compression
GPT-5.5	839	$21.78$	$37.12$	$10.43$	$2.87$	$27.80$
Gemini-3.1-Pro	1085	$25.74$	$44.65$	$5.35$	$1.01$	$23.25$
Kimi K2.6	1339	$33.21$	$42.79$	$8.85$	$6.15$	$19.00$
Qwen3.5-9B	1530	$44.86$	$37.23$	$6.98$	$5.93$	$5.00$

Table 3: Distribution of primary trace-aware error types among incorrect responses. Percentages are computed over the incorrect samples of each model.

Figure 4: Overall accuracy of Qwen3.5-9B before and after supervised fine-tuning with and without Chain-of-Thought (CoT) supervision.

Subcategory	N	$\Delta$
Mechanism Analysis	$713$	$2.95$
Deviation Attribution	$247$	$-0.81$
Performance Prediction	$261$	$8.43$
Process Extrapolation	$133$	$10.53$
Entity Design	$490$	$6.46$
Workflow Planning	$231$	$3.03$

Table 4: Subcategory-wise sample counts and accuracy changes after CoT supervision.

\Delta

denotes the percentage-point change over the original Qwen3.5-9B model.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 512 papers

May 29, 2026

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

Multimodal Large Language Models (MLLMs) have demonstrated significant achievements in general visual question answering (VQA) tasks. However, they remain brittle on mechanical engineering drawings, where high annotation density and weak domain knowledge, compounded by unreliable spatial relation reasoning under strict projection rules and geometric constraints, make decisive cues easy to miss and frequently lead to wrong answers. To bridge this gap, we introduce the first comprehensive mechanical drawing understanding dataset, MechVQA, created through a semi-automated construction and quality-control pipeline. MechVQA contains 3.3k high-density pictures with 21K question-answer pairs, spanning 10 different fine-grained tasks across three capability levels: Recognition, Reasoning, and Judging, providing a testbed to evaluate and improve MLLM understanding on real-world mechanical drawings. On top of MechVQA, we then develop the MechVL model through a multi-stage training paradigm, building a strong domain-specialized baseline. Extensive experimental results demonstrate that MechVL outperforms the strongest closed-source baseline by 7.57 percentage points on the MechVQA total score, significantly enhancing mechanical drawing understanding ability and providing a reusable foundation for deploying MLLMs in mechanical design and inspection scenarios.

Qian Kou, Xiaofeng Shi, Yulin Li +4

Beijing Academy of Artificial Intelligence (BAAI), China · Institute of Information Engineering, Chinese Academy of Sciences, China · Beijing University of Technology, China

Visual Spatial Reasoning Multimodal Large Language Models Multimodal QA

Ask PDF

Aug 10, 2026

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion. We introduce MMArch, a benchmark for architecture and civil engineering spanning ten subdomains and built entirely from figures in peer-reviewed papers. Its

1{,}212

short-answer items are produced by a decoupled planner--writer pipeline and validated through automated screening, a blind adversarial audit, and expert review, so that answering requires perceiving the relevant evidence, identifying the governing principle, and applying it, not exploiting textual or single-figure shortcuts. Evaluating

18

open-weight and proprietary MLLMs against a domain-expert panel, we find a wide gap: the strongest open-source model attains about

30\%

and the best proprietary system

52\%

, while human experts reach

95\%

, more than forty points ahead. Our error analysis shows that failures concentrate in applying principles and combining evidence across figures rather than in locating it, pointing to substantial headroom for future research. Code and data are available at https://dcx-swjtu.github.io/MMArch/.

Chenxu Du, Kang An, Tengyue Wang +8

Southwest Jiaotong University · Shanghai Jiao Tong University · South China University of Technology +7

Multimodal Reasoning Visually Grounded Reasoning Multimodal Model Evaluation

Ask PDF

Jun 9, 2026

Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation

Vision-Language Models (VLMs) demonstrate strong performance on general multimodal reasoning benchmarks, yet their ability to perform engineering reasoning remains largely unexplored. Unlike general visual question answering, engineering problem solving requires interpreting technical diagrams, selecting governing physical principles, and maintaining physically consistent multi-step reasoning. These capabilities are increasingly important for AI systems used in engineering education, scientific assistance, and technical decision-making, where reasoning failures may produce physically invalid yet superficially plausible solutions. Existing benchmarks primarily evaluate final answers and provide limited assessment of intermediate reasoning processes. We introduce EngVQA, a multimodal benchmark for evaluating engineering reasoning across 5 engineering subjects containing 696 problems. We introduce an 8-stage automatic evaluation framework for assessing VLM-generated solutions. The framework independently evaluates each stage of the solution, enabling fine-grained analysis of reasoning failures. We benchmark multiple state-of-the-art open and closed source VLMs on our evaluation framework and demonstrate substantial limitations in current engineering reasoning capabilities. Human evaluation shows strong agreement with our automated framework, achieving a Pearson correlation of 0.975 and a mean absolute error of 0.67 on a 10-point grading scale. Our results highlight the importance of process-oriented evaluation for reliable assessment of multimodal engineering reasoning systems.

Syed Wasiq, Syed Mohamad Tawseeq, Yashwant Pravinrao Bangde +1

Indian Institute of Technology Kharagpur

VLM Evaluation Visual Question Answering VLM Reasoning

Ask PDF

Abstract

Multimodal Reasoning

Explore similar work

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation