EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning

Jinchang Zhang; Yingda Tao; Jiakai Lin; Guoyu Lu

cs.CVSep 27, 2026

EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning

Authors: Jinchang Zhang, Yingda Tao, Jiakai Lin, Guoyu Lu

Organizations: Intelligent Vision and Sensing (IVS) Lab Indiana University Bloomington Bloomington, IN, USA

Abstract

Multimodal engineering benchmarks largely evaluate static understanding, such as recognizing components, interpreting diagrams, or answering technical questions. This leaves a missing middle between engineering perception and full design generation: whether a model can use an understood system state to reason about relations, constraints, and the consequences of design changes. We introduce \textsc{EngIntervene}, a benchmark for this capability. It contains 3,229 questions across seven engineering domains and organizes evaluation into four levels: state grounding (T1), relational and mechanistic reasoning (T2), constraint-aware diagnosis (T3), and intervention reasoning (T4), which asks whether a proposed modification achieves its target while preserving required constraints. The tasks instantiate a unified engineering state representation spanning objects, relations, constraints, and design objectives, and T2--T4 are scored against structured reference answers with atomic criteria. Across open- and closed-weight multimodal models, stronger grounding does not reliably translate into better diagnosis or intervention, and the best open-weight model trails the best closed model by 14.7 percentage points on the T2--T4 average. Removing or shuffling visual evidence consistently degrades performance, while benchmark-specific supervised fine-tuning improves T1 but not T2--T4. T4 further exposes a large gap between satisfying individual revision criteria and producing a fully valid intervention. Engineering reasoning thus requires not only recovering the current state, but also reliably using it to reason about constraints and post-intervention consequences. Code and benchmark artifacts are available at https://github.com/changcv2021/EngIntervene

Figures & tables

Figure 1: Overview of EngIntervene. The center illustrates the four T1–T4 capability levels with representative real examples, while the outer ring presents the seven engineering domains.

Task	Engineering State Components	Primary Response Format	Primary Evaluation Target
T1: State Grounding	$\mathcal{V}$	Multiple choice	Objects, functions, states, and source-supported design intent
T2: Relational / Mechanistic Reasoning	$\mathcal{V},\mathcal{E}$	Natural-language explanation	Connections, paths, sequences, and operating mechanisms
T3: Constraint-aware Diagnosis	$\mathcal{V},\mathcal{E},\mathcal{C}$	Natural-language diagnosis Revision	Problem location, violated conditions, and engineering consequences
T4: Intervention Reasoning	$\mathcal{V},\mathcal{E},\mathcal{C},\mathcal{O}$	selection, verification, local proposal, or prioritization withjustification	Target effects, protected constraints, and unacceptable side effects

Table 1: The four EngIntervene tasks and their core reasoning objectives.

T4 Type	# Items	% of T4	# Domains
Revision Selection	61	25.4%	5
Revision Verification and Regression Analysis	86	35.8%	6
Constrained Local Revision Proposal	68	28.3%	6
Revision Prioritization and Trade-off	25	10.4%	4

Table 2: T4 design-intervention reasoning types.

Audit Dimension	# Audited	# Passed	Pass Rate	95% CI
Source support	500	493	98.6%	97.1–99.3
Image–question consistency	500	489	97.8%	96.1–98.8
Reference-answer correctness	500	490	98.0%	96.4–98.9
Question unambiguity / answerability	500	481	96.2%	94.1–97.6
Rubric completeness (T2–T4)	300	285	95.0%	91.9–96.9
All applicable criteria satisfied	500	474	94.8%	92.5–96.4

Table 3: Independent quality audit of EngIntervene before revision.

Type	Model	T1 Acc.	T2 Atomic	T3 Atomic	T4 Atomic	T4 Strict	Avg. T2–T4	Avg. T1–T4
Closed	GPT-4o-mini	47.98	23.96	23.01	31.29	7.50	26.09	31.56
	GPT-5.4	72.68	54.40	52.33	62.73	24.17	56.48	60.53
	GPT-5.6 Luna	66.68	51.73	50.15	59.16	24.17	53.68	56.93
	Claude Opus 4.7	73.00	57.64	58.41	67.84	30.83	61.30	64.22
	Grok 4.3 (non-reasoning)	53.90	34.05	33.96	40.67	12.08	36.23	40.65
Open	Qwen3.5-4B	51.65	29.78	33.32	38.33	8.33	33.81	38.27

Table 4: Main results on EngIntervene. T4 Strict denotes the percentage of T4 examples satisfying all required conditions without triggering a critical error.

Model	Condition	T1	T2	T3	T4 Atomic	T4 Strict	Macro Avg.
Qwen3.5-27B	Full	60.03	40.40	40.93	52.12	17.92	48.37
	Text-only	39.37	29.70	34.59	46.87	15.00	37.63
	Shuffled-image	31.74	7.02	16.11	26.75	6.67	20.40
Qwen3.6-27B	Full	61.66	36.85	37.81	49.04	13.33	46.34
	Text-only	37.65	25.63	33.06	46.49	10.42	35.71
	Shuffled-image	36.25	8.70	17.62	27.04	7.50	22.40

Table 5: Visual-dependency on EngIntervene.

Model	Condition	T1	T2	T3	T4 Atomic	T4 Strict	Macro Avg.
Qwen3.5-4B	Base	44.92	29.18	32.20	43.22	9.38	37.38
Qwen3.5-4B	SFT	49.83	14.15	19.13	31.59	4.17	28.68
InternVL3.5-8B	Base	38.55	22.35	22.05	40.41	8.33	30.84
InternVL3.5-8B	SFT	43.69	13.05	16.73	34.17	5.21	26.91
Qwen2.5-VL-7B	Base	42.46	18.07	16.31	32.98	4.17	27.45
Qwen2.5-VL-7B	SFT	45.92	12.70	14.77	28.02	4.17	25.35

Table 6: SFT results on EngIntervene.

Domain	T1	T2	T3	Selection	Verification	Proposal	Prioritization	T4 Total	Overall
Mechanical Design, Manufacturing and Industrial Equipment	349	60	60	8	12	16	0	36	505
Robotics and Mechatronics	300	60	60	13	22	1	0	36	456
Automotive System Integration	300	60	60	30	0	0	0	30	450
Aerospace Integration and Reliability	300	60	60	5	12	11	6	34	454
Building Services and MEP Systems	300	60	60	0	23	11	6	40	460
Power and Energy Equipment	300	60	60	0	12	15	7	34	454

Table 7: Complete domain-by-task and T4-subtype distribution.

Task	Source-based / Normalized / Constrained Adaptation	Source-supported Constructed	Total	Constructed %
T1	1,982	167	2,149	7.8%
T2	386	34	420	8.1%
T3	374	46	420	11.0%
T4	108	132	240	55.0%
Total	2,850	379	3,229	11.7%

Table 8: Construction and adaptation statistics. The Source-based column includes normalization, translation, cropping, and constrained adaptation.

Source Category	Fine-grained Labels	# Source Records	% Records	# Items	% Items
Official licensing and qualification examinations with answer keys or mark schemes	21	213	59.7%	1,545	47.8%
Trade, technician and professional certification question banks	9	61	17.1%	444	13.8%
Certification study and exam-preparation guides	3	10	2.8%	93	2.9%
University and college examinations, problem sets and textbook exercises with official solutions	10	58	16.2%	338	10.5%
Official training manuals, trade manuals and skills-competition materials	8	10	2.8%	400	12.4%
Regulatory advisory circulars and manufacturer technical documents	4	4	1.1%	160	5.0%

Table 9: Source type composition. Source percentages use 357 source documents as the denominator, while item percentages use 3,229 questions. Displayed percentages may not sum exactly because of rounding.

Language	# Source Records	# Items from Sources
de	19	55
en	216	2,051
hi–en (mixed)	43	301
ja	48	562
zh	15	172
zh-Hant	12	66

Table 10: Source language statistics.

Metric	Value
Distinct images used by items	2,275
Total image references across items	3,936
Single-image questions	2,746
Two-image questions	287
$\geq 3$ -image questions	196
Multi-image questions	483

Table 11: Visual asset statistics.

Task	# Items	Question Length Median	Q1–Q3	Mean	1 Image	2 Images	$\geq 3$ Images	MC Options Median
T1	2,149	35.0	22.0–54.0	47.7	1,799	186	164	4
T2	420	47.5	34.0–86.8	84.0	371	35	14	N/A
T3	420	45.0	35.0–71.0	90.6	364	38	18	N/A
T4	240	114.0	63.8–136.0	134.9	212	28	0	N/A

Table 12: Input statistics by task.

Metric	Value
Source records	357
Connected source groups	227
Median items per source record	5.0
Mean items per source record	9.0
Max items from one source record	249
Median items per source group	4.0

Table 13: Source concentration and independence.

Audit Dimension	Passed / Total	Pass Rate	95% CI
Target problem is clearly defined	238 / 240	99.2%	97.0–99.8
Reference intervention is technically valid and supported	235 / 240	97.9%	95.2–99.1
Intervention mechanism is technically valid	232 / 240	96.7%	93.6–98.3
Protected constraints are complete	229 / 240	95.4%	92.0–97.4
Regression conditions are sufficiently covered	225 / 240	93.8%	89.9–96.2
Editable scope / acceptance conditions are well-defined	233 / 240	97.1%	94.1–98.6

Table 14: Dedicated audit of T4 design interventions.

Domain	T3–T4	Function / Performance	Safety / Reliability	Spatial / Clearance	Interface / Compatibility	Manufacturability / Assembly	Maintainability / Serviceability	Standards / Regulation	Cost / Resource	Labels / Item
Mechanical Design, Manufacturing and Industrial Equipment	96	82	28	46	24	58	31	8	14	3.0
Robotics and Mechatronics	96	78	32	41	29	22	24	6	9	2.5
Automotive System Integration	90	74	48	44	43	27	39	12	8	3.3
Aerospace Integration and Reliability	94	81	72	38	34	41	36	47	7	3.8
Building Services and MEP Systems	100	63	79	52	22	26	44	39	6	3.3
Power and Energy Equipment	94	76	84	27	37	19	28	24	12	3.3

Table 15: Domain-by-shared-engineering-constraint coverage for T3–T4. Each cell reports the number of questions in the domain involving the corresponding constraint category. A question may belong to multiple categories, so row sums exceed the number of T3–T4 questions in the domain. The final column reports the average number of constraint categories per question.

Visual Representation	# Items	% Items
CAD / Rendered View	183	5.7%
Engineering Drawing	783	24.2%
Assembly Drawing	78	2.4%
Mechanism / System Schematic	261	8.1%
Piping / Wiring Diagram	497	15.4%
BIM / MEP Layout	100	3.1%

Table 16: Visual representation distribution. Percentages use all 3,229 questions as the denominator. Labels are multi-label, so the sum of category counts (3,542) exceeds the number of labeled questions (3,115), corresponding to an average of 1.14 labels per question.

Task	# Items	# Labeled	CAD	Drawing	Assembly	Schematic	Piping / Wiring	BIM / MEP	Photo	Before / After + Mixed
T1	2,149	2,081	103	562	51	136	200	66	713	442
T2	420	411	24	84	15	59	156	20	80	86
T3	420	394	33	73	3	44	105	7	128	66
T4	240	229	23	64	9	22	36	7	89	36
Total	3,229	3,115	183	783	78	261	497	100	1,010	630

Table 17: Visual representation by task. Labels are multi-label and category counts may overlap. Before/after comparisons and mixed representations are combined in the final column.

Domain	# Items	# Labeled	CAD	Drawing	Assembly	Schematic	Piping / Wiring	BIM / MEP	Photo	Before / After + Mixed
Mechanical Design, Manufacturing and Industrial Equipment	505	488	31	262	32	45	25	0	117	67
Robotics and Mechatronics	456	436	89	37	6	86	52	2	133	123
Automotive System Integration	450	446	6	154	12	53	143	0	28	105
Aerospace Integration and Reliability	454	442	4	220	23	31	76	0	30	127
Building Services and MEP Systems	460	446	27	38	2	10	70	97	214	51
Power and Energy Equipment	454	408	1	6	0	20	61	1	279	59

Table 18: Visual representation by domain. Labels are multi-label and category counts may overlap. Before/after comparisons and mixed representations are combined in the final column.

Model	Goal	Mechanism	Protection	Active Reg.	Atomic	Strict
Qwen3.5-4B	42.65	24.54	66.56	30.43	38.33	8.33
Qwen3.5-9B	43.95	31.19	63.52	29.35	40.60	11.25
Qwen3.5-27B	54.32	46.79	70.34	31.52	52.12	17.92
Qwen3.6-27B	51.55	40.83	69.08	34.78	49.04	13.33
Qwen3.8-27B	55.17	52.52	67.92	51.09	53.98	18.33
Qwen3-VL-2B-Thinking	33.23	19.95	55.82	7.61	29.63	6.25

Table 19: Role-wise T4 performance (%). Role columns report item-macro satisfaction scores over their respective eligible item sets. Atomic and Strict are the T4 metrics over all 240 items.

Model	Goal-sat.	Mechanism fail	Protection fail	Active Reg. fail
Qwen3.5-4B	76	43/71 (60.56%)	11/41 (26.83%)	19/25 (76.00%)
Qwen3.5-9B	68	27/62 (43.55%)	13/41 (31.71%)	18/22 (81.82%)
Qwen3.5-27B	91	29/84 (34.52%)	19/55 (34.55%)	19/27 (70.37%)
Qwen3.6-27B	87	40/81 (49.38%)	15/44 (34.09%)	12/23 (52.17%)
Qwen3.8-27B	90	32/85 (37.65%)	19/51 (37.25%)	13/27 (48.15%)
Qwen3-VL-2B-Thinking	57	31/53 (58.49%)	9/29 (31.03%)	15/16 (93.75%)

Table 20: Remaining failures after all available Goal criteria are satisfied. Each failure rate uses goal-satisfied responses containing the corresponding independently observable role as its denominator.

Model	Selection	Verification	Proposal	Prioritization
Qwen3.5-4B	24.95 / 6.56	46.07 / 10.38	39.00 / 7.35	28.00 / 0.00
Qwen3.5-9B	25.55 / 3.28	44.76 / 11.32	46.91 / 17.65	50.00 / 20.00
Qwen3.5-27B	43.31 / 16.39	52.65 / 17.92	58.91 / 19.12	56.00 / 20.00
Qwen3.6-27B	39.02 / 13.11	53.44 / 14.15	52.27 / 13.24	34.00 / 0.00
Qwen3.8-27B	35.93 / 9.84	59.95 / 19.81	60.71 / 23.53	56.00 / 20.00
Qwen3-VL-2B-Thinking	18.01 / 4.92	33.82 / 5.66	32.60 / 7.35	42.00 / 20.00

Table 21: Performance by T4 intervention subtype (Atomic / Strict, %). Selection, Verification, Proposal, and Prioritization.

Domain	Qwen3.5-27B	Qwen3.8-27B	GPT-5.4	Luna	Claude
Mechanical Design	8.33	11.11	22.22	19.44	22.22
Robotics & Mechatronics	33.33	36.11	47.22	47.22	52.78
Automotive	10.00	0.00	40.00	43.33	40.00
Aerospace	8.82	11.76	8.82	8.82	20.59
Building Services & MEP	30.00	25.00	15.00	20.00	20.00
Power & Energy	23.53	35.29	26.47	20.59	29.41

Table 22: T4 Strict Success by engineering domain (%). The table is descriptive and should not be interpreted as a ranking of intrinsic domain difficulty.

Domain	T4	Mechanism	Protection	Active Regression
Mechanical Design	36	30	26	12
Robotics & Mechatronics	36	28	8	5
Automotive	30	22	10	10
Aerospace	34	34	34	0
Building Services & MEP	40	40	40	0
Power & Energy	34	34	11	19

Table 23: Number of T4 items with independently observable role criteria in each engineering domain.

Model	N	Goal	Mechanism	Protection	Atomic	Strict
Qwen3.5-4B	134	28.36	23.88	67.79	34.01	6.72
Qwen3.5-27B	134	44.40	49.25	71.52	50.22	14.93
Qwen3.8-27B	134	46.64	58.96	68.66	53.42	12.69
GPT-5.4	134	51.12	61.94	75.87	58.04	13.43
Claude Opus 4.7	134	58.96	67.91	75.06	63.54	20.90

Table 24: Common-scope sensitivity analysis on 134 T4 items. All reported models are evaluated on the same restricted item subset.

Domain	T1	T2	T3	Selection	Verification	Proposal	Prioritization	T4 Total	Overall
Mechanical Design, Manufacturing and Industrial Equipment	349	60	60	8	12	16	0	36	505
Robotics and Mechatronics	300	60	60	13	22	1	0	36	456
Automotive System Integration	300	60	60	30	0	0	0	30	450
Aerospace Integration and Reliability	300	60	60	5	12	11	6	34	454
Building Services and MEP Systems	300	60	60	0	23	11	6	40	460
Power and Energy Equipment	300	60	60	0	12	15	7	34	454

Table 25: Complete domain-by-task and T4-subtype distribution.

Task	Source-based / Normalized / Constrained Adaptation	Source-supported Constructed	Total	Constructed %
T1	1,982	167	2,149	7.8%
T2	386	34	420	8.1%
T3	374	46	420	11.0%
T4	108	132	240	55.0%
Total	2,850	379	3,229	11.7%

Table 26: Construction and adaptation statistics. The Source-based column includes normalization, translation, cropping, and constrained adaptation.

Source Category	Fine-grained Labels	# Source Records	% Records	# Items	% Items
Official licensing and qualification examinations with answer keys or mark schemes	21	213	59.7%	1,545	47.8%
Trade, technician and professional certification question banks	9	61	17.1%	444	13.8%
Certification study and exam-preparation guides	3	10	2.8%	93	2.9%
University and college examinations, problem sets and textbook exercises with official solutions	10	58	16.2%	338	10.5%
Official training manuals, trade manuals and skills-competition materials	8	10	2.8%	400	12.4%
Regulatory advisory circulars and manufacturer technical documents	4	4	1.1%	160	5.0%

Table 27: Source type composition. Source percentages use 357 source documents as the denominator, while item percentages use 3,229 questions. Displayed percentages may not sum exactly because of rounding.

Language	# Source Records	# Items from Sources
de	19	55
en	216	2,051
hi–en (mixed)	43	301
ja	48	562
zh	15	172
zh-Hant	12	66

Table 28: Source language statistics.

Metric	Value
Distinct images used by items	2,275
Total image references across items	3,936
Single-image questions	2,746
Two-image questions	287
$\geq 3$ -image questions	196
Multi-image questions	483

Table 29: Visual asset statistics.

Task	# Items	Question Length Median	Q1–Q3	Mean	1 Image	2 Images	$\geq 3$ Images	MC Options Median
T1	2,149	35.0	22.0–54.0	47.7	1,799	186	164	4
T2	420	47.5	34.0–86.8	84.0	371	35	14	N/A
T3	420	45.0	35.0–71.0	90.6	364	38	18	N/A
T4	240	114.0	63.8–136.0	134.9	212	28	0	N/A

Table 30: Input statistics by task.

Metric	Value
Source records	357
Connected source groups	227
Median items per source record	5.0
Mean items per source record	9.0
Max items from one source record	249
Median items per source group	4.0

Table 31: Source concentration and independence.

Audit Dimension	Passed / Total	Pass Rate	95% CI
Target problem is clearly defined	238 / 240	99.2%	97.0–99.8
Reference intervention is technically valid and supported	235 / 240	97.9%	95.2–99.1
Intervention mechanism is technically valid	232 / 240	96.7%	93.6–98.3
Protected constraints are complete	229 / 240	95.4%	92.0–97.4
Regression conditions are sufficiently covered	225 / 240	93.8%	89.9–96.2
Editable scope / acceptance conditions are well-defined	233 / 240	97.1%	94.1–98.6

Table 32: Dedicated audit of T4 design interventions.

Domain	T3–T4	Function / Performance	Safety / Reliability	Spatial / Clearance	Interface / Compatibility	Manufacturability / Assembly	Maintainability / Serviceability	Standards / Regulation	Cost / Resource	Labels / Item
Mechanical Design, Manufacturing and Industrial Equipment	96	82	28	46	24	58	31	8	14	3.0
Robotics and Mechatronics	96	78	32	41	29	22	24	6	9	2.5
Automotive System Integration	90	74	48	44	43	27	39	12	8	3.3
Aerospace Integration and Reliability	94	81	72	38	34	41	36	47	7	3.8
Building Services and MEP Systems	100	63	79	52	22	26	44	39	6	3.3
Power and Energy Equipment	94	76	84	27	37	19	28	24	12	3.3

Table 33: Domain-by-shared-engineering-constraint coverage for T3–T4. Each cell reports the number of questions in the domain involving the corresponding constraint category. A question may belong to multiple categories, so row sums exceed the number of T3–T4 questions in the domain. The final column reports the average number of constraint categories per question.

Visual Representation	# Items	% Items
CAD / Rendered View	183	5.7%
Engineering Drawing	783	24.2%
Assembly Drawing	78	2.4%
Mechanism / System Schematic	261	8.1%
Piping / Wiring Diagram	497	15.4%
BIM / MEP Layout	100	3.1%

Table 34: Visual representation distribution. Percentages use all 3,229 questions as the denominator. Labels are multi-label, so the sum of category counts (3,542) exceeds the number of labeled questions (3,115), corresponding to an average of 1.14 labels per question.

Task	# Items	# Labeled	CAD	Drawing	Assembly	Schematic	Piping / Wiring	BIM / MEP	Photo	Before / After + Mixed
T1	2,149	2,081	103	562	51	136	200	66	713	442
T2	420	411	24	84	15	59	156	20	80	86
T3	420	394	33	73	3	44	105	7	128	66
T4	240	229	23	64	9	22	36	7	89	36
Total	3,229	3,115	183	783	78	261	497	100	1,010	630

Table 35: Visual representation by task. Labels are multi-label and category counts may overlap. Before/after comparisons and mixed representations are combined in the final column.

Domain	# Items	# Labeled	CAD	Drawing	Assembly	Schematic	Piping / Wiring	BIM / MEP	Photo	Before / After + Mixed
Mechanical Design, Manufacturing and Industrial Equipment	505	488	31	262	32	45	25	0	117	67
Robotics and Mechatronics	456	436	89	37	6	86	52	2	133	123
Automotive System Integration	450	446	6	154	12	53	143	0	28	105
Aerospace Integration and Reliability	454	442	4	220	23	31	76	0	30	127
Building Services and MEP Systems	460	446	27	38	2	10	70	97	214	51
Power and Energy Equipment	454	408	1	6	0	20	61	1	279	59

Table 36: Visual representation by domain. Labels are multi-label and category counts may overlap. Before/after comparisons and mixed representations are combined in the final column.

Explore similar work

Jun 9, 2026cs.AI

Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation

Vision-Language Models (VLMs) demonstrate strong performance on general multimodal reasoning benchmarks, yet their ability to perform engineering reasoning remains largely unexplored. Unlike general visual question answering, engineering problem solving requires interpreting technical diagrams, selecting governing physical principles, and maintaining physically consistent multi-step reasoning. These capabilities are increasingly important for AI systems used in engineering education, scientific assistance, and technical decision-making, where reasoning failures may produce physically invalid yet superficially plausible solutions. Existing benchmarks primarily evaluate final answers and provide limited assessment of intermediate reasoning processes. We introduce EngVQA, a multimodal benchmark for evaluating engineering reasoning across 5 engineering subjects containing 696 problems. We introduce an 8-stage automatic evaluation framework for assessing VLM-generated solutions. The framework independently evaluates each stage of the solution, enabling fine-grained analysis of reasoning failures. We benchmark multiple state-of-the-art open and closed source VLMs on our evaluation framework and demonstrate substantial limitations in current engineering reasoning capabilities. Human evaluation shows strong agreement with our automated framework, achieving a Pearson correlation of 0.975 and a mean absolute error of 0.67 on a 10-point grading scale. Our results highlight the importance of process-oriented evaluation for reliable assessment of multimodal engineering reasoning systems.

Syed Wasiq, Syed Mohamad Tawseeq, Yashwant Pravinrao Bangde +1

Indian Institute of Technology Kharagpur

Multimodal Reasoning Benchmarks Multimodal Reasoning Engineering Vision-Language Foundation Models Reasoning Skills

Ask PDF

Aug 10, 2026cs.AI

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion. We introduce MMArch, a benchmark for architecture and civil engineering spanning ten subdomains and built entirely from figures in peer-reviewed papers. Its

1{,}212

short-answer items are produced by a decoupled planner--writer pipeline and validated through automated screening, a blind adversarial audit, and expert review, so that answering requires perceiving the relevant evidence, identifying the governing principle, and applying it, not exploiting textual or single-figure shortcuts. Evaluating

18

open-weight and proprietary MLLMs against a domain-expert panel, we find a wide gap: the strongest open-source model attains about

30\%

and the best proprietary system

52\%

, while human experts reach

95\%

, more than forty points ahead. Our error analysis shows that failures concentrate in applying principles and combining evidence across figures rather than in locating it, pointing to substantial headroom for future research. Code and data are available at https://dcx-swjtu.github.io/MMArch/.

Chenxu Du, Kang An, Tengyue Wang +8

Southwest Jiaotong University · Shanghai Jiao Tong University · South China University of Technology +7

Multimodal Large Language Models Multimodal Reasoning Engineering

Ask PDF

Jul 29, 2026cs.CV

See2Think: Do Multimodal Models Really Use Intermediate Visual States?

Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states. Existing benchmarks are limited both by task collections with narrow coverage or partially text-solvable samples and by evaluations that emphasize final answers without diagnosing how intermediate visual states are generated, rendered, and used. We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench contains 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings. Evaluating representative proprietary and open-source multimodal models, we find that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis further shows that models usually select relevant visual operations, while faithful rendering remains the clearest bottleneck and high feedback uptake does not necessarily translate into accuracy gains. Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions.

Siyu Yan, Zhuoran Yan, Haiying Xu +10

1The Hong Kong University of Science and Technology · 3Shanghai AI Laboratory · 2Central South University +4

Large Multimodal Models Multimodal Model Visual Reasoning Think

Ask PDF