LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding

Ke Li; Tianyi Wang; Enning You; Chaoqun Wan; Zhen Huang; Shuxian Liang; Xian-Sheng Hua

cs.CVOct 14, 2026

LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding

Authors: Ke Li, Tianyi Wang, Enning You, Chaoqun Wan, Zhen Huang, Shuxian Liang, Xian-Sheng Hua

Organizations: Institute of AI for Engineering, Tongji University

Abstract

Real-world engineering drawings are often large-format and information-dense. Answering practical engineering questions therefore requires both fine-grained understanding of engineering information and complex reasoning over objects, relations, and evidence distributed across different regions or views. Existing benchmarks cover a range of engineering-drawing understanding tasks involving local regions, individual views, specific elements, or more constrained visual settings. However, systematic evaluation of understanding over realistic, large-format engineering drawings remains limited. To address this gap, we introduce LED-BENCH, a multidisciplinary benchmark designed to evaluate fine-grained understanding and complex reasoning over large-format engineering drawings, spanning the mechanical, architectural, and transportation domains. Extensive evaluation of 11 representative multimodal large language models, including 8 proprietary models and 3 open-source models, reveals substantial remaining difficulty: GPT-5.6 Sol achieves 52.65% average accuracy, performance varies across engineering domains, and all evaluated models perform worse on questions requiring broader evidence integration. These results highlight fine-grained understanding, evidence integration, and complex reasoning over large-format engineering drawings as persistent challenges for current multimodal models.

Figures & tables

Benchmark	Domain	Images	Tasks	Visual Source	Task Focus
MechVQA	Mech.	3,281	20,778	Mechanical drawings	Recognition / Reasoning / Judging
AECV-Bench	AEC	21	192	AEC floor plans	OCR / Counting / Spatial / Comparative
ArchPlanVQA	Arch.	11,477	26,266	Architectural floor plans	Semantic understanding / Engineering reasoning
DrawingVQA	AEC	33	92	Construction drawings	Perceptual / Contextual / Domain-expert
DesignQA	Mech.	230	1,451	Engineering drawings / CAD	Requirement understanding / Compliance
TriView2CAD	Civil / CAD	203k	$\approx$ 645K	Bridge-pier three-view CAD	Orthographic-projection reasoning

Table 1: Comparison with representative engineering-oriented multimodal benchmarks.

Figure 2: Expert-led, model-assisted benchmark construction. Domain experts specify questions and supporting evidence, language models expand the candidate set, and experts review each retained question and reference answer against its source drawing.

Domain	Drawings	Questions	Q/Draw.	Medium	Hard
Mechanical	45	174	3.87	119	55
Architecture	21	75	3.57	37	38
Transportation	41	128	3.12	55	73
Total	107	377	3.52	211	166

Table 2: Dataset composition across engineering domains. Q/Draw. denotes the average number of questions associated with each complete drawing. Medium and Hard are relative difficulty levels within LED-Bench.

Figure 3: Dataset composition and benchmark focus. (a) Question distribution and Medium / Hard split across the three domains. (b) Conceptual hierarchy of capabilities involved in large-format engineering drawing understanding and reasoning.

Closed-source MLLMs
Model	Mechanical $\uparrow$	Architecture $\uparrow$	Transportation $\uparrow$	Domain Avg $\uparrow$
GPT-5.6 Sol	58.81	48.89	50.26	52.65
GPT-5.6 Luna	50.38	47.11	44.27	47.26
Qwen3.8-Max	46.36	48.00	46.09	46.82
Gemini-3.5 Flash	57.47	35.11	31.77	41.45
GPT-5.5	41.00	17.78	39.84	32.87

Table 3: Performance across engineering domains. The best result in each column is shown in bold, and the second-best is underlined. Domain Avg uses unrounded domain scores; recomputing from the two-decimal entries can differ by 0.01 percentage points.

Closed-source MLLMs
Model	Medium $\uparrow$	Hard $\uparrow$	$\bm{\Delta}$
GPT-5.6 Sol	56.57	48.47	8.10
GPT-5.6 Luna	48.59	44.60	3.99
Qwen3.8-Max	49.97	42.09	7.88
Gemini-3.5 Flash	47.39	34.49	12.90
GPT-5.5	34.34	29.36	4.98

Table 4: Domain-avg accuracy (%) across the two difficulty levels. The best and second-best accuracies in each difficulty level are shown in bold and underlined, respectively.

\Delta

denotes the absolute decrease from Medium to Hard.

Figure 4: Failure stages and domain-level intervention performance. (a) Distribution of the four failure stages in the coded error sample used for the diagnostic analysis. (b) Four-model mean accuracy across Mechanical, Architecture, and Transportation for whole-sheet direct answering and three visual-access interventions.

Method	Intended support	Domain Avg $\uparrow$	$\Delta$
Whole-sheet direct answering	Baseline	36.38	–
Fixed overlapping crops	Local evidence access	49.34	12.96
Locate then reread	Target localization + rereading	41.43	5.05
Search then reread	Multi-region search + rereading	49.74	13.35

Table 5: Domain-avg accuracy (%) for the whole-sheet baseline and three visual-access interventions.

\Delta

denotes the absolute change relative to whole-sheet direct answering.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Exploratory rasterization study. Strict accuracy among adjudicated answers as a function of client-side PDF rasterization resolution for four representative response models. Results are reported across the tested rendering resolutions.

Subset	$N$	Agreement (%)	Cohen’s $\kappa$
Overall	90	95.6	0.902
Mechanical	30	93.3	0.815
Architecture	30	96.7	0.932
Transportation	30	96.7	0.930
Unanimous judge votes	71	97.2	0.934
2:1 split judge votes	19	89.5	0.791

Appendix

Table 6: Blinded human validation of the LLM-based majority-vote scoring protocol. Agreement is the fraction of responses for which the human binary label matches the three-judge majority decision.

Closed-source MLLMs
Model	R1 $\uparrow$	R2 $\uparrow$	R3 $\uparrow$	Mean $\pm$ SD
GPT-5.6 Sol	54.15	53.79	50.02	52.65 $\pm$ 2.29
GPT-5.6 Luna	50.02	47.72	44.02	47.26 $\pm$ 3.03
Qwen3.8-Max	46.88	47.09	46.48	46.82 $\pm$ 0.31
Gemini-3.5 Flash	39.45	44.02	40.88	41.45 $\pm$ 2.34
GPT-5.5	32.98	33.28	32.35	32.87 $\pm$ 0.47

Appendix

Table 7: Run-to-run stability under the primary domain-avg metric. R1–R3 report domain-avg accuracy (%) for each independent run; the final column reports the arithmetic mean and sample standard deviation across the three runs.

Closed-source MLLMs
Model	Mechanical $\uparrow$	Architecture $\uparrow$	Transportation $\uparrow$	Domain Avg $\uparrow$
GPT-5.6 Sol	58.81 $\pm$ 0.88	48.89 $\pm$ 4.07	50.26 $\pm$ 4.30	52.65 $\pm$ 2.29
GPT-5.6 Luna	50.38 $\pm$ 1.76	47.11 $\pm$ 4.29	44.27 $\pm$ 6.64	47.26 $\pm$ 3.03
Qwen3.8-Max	46.36 $\pm$ 2.02	48.00 $\pm$ 2.67	46.09 $\pm$ 1.56	46.82 $\pm$ 0.31
Gemini-3.5 Flash	57.47 $\pm$ 1.00	35.11 $\pm$ 4.29	31.77 $\pm$ 3.52	41.45 $\pm$ 2.34
GPT-5.5	41.00 $\pm$ 1.85	17.78 $\pm$ 4.07	39.84 $\pm$ 1.35	32.87 $\pm$ 0.47

Appendix

Table 8: Domain-level run-to-run stability. Values are mean

\pm

sample standard deviation of the three run-wise accuracies (%).

Stage	Operational interpretation
Local Evidence Understanding	Relevant local text, numbers, identifiers, specifications, symbols, or geometric cues are read or interpreted incorrectly.
Object Association	Evidence is plausibly read but associated with the wrong engineering object, instance, space, component, annotation target, or view-specific counterpart.
Relation Reconstruction	Relevant objects are identified, but connectivity, adjacency, path, direction, order, continuity, scope, datum, or cross-region/view correspondence is reconstructed incorrectly.
Integrated Engineering Reasoning	Upstream evidence is substantially available, but the final answer fails during evidence combination, multi-step derivation, arithmetic or unit handling, aggregation, constraint application, or answer completion.

Appendix

Table 9: Operational interpretation of the four conceptual failure stages used in the diagnostic analysis.

Method	Input / procedure	Calls
Whole-sheet direct answering	Complete drawing and question; answer directly from the full visual input	1
Fixed overlapping crops	Complete drawing plus four predefined overlapping crops in one request	1
Locate then reread	Locate at most two likely target regions, reread the corresponding crops, and answer using the full drawing and reread observations	3
Search then reread	Search for multiple relevant regions, reread up to two selected regions, and integrate the resulting observations	3

Appendix

Table 10: Visual-access intervention inputs and normal deployment calls per response. Failures and fallback paths may change the realized count.

Document Understanding

Document understanding enables machines to interpret text, layout, tables, and images to extract information and answer questions about documents.

See all 93 papers

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence
Aug 10, 2026Chenxu Du, Kang An, Tengyue Wang +8Multimodal Reasoning Visually Grounded Reasoning
Southwest Jiaotong University · Shanghai Jiao Tong University · South China University of Technology +7
MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding
May 29, 2026Qian Kou, Xiaofeng Shi, Yulin Li +4Visual Spatial Reasoning Multimodal Large Language Models
Beijing Academy of Artificial Intelligence (BAAI), China · Institute of Information Engineering, Chinese Academy of Sciences, China · Beijing University of Technology, China
Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams
Jun 2, 2026Abhishek Kumar, Isha Motiyani, Tilak Kasturi +3VLM Evaluation Visual Question Answering
Predii · Oak Ridge National Laboratory · Independent Researcher

Aug 10, 2026J/K move · Enter open · S save

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

Chenxu Du, Kang An, Tengyue Wang, Zhongyu Yang+7

Southwest Jiaotong University · Shanghai Jiao Tong University · South China University of Technology+7

Multimodal Reasoning Visually Grounded Reasoning Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) perform strongly on engineering imagery, yet existing benchmarks mostly test drawing recognition, information extraction, or compliance checking, leaving open whether models can combine distributed visual evidence with engineering principles to reach a conclusion. We introduce MMArch, a benchmark for architecture and civil engineering spanning ten subdomains and built entirely from figures in peer-reviewed papers. Its

1{,}212

short-answer items are produced by a decoupled planner--writer pipeline and validated through automated screening, a blind adversarial audit, and expert review, so that answering requires perceiving the relevant evidence, identifying the governing principle, and applying it, not exploiting textual or single-figure shortcuts. Evaluating

18

open-weight and proprietary MLLMs against a domain-expert panel, we find a wide gap: the strongest open-source model attains about

30\%

and the best proprietary system

52\%

, while human experts reach

95\%

, more than forty points ahead. Our error analysis shows that failures concentrate in applying principles and combining evidence across figures rather than in locating it, pointing to substantial headroom for future research. Code and data are available at https://dcx-swjtu.github.io/MMArch/.

Abstract

Document Understanding

Explore similar work

MMArch: Benchmarking Multimodal Reasoning Grounded in Architectural Evidence

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams