Learning to Ground Before Reading: Unified PCB Engineering Drawing Parsing with Compact Vision-Language Models

Jinghao Liu; Xingrun Liu; Gengchen Sun; Han Xiao; Xingyu Chen; Yuhui Deng

cs.CVAug 29, 2026

Learning to Ground Before Reading: Unified PCB Engineering Drawing Parsing with Compact Vision-Language Models

Authors: Jinghao Liu, Xingrun Liu, Gengchen Sun, Han Xiao, Xingyu Chen, Yuhui Deng

Organizations: Beijing Normal–Hong Kong Baptist University, Zhuhai 519087, China · Guangdong Provincial Key Laboratory of Interdisciplinary Research and Application for Data Science, Zhuhai 519087, China · Faculty of Science, Hong Kong Baptist University, Hong Kong SAR 999077, China · Hong Kong aiKnow Limited, Hong Kong SAR, China

Abstract

PCB engineering drawings mix sparse graphics, dense tables, and text whose meaning depends on page position. Localizing the regions and sending crops to specialized recognizers are determined as the methods for most parsers, so missed regions cannot be recovered downstream. We train a compact VLM to read the full page and get a sequence of region classes, normalized boxes, and text or HTML content. Bounding boxes are converted to coordinate tokens for supervision. Inference uses no detector or crop parser. The joint target is difficult to optimize because class and box tokens are sparse relative to the much longer content sequences. Our localization-first curriculum learns the class-box format before adding content targets with content-aware resampling. On the fixed validation split of the Engineering Drawing Dataset (ED dataset), Localization-First improves strict localization F1 by 0.0955 over joint training (paired image-bootstrap 95% interval: [0.0350, 0.1572]). G-Unified has the lowest NED, highest cell F1, and only nonzero exact-match score. It provides a detector-free baseline for full-page PCB drawing parsing.

Figures & tables

Figure 1: Three parsing modes considered in this study. (a) Grounding Only predicts region classes and locations without content. (b) Cascaded Grounding & Parsing first localizes and crops each region, then sends the crop to a separate VLM. (c) Unified Grounding & Parsing predicts classes, boxes, and content directly from the full page in one VLM call. Green highlights the unified mode used in the primary experiments.

Variant	Backbone	Records/epoch	Training intervention
2B unified	2B	241	Original joint target
4B baseline	4B	241	Original joint target
4B no-thinking	4B	241	Direct structured answer only
4B stable	4B	802	Content-heavy continued adaptation
4B Local-First	4B	241 / 615	Box-only warm-up, then full targets
4B hard-class	4B	961	Hard-label sampling and dirty-content removal

Table 1: Unified-training variants. Counts are records sampled per epoch, not distinct drawings. All rows use the same validation split of the ED dataset.

Figure 2: End-to-end Local-First parsing with full-page localization and structured content extraction.

Model	Pred.	Prec.	Rec.	$F_{1}$	mIoU	NED $\downarrow$	Cell $F_{1}$	EM
2B unified	231	.2121	.1801	.1948	.6982	—	—	—
4B baseline	144	.5694	.3015	.3942	.8083	.8825	.1464	.0000
4B no-thinking	191	.5550	.3897	.4579	.8293	.8629	.2230	.0000
4B stable	188	.5798	.4007	.4739	.8751	.8347	.1497	.0145
4B Local-First	165	.6485	.3934	.4897	.8907	.8143	.1516	.0299
4B hard-class	173	.6416	.4081	.4989	.8982	.8466	.1945	.0294

Table 2: Strict one-pass results on the same 26 validation drawings from the ED dataset (272 boxes; 78 content-bearing references). NED is lower-is-better; all other metrics are higher-is-better. Missing content receives NED 1 and zero task credit.

Figure 3: Unified-model training and held-out comparison. (a) Cold-start joint losses for 2B and 4B models. (b) Warm-start and continued-run losses. (c) The two Local-First phases. (d) Gradient norms and (e) learning-rate schedules. (f) Held-out metrics on the ED dataset, with sequence similarity shown as $1-\mathrm{NED}$ . Panels (a)–(e) show optimization traces; panel (f) reports held-out scores.

Paired contrast	$\Delta F_{1}$	95% interval	$\Pr(\Delta>0)$
Local-First $-$ baseline	.0955	$[.0350,.1572]$	.9986
Local-First $-$ stable	.0158	$[-.0598,.0862]$	.6346
Hard-class $-$ Local-First	.0092	$[-.0428,.0551]$	.6430

Table 3: Paired image bootstrap for strict micro-F1 on the ED dataset (

B=5{,}000

). The last column is the fraction of replicates with positive difference, not a parametric

p

-value.

Figure 4: Strict diagnostics on the ED dataset. (a) Localization precision, recall, F1, and matched IoU. (b) Full-denominator content metrics, with

1-\mathrm{NED}

used so every cell is higher-is-better. (c) Per-class F1. Paired bootstrap intervals are reported in Table 3 .

System	Pred.	Precision	Recall	$F_{1}$	mIoU
eDOCr2 reproduction [ 21 ]	214	.2383	.1875	.2099	.8280
G-Unified Local-First	165	.6485	.3934	.4897	.8907
Hybrid-VL reproduction [ 10 ]	268	.6754	.6654	.6704	.8215

Table 4: Strict localization on the same 26 validation drawings from the ED dataset (272 boxes). Every emitted record is counted; higher is better.

System	Content stage	Localized	Usable	NED $\downarrow$	Cell $F_{1}$	EM
eDOCr2	Tesseract + Qwen2-VL-7B	13/78	9/78	.9856	.0602	.0000
Hybrid-VL	Florence-2-base-ft	48/78	0/78	1.0000	.0000	.0000
G-Unified Local-First	page-level Qwen3.5-4B	27/78	23/78	.8143	.1516	.0299

Table 5: Strict end-to-end content extraction on the same 78 content references from 20 content-bearing drawings in the ED dataset. NED is lower-is-better; cell F1 and exact match (EM) are higher-is-better.

Decode	Format rate	Usable	Mean NED $\downarrow$	EM
Original saved output	.000	0/78	1.000	.000
Repair-only	1.000	78/78	.515	.000
Prefix-regenerate + repair	1.000	78/78	.553	.000

Table 6: Florence decoder diagnostics on 78 human-annotated crops from the ED dataset.

Figure 5: Qualitative localization comparison on three drawings from the ED dataset (V02, V09, and V10). Columns show eDOCr2, Hybrid-VL, and G-Unified; colors follow the shared six-class ontology.

Document Image Understanding

Document image understanding turns page images into recognized text, layout, structured data, and answers grounded in visual evidence.

See all 148 papers

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams
Jun 2, 2026Abhishek Kumar, Isha Motiyani, Tilak Kasturi +3VLM Evaluation Visual Question Answering
Predii · Oak Ridge National Laboratory · Independent Researcher
Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation
Oct 14, 2026Yuheng Chen, Yufan Chen, Zhuojun Cai +8Document Parsing Benchmarks Document Parsing
Karlsruhe Institute of Technology · University of Alberta · Hunan University
LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding
Oct 14, 2026Ke Li, Tianyi Wang, Enning You +4Document Understanding Document Image Understanding
Institute of AI for Engineering, Tongji University

Jun 2, 2026J/K move · Enter open · S save

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

Abhishek Kumar, Isha Motiyani, Tilak Kasturi, Ethan Seefried+2

Predii · Oak Ridge National Laboratory · Independent Researcher

VLM Evaluation Visual Question Answering

Open paper AskPDF arXiv

Engineering diagrams pose a distinct challenge for vision-language models: unlike natural images or general documents, they encode information through dense spatial layouts, domain-specific symbols, and cross-references between visual callouts and structured parts tables. Despite their centrality to service, repair, and design workflows, there is no public benchmark for measuring VLM capabilities in this domain; existing datasets primarily focus on flowcharts, scientific figures, or business documents. To address this gap, we introduce Enginuity, the first open dataset and benchmark for evaluating VLMs on complex engineering diagrams. We define two tasks over a corpus of U.S. military service and repair manuals: structured parts-table extraction (Task 1) and free-form visual diagram question answering (VQA)(Task 2) for benchmarking. We evaluate four frontier VLMs (GPT-5.2 Chat, Claude Opus 4.7, Gemma 4, Qwen3-VL-32B-Instruct) under zero-shot and chain-of-thought prompting. On Task 1, models reach Recall@all of 0.61-0.87 but Token F1pen of only 0.03-0.18, exposing a systematic gap between part identification and description fidelity. Task 2 reveals a consistent factual-reasoning gap across all models. A supporting analysis shows that token-overlap metrics under-report model capability on technical descriptions by 2-6x relative to semantic similarity, motivating LLM-as-judge calibration for domain-specific evaluation. We release the dataset, annotations, evaluation harness, and per-sample model outputs to support a reproducible study of VLM capability on engineering content.

Abstract

Document Image Understanding

Explore similar work

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation

LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding