FlowExtract: Procedural Knowledge Extraction from Maintenance Flowcharts
Authors: Guillermo Gil de Avalle, Laura Maruster, Eric Sloot, Christos Emmanouilidis
Organizations: University of Groningen, Nettelbosje 2, Groningen, The Netherlands · Philips Consumer Lifestyle B.V., Oliemolenstraat 5, Drachten, The Netherlands
Maintenance procedures in manufacturing facilities are often documented as flowcharts in static PDFs or scanned images. They encode procedural knowledge essential for asset lifecycle management, yet inaccessible to modern operator support systems. Vision-language models, the dominant paradigm for image understanding, struggle to reconstruct connection topology from such diagrams. We present FlowExtract, a pipeline for extracting directed graphs from ISO 5807-standardized flowcharts. The system separates element detection from connectivity reconstruction, using YOLOv8 and EasyOCR for standard domain-aligned node detection and text extraction, combined with a novel edge detection method that analyzes arrowhead orientations and traces connecting lines backward to source nodes. Evaluated on industrial troubleshooting guides, FlowExtract achieves very high node detection and substantially outperforms vision-language model baselines on edge extraction, offering organizations a practical path toward queryable procedural knowledge representations. The implementation is available athttps://github.com/guille-gil/FlowExtract.
Figures & tables
Figure 1: Human-in-the-loop workflow for knowledge extraction from legacy maintenance documentation.
Class
Training
Validation
Test
Total
Arrowhead
357
48
93
498
Process
177
23
45
245
Document
119
11
38
168
Decision
118
15
29
162
Terminator
35
4
9
48
Connector
17
2
5
24
Table 1: Class Distribution Across Dataset Splits
Figure 2: FlowExtract pipeline overview (a) and edge detection methodology (b), showing arrowhead orientation analysis and line tracing for straight, L-shaped, and multi-branch connections.
Figure 3: Flowchart symbols taxonomy (based on ISO 5807:1985) and arrowheads.
Task
Precision
Recall
F1
Accuracy
Node Detection
98.4%
99.2%
98.8%
–
Node Classification
–
–
–
97.6%
Text Extraction
–
–
–
99.2%
Edge Detection
85.5%
54.6%
66.7%
–
Arrowhead Detection
–
–
–
73.1%
Edge Label Assignment
–
–
–
73.8%
Table 2: Detection Performance on Test Set
Method
Node F1
Edge F1
Qwen2-VL-7B
0.340
0.107
Pixtral-12B
0.295
0.015
FlowExtract
0.988
0.667
Table 3: Results Comparison with VLM Baselines from [ 23 ]
Figure 4: Example of extraction results from one of the maintenance diagrams. The original textual content within the nodes has been computationally redacted (opaque fills) to anonymize proprietary procedural data, while preserving the structural morphology.
Flowcharts are widely used in industrial requirements, but usually remain embedded as static images. Vision Language Models (VLMs) show promise in the conversion of these flowcharts into machine-readable models for RE activities, yet, when directly applied to flowchart conversion, they often fail on topology-critical visual details. To address this, we propose EdgeFlow that augments a VLM's original input with a deterministically extracted Canny edge map-acting as a structural prior-to improve flowchart-to-Mermaid conversion, without requiring annotated training data or domain-specific model fine-tuning. We evaluate EdgeFlow on IndusReqFlow, a dataset sourced from real-world requirements. Compared with off-the-shelf VLMs, EdgeFlow improves node-level F1 by 17.39 percentage points and edge-level F1 by 16.94 percentage points. At the path level, EdgeFlow improves path F1 by 11.06 percentage points, enabling better support for model-based testing. These results demonstrate that EdgeFlow provides a practical, training-free means to improve topology-preserving flowchart-to-Mermaid conversion for industrial RE. Cross-dataset evaluation results on a public synthetic benchmark show no significant improvement; this highlights the need for diverse benchmarks incorporating industrial data for the comprehensive evaluation of future VLM-based RE tools.
Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial domains. Recent vision-language models (VLMs) have become increasingly capable of recognizing diagram elements and reasoning about their content, while complete diagram topology extraction remains comparatively underexplored. In this paper, we study diagram-to-graph topology extraction: extracting all diagram entities and the complete relations among them. To enable large-scale supervised training and systematic evaluation of this task, we introduce Knossos, a benchmark of 19,200 diagrams across six diverse domains, with 245,179 nodes and 439,740 edges. Its symbolic generation process provides exact alignment between rendered diagrams and annotations of complete topology, relation types, and connector geometry. To address the modeling challenge of complete topology extraction, we also present Ariadne, a structured framework that decomposes the task into node inventory extraction and source-conditioned edge prediction. Extensive experiments show that training on Knossos substantially improves complete topology extraction in smaller open-source VLMs. Ariadne further improves over one-step extraction under matched supervision, demonstrating the additional benefit of structured decomposition. It achieves the highest average Edge F1 among the evaluated methods on Knossos, while both backbone variants also improve over their unadapted counterparts on the real-world external benchmark. Code and benchmark are available at https://github.com/bangwayne/knossos_Ariadne_Public.
Diagram-to-graph topology extraction aims to extract a graph of entities and their connections from a structural diagram. This task remains challenging for current vision-language models because it requires both fine-grained perceptual grounding and topology-aware reasoning with global consistency. We present TopoBench-180, a human-verified benchmark for diagram-to-graph topology extraction, and TopoAgent, a structure-aware perception-to-reasoning framework for reliable topology extraction using large vision-language models. TopoBench-180 contains 180 structural diagrams spanning Web-style and Network-style categories, paired with canonical graph annotations. TopoAgent progressively extracts the target graph by combining grounded perception, global structural priors, canonical node inventory construction, node-centric local-to-global relation reasoning, and topological consistency enforcement. Experiments on TopoBench-180 show that TopoAgent outperforms strong vision-language model baselines and recent visual reasoning frameworks, especially on edge extraction. More broadly, this work fills an important gap in multimodal structured understanding by establishing a benchmark and framework for diagram-to-graph topology extraction. The benchmark and associated resources will be publicly released at https://huggingface.co/datasets/WayneGuo0011/TopoBench-180.