Diagram-to-graph topology extraction aims to extract a graph of entities and their connections from a structural diagram. This task remains challenging for current vision-language models because it requires both fine-grained perceptual grounding and topology-aware reasoning with global consistency. We present TopoBench-180, a human-verified benchmark for diagram-to-graph topology extraction, and TopoAgent, a structure-aware perception-to-reasoning framework for reliable topology extraction using large vision-language models. TopoBench-180 contains 180 structural diagrams spanning Web-style and Network-style categories, paired with canonical graph annotations. TopoAgent progressively extracts the target graph by combining grounded perception, global structural priors, canonical node inventory construction, node-centric local-to-global relation reasoning, and topological consistency enforcement. Experiments on TopoBench-180 show that TopoAgent outperforms strong vision-language model baselines and recent visual reasoning frameworks, especially on edge extraction. More broadly, this work fills an important gap in multimodal structured understanding by establishing a benchmark and framework for diagram-to-graph topology extraction. The benchmark and associated resources will be publicly released at https://huggingface.co/datasets/WayneGuo0011/TopoBench-180.
Figures & tables
Figure 1: Motivating comparison for diagram-to-graph topology extraction. General-purpose VLMs often miss fine-grained visual relations or produce topologically inconsistent graphs, while prior visual reasoning baselines only partially address these challenges. In contrast, TopoAgent combines grounded perception with topology-aware reasoning to improve both local relation extraction and global graph consistency.
Figure 2: Pipeline for constructing the TopoBench-180 benchmark. Candidate diagrams are collected from existing public datasets, public web sources, and curated internal sources. The initial pool of raw images is then filtered through a multi-stage quality gating pipeline consisting of content-based de-duplication, VLM-driven pruning, and annotator cross-audit. For each retained diagram, a draft graph is first generated under a unified topology schema and then manually verified and refined by annotators to produce the final canonical graph annotation.
Figure 3: Overview of the proposed TopoAgent framework for diagram-to-graph topology extraction. The pipeline begins with a perception and grounding stage that decomposes the input diagram into grounded regions, aligns them with OCR-recognized text, and constructs region-level structured summaries. These grounded representations are then passed to a local-to-global topology reasoning pipeline, which produces a structured topology JSON representing the extracted graph.
Web-style
Network-style
Model
Params
Node F1
Edge P
Edge R
Edge F1
Node F1
Edge P
Edge R
Edge F1
Closed-source VLMs
GPT-4o
–
93.3
37.8
30.0
32.8
85.2
57.8
47.1
53.5
GPT-4o-mini
–
90.5
34.0
31.6
31.6
72.0
46.6
34.6
38.2
Open-source VLMs
Qwen2.5-VL-Instruct
7B
83.8
36.1
41.1
37.8
63.9
37.1
26.3
29.6
Table 1: Main results on TopoBench-180. We report Node F1 and directed Edge Precision (P), Recall (R), and F1 on Web-style and Network-style diagrams. Model sizes are shown when available. Best results are highlighted in bold.
Web-style
Network-style
Variant
Node F1
Edge P
Edge R
Edge F1
Node F1
Edge P
Edge R
Edge F1
w/o Region Summary
87.2
43.9
41.5
40.8
79.8
61.1
45.5
50.4
w/o Figure Classification
95.3
41.0
49.7
45.6
84.7
57.0
51.8
52.6
w/o Node-centric Reasoning
95.7
43.2
37.2
39.4
86.6
59.7
44.6
49.3
w/o Consistency Enforcement
86.6
36.9
44.6
38.9
81.7
54.3
49.5
50.7
TopoAgent (Full)
95.6
43.9
49.9
45.8
87.3
63.0
56.8
58.1
Table 2: Ablation study of TopoAgent on TopoBench-180. We evaluate the contribution of each major component, including region-level perceptual summarization, global figure attribute inference, node-centric local-to-global reasoning, and final topological consistency enforcement. We report Node F1 and directed Edge Precision (P), Recall (R), and F1 on the Web-style and Network-style subsets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Metric
Min
Mean
Max
Web-style
Nodes
5
10.09
23
Edges
5
14.35
37
Density
0.055
0.174
0.450
Network-style
Nodes
4
11.56
37
Edges
5
23.09
79
Density
0.037
0.230
1.000
Appendix
Table 3: Summary statistics of structural complexity in TopoBench-180. We report the minimum, mean, and maximum node count, edge count, and graph density for the Web-style and Network-style subsets.
Dataset
Backbone
Setting
Node F1
Edge P
Edge R
Edge F1
Web-style
GPT-4o
Single-pass
93.3
37.8
30.0
32.8
GPT-4o
TopoAgent
95.6
43.9
49.9
45.8
Qwen2.5-VL-72B
Single-pass
95.5
47.5
42.3
43.8
Qwen2.5-VL-72B
TopoAgent
96.1
49.2
46.8
47.0
Qwen2.5-VL-7B
Single-pass
83.8
36.1
41.1
37.8
Qwen2.5-VL-7B
TopoAgent
89.3
39.5
44.0
39.6
Appendix
Table 4: Generalization across VLM backbones on the Web-style and Network-style subsets. We compare single-pass inference with TopoAgent using GPT-4o, Qwen2.5-VL-72B, and Qwen2.5-VL-7B. All precision, recall, and F1 scores are macro-averaged over diagrams.
Web-style
Method
Node F1
Edge P
Edge R
Edge F1
GPT-4o single-pass
93.3
37.8
30.0
32.8
GPT-4o single-pass + OCR/crops
95.6
43.1
36.9
40.5
TopoAgent
95.6
43.9
49.9
45.8
Appendix
Table 5: Controlled comparison with a GPT-4o single-pass baseline augmented with the same OCR results and region crops used by TopoAgent. All metrics are macro-averaged over diagrams.
Dataset
Method
Edge P
Edge R
Edge F1
Web-style
GPT-4o single-pass
37.8
30.0
32.8
GPT-4o w/ Oracle Nodes
48.1
53.8
49.5
TopoAgent
43.9
49.9
45.8
TopoAgent w/ Oracle Nodes
56.7
62.5
57.9
Network-style
GPT-4o single-pass
57.8
47.1
53.5
GPT-4o w/ Oracle Nodes
66.8
60.7
64.0
Appendix
Table 6: Oracle-node analysis on the Web-style and Network-style subsets. Gold node sets are provided to each method, and only edge extraction is evaluated. All precision, recall, and F1 scores are macro-averaged over diagrams.
Dataset
Total Errors
Direction
Hallucinated
Missed
Node-name Mismatch
Web-style
911
240 (26.3%)
301 (33.0%)
300 (32.9%)
70 (7.7%)
Network-style
1915
56 (2.9%)
304 (15.9%)
1072 (56.0%)
483 (25.2%)
Appendix
Table 7: Error-type breakdown of TopoAgent on the Web-style and Network-style subsets. Percentages denote the fraction of total errors within each subset.
Method
Avg. API Calls
Avg. Time
Avg. Prompt Tok.
Avg. Output Tok.
Avg. Total Tok.
GPT-4o single-pass
1.00
14.34s
1,677.7
1,037.8
2,715.4
TopoAgent
14.04
98.05s
28,350.0
3,029.2
31,379.4
Appendix
Table 8: Average inference cost on 50 randomly sampled benchmark diagrams. Prompt tokens include textual instructions, image-token accounting, and auxiliary contextual information; output tokens correspond to generated JSON results and intermediate responses.
Figure 7: Successful qualitative examples of TopoAgent. The combined figure shows one Web-style example on the left and one Network-style example on the right. In both cases, TopoAgent achieves perfect topology extraction, with Node F1 = 1.0 and Edge F1 = 1.0. These examples show that the framework performs particularly well when the diagram structure is relatively simple, the number of nodes is small, node names are unique and clearly identifiable, and the figure contains few irrelevant visual structures.
Figure 8: Challenging qualitative examples of TopoAgent. The combined figure shows one Web-style example on the left and one Network-style example on the right. TopoAgent achieves Edge F1 scores of 31.3 and 45.0, respectively. These examples illustrate typical failure factors, including many nodes, long or crossing arrows, ambiguous edge directionality, difficult-to-read text, and visually cluttered structure, all of which increase the difficulty of node grounding and directed edge extraction.