Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models

Bangwei Guo; Xujiang Zhao; Shengyu Chen; Yanchi Liu; Wei Cheng; Xi Zhu; Guoning Zhang; Dimitris N. Metaxas

cs.CVOct 3, 2026

Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models

Authors: Bangwei Guo, Xujiang Zhao, Shengyu Chen, Yanchi Liu, Wei Cheng, Xi Zhu, Guoning Zhang, Dimitris N. Metaxas,

Organizations: Rutgers University · Meta · NEC Labs America

Abstract

Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial domains. Recent vision-language models (VLMs) have become increasingly capable of recognizing diagram elements and reasoning about their content, while complete diagram topology extraction remains comparatively underexplored. In this paper, we study diagram-to-graph topology extraction: extracting all diagram entities and the complete relations among them. To enable large-scale supervised training and systematic evaluation of this task, we introduce Knossos, a benchmark of 19,200 diagrams across six diverse domains, with 245,179 nodes and 439,740 edges. Its symbolic generation process provides exact alignment between rendered diagrams and annotations of complete topology, relation types, and connector geometry. To address the modeling challenge of complete topology extraction, we also present Ariadne, a structured framework that decomposes the task into node inventory extraction and source-conditioned edge prediction. Extensive experiments show that training on Knossos substantially improves complete topology extraction in smaller open-source VLMs. Ariadne further improves over one-step extraction under matched supervision, demonstrating the additional benefit of structured decomposition. It achieves the highest average Edge F1 among the evaluated methods on Knossos, while both backbone variants also improve over their unadapted counterparts on the real-world external benchmark. Code and benchmark are available at https://github.com/bangwayne/knossos_Ariadne_Public.

Figures & tables

Figure 1: Overview of the Knossos benchmark. (a) Representative diagrams from the six domains. (b) The generation pipeline samples generation controls, instantiates a symbolic topology, places nodes and routes edges, and renders the diagram together with aligned node, typed-edge, and geometry annotations. Structural validation and VLM-based screening produce the frozen training and test release. (c) Dataset statistics summarize difficulty, topology scale, edge directionality, and multi-link prevalence across 19,200 diagrams, 245,179 nodes, and 439,740 typed edges.

Figure 2: Overview of Ariadne. Ariadne decomposes topology extraction into node inventory prediction and source-group edge extraction. Edge prediction is restricted to active sources within each group while retaining all predicted nodes as candidate targets. The group-wise edge multisets are validated and merged into the final graph, with node bounding boxes and separately predicted connector traces providing auxiliary geometric grounding.

Figure 3: Node extraction results on the internal and external test sets. Panel (a) reports the unweighted mean Node F1 across the six Knossos domains. Panel (b) reports the unweighted mean across the Web-style and Network-style subsets of TopoBench-180. Colors distinguish open-source VLMs, closed-source VLMs, visual reasoning methods, and our Ariadne models. Full model names are listed in Table 1 .

Open-source VLMs
	Knossos							TopoBench-180
Method	Food Web	Network	Workflow	Natural Process	Circuit	Map Route	Avg.	Web-style	Network-style
Qwen3-VL-32B	28.8	48.7	76.4	65.8	36.7	71.9	54.7	54.7	52.3
Qwen3-VL-8B	27.5	47.0	59.9	49.5	31.6	61.1	46.1	44.1	49.8
GLM-4.6V-Flash	30.0	25.0	46.3	48.4	27.3	56.3	38.9	55.2	45.1
MiMo-VL-7B-RL	24.7	37.1	54.3	40.2	26.1	61.7	40.7	42.2	46.5

Table 1: Edge extraction results across six Knossos domains and two external TopoBench-180 domains, measured by Edge F1 scores (%).

Figure 4: Average Typed-Edge F1 scores across Natural Process, Circuit, and Map Route. Each bar is the arithmetic mean of the three domain-level Macro F1 scores reported in Appendix Table 5 .

Table 2: Ablations on staged topology extraction and external generalization. (a) compares one-step and staged extraction across model and training settings. (b) varies the upstream node inventory while keeping the trained Qwen3 edge predictor fixed on the public TopoBench subset.

Factor	Subset	$N$	GPT-5.6	Ariadne-Qwen
Difficulty	Simple	164	94.4	96.5
	Medium	571	84.9	94.7
	Difficult	465	76.8	89.0
Direction	Directed	132	86.7	89.0
	Bidirectional	37	86.7	88.6
	Mixed	31	86.2	91.0

Table 3: Edge F1 (%) across structural subsets.

N

denotes the number of images. To better isolate each structural factor while reducing variation from other domain-specific properties, difficulty is averaged across all six Knossos domains, directionality is evaluated within Map Route, and multi-link effects are evaluated across Circuit, Map Route, and Natural Process.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Canonical labels	Node instances	Avg./image	Min–max
Food Web	31	38,585	12.06	6–17
Network	40	44,315	13.85	8–17
Workflow	60	47,746	14.92	8–20
Natural Process	59	40,527	12.66	7–18
Circuit	60	45,142	14.11	7–23
Map Route	60	28,864	9.02	6–15

Appendix

Table 4: Canonical vocabulary and node-occurrence statistics in the frozen release. Domain identifies the six diagram domains; Canonical labels is the number of domain-local vocabulary entries represented in that domain; Node instances is the total number of node occurrences across its 3,200 diagrams; Avg./image is the mean number of nodes per diagram; and Min–max is the observed per-diagram node-count range. The total row aggregates all 19,200 diagrams, counting a shared label separately in each domain.

Figure 5: Difficulty-conditioned Natural Process grammar and three frozen instantiations. Left: the executable constraints used at each difficulty. Here $L_{k}$ is the node set at process stage $k$ , $K$ is the number of stages, and the two coverage constraints require every node in adjacent stages to participate in at least one forward relation. The listed ranges are grammar-level cardinality ranges, while “shown instance” gives the realized cardinalities in that row. Middle: the exact symbolic nodes and edge multiset of the selected example, with geometry regularized only for legibility. Right: the corresponding unmodified rendered image and its relation-type legend. From top to bottom, the examples contain 9/9, 11/14, and 14/18 nodes/relations. All three are drawn directly from the frozen test set; no topology, annotation, or rendered connector was edited for the figure.

Figure 6: Representative visual-appearance pools in Knossos. Each row is a domain, each column is an appearance family, and each cell shows the two candidate variants available for one representative vocabulary entry. Workflow appearances are rendered programmatically rather than generated. The figure shows the candidate pool; the frozen renderer uses the domain-specific subset and sampling policy described below.

Figure 7: Geometry annotations on a released Food Web sample. Teal dashed boxes denote finalized node bounding boxes. Gold polylines denote rendered connector paths ( visible_path ), and the five circular markers on each path are trace points sampled at equal arc-length intervals in source-to-target order. All coordinates are stored in pixels on the final image canvas.

Open-source VLMs
Method	Natural Process	Circuit	Map Route
Qwen3-VL-32B	49.5	30.7	69.3
Qwen3-VL-8B	41.8	24.0	55.8
GLM-4.6V-Flash	34.9	19.8	48.3
MiMo-VL-7B-RL	34.5	21.3	57.4
Keye-VL-1.5-8B	31.6	11.5	28.2

Appendix

Table 5: Typed-Edge F1 (%) on the three Knossos domains with explicit relation semantics.

Method	Box mIoU $\uparrow$	[email protected] $\uparrow$	Connector L2 $\downarrow$
GPT-5.6	81.81	93.73	11.34
Ariadne-GLM	86.42	96.96	6.14
Ariadne-Qwen3	96.59	97.55	4.13

Appendix

Table 6: Geometric grounding on the fixed 120-image Knossos subset. Box mIoU and [email protected] are percentages; connector L2 is measured in original-image pixels.

	Box mIoU $\uparrow$			Connector L2 $\downarrow$
Domain	GPT-5.6	Ariadne-GLM	Ariadne-Qwen3	GPT-5.6	Ariadne-GLM	Ariadne-Qwen3
Food Web	88.71	86.20	96.06	17.54	6.80	5.74
Network	91.78	92.91	98.55	7.51	4.07	4.14
Workflow	95.15	84.62	97.75	6.80	4.78	2.86
Natural Process	67.74	85.78	94.94	6.68	6.09	3.37
Circuit	69.12	82.98	93.49	20.24	8.36	4.67

Appendix

Table 7: Per-domain geometric grounding on the same 120-image subset. Box mIoU is reported in percent and connector L2 in pixels.

Figure 8: Geometric grounding examples on two original Knossos test images. Boxes show node localization, while colored polylines and markers show reference-conditioned connector predictions. No predicted coordinates are manually adjusted.

Open-source VLMs
	Knossos							TopoBench-180
Method	Food Web	Network	Workflow	Natural Process	Circuit	Map Route	Avg.	Web-style	Network-style
Qwen3-VL-32B	99.3	99.5	99.4	97.9	87.8	93.9	96.3	96.3	87.5
Qwen3-VL-8B	99.3	99.6	99.4	96.0	86.3	90.9	95.2	95.4	88.9
GLM-4.6V-Flash	98.9	98.7	99.0	98.0	85.4	90.8	95.1	94.3	86.3
MiMo-VL-7B-RL	87.0	98.6	92.3	88.7	83.3	90.4	90.0	73.7	81.2

Appendix

Table 8: Node F1 (%) across the frozen Knossos internal test set and TopoBench-180. Avg. is the unweighted macro-average of the six Knossos domain-level scores. Dashes indicate results that are not available.

Backbone	Protocol	Calls/img	API calls/img	S	M	D	Overall	Tokens/img	Cost/img ($)
GLM-4.6V-Flash	One-step	1.00	0.00	61.81	106.00	153.92	107.24	–	0.000
GLM-4.6V-Flash	Staged	5.23	0.00	83.39	126.86	204.33	138.19	–	0.000
GPT-5.6	One-step	1.00	1.00	13.89	26.98	28.90	23.26	3,843	0.052
GPT-5.6	Staged	5.23	5.23	21.99	34.55	54.81	37.12	10,747	0.147

Appendix

Table 9: Inference efficiency on the same 30-image test slice (10 simple, 10 medium, and 10 difficult). S/M/D denote mean latency (seconds per image) by difficulty; Overall is the mean across all 30 images.

Stage	Epochs	Steps	Learning rate	Initialization
NODE	1	1,125	$5\!\times\!10^{-5}$	Pretrained VLM
EDGE	1	5,960	$1\!\times\!10^{-5}$	Node predictor adapter
TRACE	1	5,801	$1\!\times\!10^{-5}$	Node predictor adapter

Appendix

Table 10: Stage-specific supervised-training configuration. “Initialization” denotes the adapter state from which each specialist is trained. Steps are optimizer steps after gradient accumulation.

Structured Prediction

Structured prediction maps inputs to outputs whose labels, parts, or decisions must fit together as sequences, sets, graphs, or hierarchies.

See all 30 papers

TopoAgent: A Structure-Aware Perception-to-Reasoning Framework for Diagram-to-Graph Topology Extraction with Large Vision-Language Models
Aug 27, 2026Bangwei Guo, Xujiang Zhao, Yanchi Liu +7VLM Reasoning
Rutgers University · Meta · NEC Labs America +2
Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams
Jun 2, 2026Abhishek Kumar, Isha Motiyani, Tilak Kasturi +3VLM Evaluation Visual Question Answering
Predii · Oak Ridge National Laboratory · Independent Researcher
DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Apr 28, 2026Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande +4VLM Evaluation Visual Reasoning
Arizona State University · Adobe Research · University of Maryland

Aug 27, 2026J/K move · Enter open · S save

TopoAgent: A Structure-Aware Perception-to-Reasoning Framework for Diagram-to-Graph Topology Extraction with Large Vision-Language Models

Bangwei Guo, Xujiang Zhao, Yanchi Liu, Wei Cheng+6

Rutgers University · Meta · NEC Labs America+2

VLM Reasoning

Open paper AskPDF arXiv

Diagram-to-graph topology extraction aims to extract a graph of entities and their connections from a structural diagram. This task remains challenging for current vision-language models because it requires both fine-grained perceptual grounding and topology-aware reasoning with global consistency. We present TopoBench-180, a human-verified benchmark for diagram-to-graph topology extraction, and TopoAgent, a structure-aware perception-to-reasoning framework for reliable topology extraction using large vision-language models. TopoBench-180 contains 180 structural diagrams spanning Web-style and Network-style categories, paired with canonical graph annotations. TopoAgent progressively extracts the target graph by combining grounded perception, global structural priors, canonical node inventory construction, node-centric local-to-global relation reasoning, and topological consistency enforcement. Experiments on TopoBench-180 show that TopoAgent outperforms strong vision-language model baselines and recent visual reasoning frameworks, especially on edge extraction. More broadly, this work fills an important gap in multimodal structured understanding by establishing a benchmark and framework for diagram-to-graph topology extraction. The benchmark and associated resources will be publicly released at https://huggingface.co/datasets/WayneGuo0011/TopoBench-180.

Abstract

Structured Prediction

Explore similar work

TopoAgent: A Structure-Aware Perception-to-Reasoning Framework for Diagram-to-Graph Topology Extraction with Large Vision-Language Models

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams