DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context

Ling Yue; Tingwen Zhang; Jiaying Wang; Zhen Xu; Shaowu Pan

cs.IRFeb 28, 2026

DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context

Authors: Ling Yue, Tingwen Zhang, Jiaying Wang, Zhen Xu, Shaowu Pan

Organizations: Rensselaer Polytechnic Institute · University of Chicago

Abstract

Scientific papers use schematic diagrams to communicate methods, workflows, and system structure, yet existing scientific-figure corpora often mix them with plots, screenshots, and photographs and rarely preserve document context. We introduce DiagramBank, a quality-audited dataset of 57,100 schematic diagrams curated from OpenReview-hosted AI/ML venues. Each record links a diagram image to its paper title, abstract, figure caption, in-text figure-reference spans, venue/year metadata, provenance fields, and filtering labels. DiagramBank is a reusable resource for scientific-document understanding, diagram retrieval, corpus analysis, and future benchmark construction. We describe its extraction and cascade-filtering pipeline, release schema, confidence-controlled views, dataset card, and indexing utilities. A manual blind audit of the released cascade-filtered records estimates 93.67% precision, and a separate CLIP threshold analysis characterizes the precision--coverage trade-off for simpler filtering views. We further provide lightweight metadata-indexing and authoring examples to illustrate downstream protocols without treating these utilities as standalone methods. The code is public at: https://github.com/csml-rpi/DiagramBank.

Figures & tables

Figure 1: DiagramBank dataset and resource overview. This illustrative schematic contrasts weak figure-only extraction with context-rich records, a verified diagram collection, and dataset-grounded diagram-generation use.

Resource	Domain and scale	Schematic focus	Document context	Quality / retrieval support
SciCap	Scientific figures; 416,000 figures	Mixed scientific figures for caption generation	Partial paper metadata and captions; no body-reference context	Captioning supervision; not diagram-filtered
DocFigure	Scientific documents; 33,000 figures	Figure-type classification over document figures	Caption information only; no paper-level context	Type labels; not retrieval-ready
ACL-Fig	Computational-linguistics papers; 112,000 figures from 56,000 papers	Figures and tables from ACL papers	Paper metadata, captions, and limited body context	Classification-oriented; limited diagram-specific QA
AI2D	School science; 5,000 diagrams	Dense labels for educational diagrams	No scientific-paper metadata or caption context	Diagram annotations outside scholarly-paper domain
SMSMO	Scientific summaries; 24,000 papers	Representative figure selection for summaries	Paper metadata and captions; no extracted figure-reference spans	Summary evaluation rather than diagram retrieval
Generation systems	General or scientific diagrams; system dependent	Produced or planned diagrams	System-dependent context	Evaluate generation behavior, but do not release a diagram corpus

Table 1: Comparison with related figure and diagram resources. DiagramBank is differentiated by diagram-specific filtering, multi-level document context, measured release precision, and retrieval-ready metadata rather than by a new downstream method.

Cascade stage	Candidate figures	Kept diagrams	Decision rule
Tier 1: CLIP predicts diagram	60,717	52,034	Two independent VLM judges classify each candidate; 46,524 unanimous and 3,645 majority cases are kept, and a larger VLM confirms 1,865 of 10,548 minority cases.
Tier 2: non-diagram with CLIP confidence $<0.70$	33,459	5,066	Both smaller judges must independently relabel the figure as diagram; the larger VLM then confirms 5,066 of 5,172 consensus candidates.
Final cascade release	94,176	57,100	Accepted-paper candidate pool; final venue counts are ICLR 20,516, ICML 11,267, NeurIPS 19,655, and TMLR 5,662.

Table 2: Cascade funnel for the primary DiagramBank release. CLIP is the low-cost routing stage; the kept records are classified by independent VLM judgments and, where needed, confirmation by a larger VLM. Counts summarize the final cascade logs; exact model identifiers are reported in the release documentation.

Venue	All CLIP diagrams		High-confidence CLIP diagrams
	All papers	Accepted papers	All papers	Accepted papers
ICLR	46,460	19,261	31,325	12,550
ICML	12,267	12,204	8,045	8,005
NeurIPS	21,569	20,582	14,226	13,533
TMLR	9,126	5,761	6,169	3,849
Total	89,422	57,808	59,765	37,937

Table 3: Intermediate CLIP-based subset sizes for the 2017–2025 analysis slice. Accepted-paper columns are filters over each CLIP view; Table 2 separately reports the vision-language-model cascade pool.

View / denominator	Scope	Count	Use in paper
Classified figure pool	Release coverage	475,616	Initial figure pool used for CLIP routing and cascade accounting across venue-specific OpenReview coverage.
Non-table analysis slice	2017–2025	452,339	Denominator for aggregate figure-type statistics shown in the compact corpus-statistics graphic.
All-CLIP diagram view	2017–2025	89,422	Intermediate view containing every figure whose first-stage CLIP label is diagram.
High-confidence CLIP view	2017–2025	59,765	Intermediate view applying $\texttt{clip\_confidence}\geq 0.85$ .
Accepted-paper cascade candidates	Release coverage	94,176	Candidate pool routed through the VLM cascade (60,717 Tier 1 + 33,459 Tier 2).
Primary DiagramBank release	Release coverage	57,100	Cascade-filtered schematic diagrams distributed as the main resource.

Table 4: Denominators and release views. The release coverage is venue-specific (ICLR 2017–2026, NeurIPS 2021–2022 and 2024–2025, ICML 2023–2025, and TMLR 2022–2026), while several descriptive statistics use a 2017–2025 non-table analysis slice.

Panel of Figure 2 — Figure 2: Compact corpus statistics. (a) CLIP-predicted figure-type distribution over 452,339 extracted non-table figures, annotated with counts and mean confidence scores. (b) Venue-level visual density: bars show figures per paper, the line shows average caption length, and bar labels show breadth, defined as the percentage of papers with at least one extracted non-table figure.

Figure 3: Example metadata-indexing use. DiagramBank metadata supports title retrieval, abstract reranking, and caption-level exemplar retrieval.

Evaluation item	Value
Queries	20 scientific-diagram queries
Pool	Top-50 from both retrieval strategies
Judged pairs	1,862 unique query–diagram pairs
Relevance judges	Three VLM judges; exact model identifiers are in the release documentation
Rubric	0 = not relevant; 1 = marginal; 2 = high
Aggregation	Majority vote; Fleiss’ $\kappa=0.425$ ; 52% unanimous

Table 5: Automatic retrieval sanity check. DiagramBank’s multi-level metadata supports complementary paper-level and caption-level search strategies.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Text-only output: a generic horizontal process diagram generated from the Code2MCP query alone.

(b) DiagramBank-conditioned output: the same query with retrieved exemplars, adding code-repository and service-output structure.

Document Image Understanding

Document image understanding turns page images into recognized text, layout, structured data, and answers grounded in visual evidence.

See all 148 papers

SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams
Oct 20, 2026Hao Wang, Zixuan Li, Long Bai +4VLM Evaluation Image Generation Evaluation
Institute of Computing Technology, Chinese Academy of Sciences · State Key Laboratory of AI Safety · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences +2
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
Aug 12, 2026Weihao Bo, Shan Zhang, Yanpeng Sun +7Code Generation Multimodal QA
Nanjing University of Science and Technology · Baidu Inc · AIML, Adelaide University +4
From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
Sep 1, 2026Raul Ortega, José Manuel Gómez-PérezVisual Question Answering Scientific QA
Language Technology Research Laboratory Expert.ai 17 Henri Dunant, 28036 Madrid, Spain

Oct 20, 2026J/K move · Enter open · S save

SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

Hao Wang, Zixuan Li, Long Bai, Xiaolong Jin+3

Institute of Computing Technology, Chinese Academy of Sciences · State Key Laboratory of AI Safety · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences+2

VLM Evaluation Image Generation Evaluation Rubric-Based Evaluation Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

Recent agents have demonstrated strong capabilities in generating images from textual prompts, motivating their utilization for scientific diagram generation. Scientific diagrams naturally span diverse types, each characterized by distinct visual structures and content organization. Consequently, during peer review and other scientific evaluation, human experts naturally consider type-specific criteria when assessing their quality. However, existing VLM-based evaluation benchmarks fail to reflect the above human evaluation practices for two main reasons: 1) benchmark datasets typically lack a systematic categorization of different diagram types and thus adopt the same generic criteria for all diagrams, and 2) their evaluation criteria are often designed heuristically, without explicit alignment to human evaluation preferences. Motivated by these limitations, we introduce SciDiagBench, which consists of a benchmark dataset that systematically categorizes scientific diagrams into 11 types and a human preference-guided evaluation method. Specifically, SciDiagBench contains 165 scientific diagrams across 10 broad domains, with types including graphical abstracts, technical roadmaps, and illustrative cases. Based on this dataset, SciDiagBench develops an evaluation method that leverages a preference-guided calibration process to incorporate human preferences into the generation of type-specific criteria. Ultimately, SciDiagBench enables more reliable automated evaluation that better aligns with human evaluation across diverse scientific diagrams.

Abstract

Document Image Understanding

Explore similar work

SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding