SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

Hao Wang; Zixuan Li; Long Bai; Xiaolong Jin; Jiafeng Guo; Tat-Seng Chua; Tian Tian

cs.CLOct 20, 2026

SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

Authors: Hao Wang, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Tat-Seng Chua, Tian Tian

Organizations: Institute of Computing Technology, Chinese Academy of Sciences · State Key Laboratory of AI Safety · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · National University of Singapore ·

Abstract

Recent agents have demonstrated strong capabilities in generating images from textual prompts, motivating their utilization for scientific diagram generation. Scientific diagrams naturally span diverse types, each characterized by distinct visual structures and content organization. Consequently, during peer review and other scientific evaluation, human experts naturally consider type-specific criteria when assessing their quality. However, existing VLM-based evaluation benchmarks fail to reflect the above human evaluation practices for two main reasons: 1) benchmark datasets typically lack a systematic categorization of different diagram types and thus adopt the same generic criteria for all diagrams, and 2) their evaluation criteria are often designed heuristically, without explicit alignment to human evaluation preferences. Motivated by these limitations, we introduce SciDiagBench, which consists of a benchmark dataset that systematically categorizes scientific diagrams into 11 types and a human preference-guided evaluation method. Specifically, SciDiagBench contains 165 scientific diagrams across 10 broad domains, with types including graphical abstracts, technical roadmaps, and illustrative cases. Based on this dataset, SciDiagBench develops an evaluation method that leverages a preference-guided calibration process to incorporate human preferences into the generation of type-specific criteria. Ultimately, SciDiagBench enables more reliable automated evaluation that better aligns with human evaluation across diverse scientific diagrams.

Figures & tables

Figure 1: Overview of the dynamic hierarchical evaluation framework, combining type-specific criteria generation with human preference alignment.

Agent	TL	LD	PE	AF	CSA	CF	TR	GA	ESD	IC	MD	Overall
Vanilla	0.4406	0.4168	0.5626	0.4670	0.4947	0.5820	0.5094	0.6228	0.5258	0.3740	0.4566	0.4957
PaperBanana	0.5272	0.4917	0.5975	0.5665	0.4926	0.5703	0.5361	0.6195	0.4610	0.4824	0.4985	0.5312
Paper2Figure	0.4714	0.4514	0.6060	0.4314	0.3337	0.4594	0.4057	0.6034	0.4555	0.4865	0.3241	0.4571
AutoFigure	0.2172	0.1162	0.1189	0.2132	0.2222	0.1919	0.2905	0.2956	0.1604	0.1476	0.2137	0.1989

Table 1: Results across different diagram types on SciDiagBench .

Method	Structure			Content			Aesthetics
Method	VOL	LA	Weighted	VER	CA	Weighted	Aesthetics
Vanilla	0.5083	0.4756	0.4909	0.5057	0.4650	0.4829	0.5393
PaperBanana	0.5412	0.5259	0.5326	0.5318	0.5158	0.5224	0.5544
Paper2Figure	0.4881	0.4092	0.4495	0.4916	0.3991	0.4433	0.5108
AutoFigure	0.2318	0.1892	0.2106	0.1549	0.1943	0.1808	0.2020

Table 2: Results across different evaluation dimensions on SciDiagBench .

Figure 2: Sensitivity of evaluation results to different judge models. (a) Overall relative scores of different generation agents. (b) Raw score distributions produced by Gemini 3.1 Pro and GPT-5.4.

Human stratum	GPT	$\Delta$ GPT	Gemini	$\Delta$ Gemini	GPT DAR	Gemini DAR
Strong Ref.	0.4690	-0.0310	0.3806	-0.1194	0.5417	0.8750
Slight Ref.	0.4882	-0.0118	0.4357	-0.0643	0.4783	0.7826
Slight Gen.	0.5210	+0.0210	0.6249	+0.1249	0.6552	0.9310
Strong Gen.	0.5511	+0.0511	0.6838	+0.1838	1.0000	1.0000

Table 3: Human preference alignment evaluation on SciDiagBench -VanillaPref. Ref. and Gen. denote reference-preferred and generated-preferred, respectively.

\Delta

denotes the judge score minus 0.5, which indicates the direction and strength of the judge preference.

Experimental setup		Directional Agreement Rate			Strength Alignment
TSC	HPA	Overall	Gen.-pref.	Ref.-pref.	Kendall’s $\tau_{b}$	Spearman’s $\rho$
–	–	0.6952	0.9143	0.5857	0.4787	0.5979
–	✓	0.7048	0.9429	0.5857	0.2085	0.2746
✓	–	0.7619	0.9429	0.6714	0.4974	0.6205
✓	✓	0.8571	0.9429	0.8143	0.5943	0.7329

Table 4: Ablation results for preference alignment evaluation on SciDiagBench -VanillaPref.

Experimental setup		Stability metrics
TSC	HPA	ICC	Between var.	Within var.	Direction flip	Criterion flip	Criterion entropy
–	–	0.8890	0.0119	0.00149	0.1333	0.1958	0.1727
–	✓	0.8718	0.0113	0.00166	0.1212	0.1939	0.1708
✓	–	0.8613	0.0141	0.00228	0.1333	0.1883	0.1664
✓	✓	0.8714	0.0177	0.00261	0.1212	0.2030	0.1797

Table 5: Stability over three independent scoring runs. Between var. and within var. denote between-instance and within-instance variance.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Representative examples of categories excluded from end-to-end generation by agents.

Panel of Figure 4 — Figure 4: Example illustrating potential training-data contamination filtering. The reference diagram (a) and Vanilla-generated output (b) share a similar organization of evolutionary stages and branches, motivating precautionary exclusion during manual screening. Both images are shown without cropping or other content modifications.

Figure 5: Domain distribution of all 165 diagrams. Bar lengths indicate counts; labels at bar ends indicate percentages.

Dimension	TL	LD	PE	AF	CSA	CF	TR	GA	ESD	IC	MD
Structure	0.45	0.46	0.45	0.50	0.50	0.52	0.50	0.41	0.46	0.35	0.45
Content	0.40	0.39	0.40	0.35	0.35	0.33	0.35	0.40	0.38	0.50	0.40
Aesthetics	0.15	0.15	0.15	0.15	0.15	0.15	0.15	0.19	0.16	0.15	0.15

Appendix

Table 6: Type-specific top-level dimension weights

W_{d,\tau}

. Each column sums to one. Type abbreviations follow Table 1 .

Figure 6: Human annotation interface used to compare a generated diagram with its reference. The screenshot is reproduced unchanged and includes Figure 6A from Lin Zhang, Chang Deng, Xu Ji, and Xiaojing Liu, “Liquid metals power advanced nuclear energy systems,” The Innovation 6, 100959 (2025), DOI: 10.1016/j.xinn.2025.100959 . The source figure is reproduced under the CC BY-NC-ND 4.0 license; no changes were made.

Panel of Figure 7 — Figure 7: Top row: (a) the original human-created reference, (b) AutoFigure output, and (c) Paper2Figure output. Bottom row: (d) Vanilla output and (e) PaperBanana output.

Method	Dimension	Diagram type											Overall
Method	Dimension	TL	LD	PE	AF	CSA	CF	TR	GA	ESD	IC	MD	Overall
PaperBanana	Structure (weighted)	0.5379	0.4843	0.5797	0.5632	0.5091	0.5834	0.5243	0.6204	0.4572	0.5037	0.4953	0.5326
	VOL	0.5302	0.5086	0.5738	0.5697	0.5076	0.5786	0.5295	0.6186	0.4833	0.5361	0.5176	0.5412
	LA	0.5459	0.4659	0.5893	0.5553	0.5105	0.5896	0.5174	0.6228	0.4355	0.4764	0.4766	0.5259
	Content (weighted)	0.5002	0.4827	0.6195	0.5574	0.4810	0.5557	0.5432	0.6090	0.4478	0.4586	0.4914	0.5224
	VER	0.5313	0.5168	0.6306	0.5683	0.4821	0.5371	0.5577	0.5858	0.4685	0.4779	0.4935	0.5318

Appendix

Table 7: Per-category dimension-level results for PaperBanana and Vanilla. Structure and Content rows are the weighted top-level scores; VOL, LA, VER, and CA are their corresponding sub-dimensions.

VLM variants
Model	TL	LD	PE	AF	CSA	CF	TR	GA	ESD	IC	MD	Overall
Default	0.5272	0.4917	0.5975	0.5665	0.4926	0.5703	0.5361	0.6195	0.4610	0.4824	0.4985	0.5312
Gemini 3.1 Pro	0.4439	0.3824	0.5617	0.5380	0.4992	0.5835	0.4835	0.5699	0.5200	0.4751	0.4204	0.4980
Qwen3.5-Plus	0.4723	0.4103	0.5582	0.4905	0.4895	0.5721	0.4773	0.5850	0.4465	0.4750	0.4681	0.4950
Seed 2.0 Pro	0.5277	0.5108	0.5758	0.4620	0.4848	0.5614	0.4597	0.5808	0.4371	0.4455	0.4624	0.5007
Multimodal Generative Model variants

Appendix

Table 8: PaperBanana results under different underlying models. The default configuration pairs GPT-5.4 with Gemini 3 Pro Image; each variant replaces one component while holding the other fixed. Type abbreviations follow Table 1 .

Experimental setup		Directional Agreement Rate			Strength Alignment
TSC	HPA	Overall	Gen.-pref.	Ref.-pref.	Kendall’s $\tau_{b}$	Spearman’s $\rho$
–	–	0.4952	0.7143	0.3857	0.0989	0.1292
–	✓	0.5143	0.8286	0.3571	0.2680	0.3444
✓	–	0.5143	0.7143	0.4143	0.0625	0.0710
✓	✓	0.5714	0.7143	0.5000	0.2277	0.2934

Appendix

Table 9: Human-alignment results for the

2\times 2

evaluation-framework analysis on the 105 high-agreement instances using GPT-5.4. Gen. and Ref. recall correspond to human preferences for the generated and reference diagrams, respectively. DAR is the primary endpoint.

Figure 8: Stability across diagram types. The left panel reports the type-level standard deviation (SD) across three runs, while the right panel reports criterion-level pairwise flip rates with $\varepsilon=0$ .

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models
Jun 24, 2026Davie ChenScientific Visualization Multimodal Model Evaluation
From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
Sep 1, 2026Raul Ortega, José Manuel Gómez-PérezVisual Question Answering Scientific QA
Language Technology Research Laboratory Expert.ai 17 Henri Dunant, 28036 Madrid, Spain
DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context
Feb 28, 2026Ling Yue, Tingwen Zhang, Jiaying Wang +2Document Image Understanding Document Understanding
Rensselaer Polytechnic Institute · University of Chicago

Jun 24, 2026J/K move · Enter open · S save

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

Davie Chen

Scientific Visualization Multimodal Model Evaluation T2I Generation Evaluation

Open paper AskPDF arXiv

Text-to-image and multimodal generative models are increasingly used to produce scientific figures such as mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts. Yet existing image-generation benchmarks (e.g., GenEval, T2I-CompBench, DPG-Bench) evaluate natural images and measure compositionality, object counting, or photorealism. None of them measure what makes a generated scientific figure usable: correct and legible text labels, faithful depiction of entities and their relations, coherent diagrammatic structure, and adherence to disciplinary drawing conventions. We introduce SciDraw-Bench, a benchmark of 32 structured scientific-figure generation tasks spanning eight figure types and ten disciplines, where each task pairs a natural-language prompt with a machine-checkable specification of required labels, relations, components, conventions, and negative constraints. We propose a four-dimensional evaluation protocol: Text Fidelity (OCR-based label recall and character error rate), Semantic Correctness (vision-language-model judging against the specification), Structural Quality, and Convention Adherence, together with a meta-evaluation protocol and a preliminary inter-judge reliability analysis (human-rating validation is ongoing). We evaluate a domain-specific system, SciDraw AI, against representative general-purpose text-to-image models, and outline a code-to-figure baseline as a planned extension. In a pilot over all eight figure types, the domain-specific system substantially outperforms the general-purpose baselines on every dimension and figure type, with the largest gaps on semantic correctness and convention adherence; text fidelity remains the hardest dimension for all systems.

Abstract

VLM Evaluation

Explore similar work

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding

DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context