SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detection

You Hu; Chenzhuo Zhao; Changfa Mo; Haotian Liu; Xiaobai Li

cs.CVApr 9, 2026

SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detection

Authors: You Hu, Chenzhuo Zhao, Changfa Mo, Haotian Liu, Xiaobai Li

Organizations: Zhejiang University, Hangzhou, China · Independent Researcher, Beijing, China · University of Oulu, Oulu, FINLAND

Abstract

Modern multimodal generators can now produce scientific figures at near-publishable quality, creating a new challenge for visual forensics and research integrity. Unlike conventional AI-generated natural images, scientific figures are structured, text-dense, and tightly aligned with scholarly semantics, making them a distinct and difficult detection target. However, existing AI-generated image detection benchmarks and methods are almost entirely developed for open-domain imagery, leaving this setting largely unexplored. We present the first benchmark for AI-generated scientific figure detection. To construct it, we develop an agent-based data pipeline that retrieves licensed source papers, performs multimodal understanding of paper text and figures, builds structured prompts, synthesizes candidate figures, and filters them through a review-driven refinement loop. The resulting benchmark covers multiple figure categories, multiple generation sources and aligned real--synthetic pairs. We benchmark representative detectors under zero-shot, cross-generator, and degraded-image settings. Results show that current methods fail dramatically in zero-shot transfer, exhibit strong generator-specific overfitting, and remain fragile under common post-processing corruptions. These findings reveal a substantial gap between existing AIGI detection capabilities and the emerging distribution of high-quality scientific figures. We hope this benchmark can serve as a foundation for future research on robust and generalizable scientific-figure forensics. The dataset is available at https://github.com/Joyce-yoyo/SciFigDetect.

Figures & tables

Figure 1. Overview of our benchmark. Representative real–synthetic examples from three figure categories: Illustration , Overview , and Experimental Figures . For each case, we show the real figure, the prompt, and synthetic counterparts generated by Nano Banana and GPT .

Dataset	Image Content	Generator Source	Public Availability	Real Images	Fake Images	Resolution	Structured / Text-Dense	Paper-Context Aware
UADFV ( Yang et al., 2019 )	Face	GAN	$\times$	241	252	Varied	$\times$	$\times$
FakeSpotter ( Wang et al., 2021 )	Face	GAN	$\times$	6,000	5,000	Varied	$\times$	$\times$
DFFD ( Dang et al., 2020 )	Face	GAN	$\checkmark$	58,703	240,336	Varied	$\times$	$\times$
APFDD ( Gandhi and Jain, 2020 )	Face	GAN	$\times$	5,000	5,000	Varied	$\times$	$\times$
ForgeryNet ( He et al., 2021 )	Face	GAN	$\checkmark$	1,438,201	1,457,861	Varied	$\times$	$\times$
DeepArt ( Wang et al., 2023 )	Art	Diffusion	$\checkmark$	64,479	73,411	Varied	$\times$	$\times$

Table 1. Comparison with representative AI-generated image detection datasets. Our benchmark differs from prior work by focusing on structured, text-dense scientific figures and by explicitly preserving figure-related paper context.

Figure 2. Overview of the data construction pipeline. From licensed source papers and figure-related context, our framework performs multimodal understanding, prompt planning, and iterative generation–review refinement to construct benchmark samples $z=(c,f_{\mathrm{real}},f_{\mathrm{syn}},a)$ .

Figure 3. Dataset statistics. Left: per-category sample counts for real, Nano Banana, and GPT-generated figures. Right: topic distribution of the collected diagrams. Together, they show both the scale and diversity of our benchmark.

	All Figures				Illustrations				Overviews				Experimental Figures
Method	Acc ${}_{\text{real}}$	Acc ${}_{\text{fake}}$	$\mathrm{AvgAcc}$	AP	Acc ${}_{\text{real}}$	Acc ${}_{\text{fake}}$	$\mathrm{AvgAcc}$	AP	Acc ${}_{\text{real}}$	Acc ${}_{\text{fake}}$	$\mathrm{AvgAcc}$	AP	Acc ${}_{\text{real}}$	Acc ${}_{\text{fake}}$	$\mathrm{AvgAcc}$	AP
CNNSpot ( Wang et al., 2020 )	99.60	2.74	35.03	72.19	100.00	1.20	34.13	72.12	100.00	1.22	34.14	74.80	99.49	3.17	35.28	72.23
PatchFor ( Chai et al., 2020 )	99.94	0.16	33.42	67.29	100.00	0.11	33.41	67.40	99.85	0.08	33.33	66.47	99.95	0.18	33.44	67.61
UniFD ( Ojha et al., 2023 )	89.00	7.80	48.40	49.11	96.73	2.07	49.40	49.15	97.57	1.90	49.73	46.41	86.19	8.39	47.29	45.40
LGrad ( Tan et al., 2023 )	98.89	8.48	53.68	65.30	99.78	8.17	53.98	75.40	99.39	11.17	55.28	80.87	98.72	8.09	53.40	61.81
NPR ( Tan et al., 2024b )	94.92	13.24	40.47	75.33	98.26	11.76	40.60	80.27	99.24	13.07	41.79	84.34	93.81	13.44	40.23	73.76

Table 2. Zero-shot performance of existing AI-generated image detectors on SciFigDetect. Models are evaluated without any adaptation to scientific figures.

\mathrm{Acc}_{real}

denotes accuracy on real figures, and

\mathrm{Acc}_{fake}

summarizes accuracy on synthetic figures.

\mathrm{AvgAcc}

is the unweighted mean of

\mathrm{Acc}_{real}

\mathrm{Acc}_{GPT}

, and

\mathrm{Acc}_{Banana}

. Best and second-best results are highlighted in bold and underline, respectively.

	Train on Banana				Train on GPT				Train on Banana+GPT
Method	Acc ${}_{\text{real}}$	Acc ${}_{\text{GPT}}$	Acc ${}_{\text{Banana}}$	$\mathrm{AvgAcc}$	Acc ${}_{\text{real}}$	Acc ${}_{\text{GPT}}$	Acc ${}_{\text{Banana}}$	$\mathrm{AvgAcc}$	Acc ${}_{\text{real}}$	Acc ${}_{\text{GPT}}$	Acc ${}_{\text{Banana}}$	$\mathrm{AvgAcc}$
CNNSpot ( Wang et al., 2020 )	98.67	40.04	78.46	72.39	97.93	77.95	11.11	62.33	94.84	78.75	73.85	82.48
PatchFor ( Chai et al., 2020 )	90.27	6.03	53.28	49.86	96.28	98.92	8.38	67.86	98.17	95.14	11.58	68.30
UniFD ( Ojha et al., 2023 )	88.70	67.00	81.00	78.90	90.10	66.40	42.50	66.33	93.20	66.30	66.70	75.40
LGrad ( Tan et al., 2023 )	95.28	50.68	82.92	76.29	96.95	89.89	23.89	70.24	94.84	94.45	86.74	92.01
NPR ( Tan et al., 2024b )	65.80	87.69	98.93	84.14	81.41	96.96	51.21	76.53	93.75	93.43	94.69	93.96

Table 3. Cross-generator image classification under three training protocols. Models are evaluated on real, GPT-generated, and Nano Banana-generated scientific figures.

\mathrm{AvgAcc}

denotes the unweighted mean of

\mathrm{Acc}_{real}

\mathrm{Acc}_{GPT}

, and

\mathrm{Acc}_{Banana}

, so that each subset contributes equally regardless of its size. All reported numbers are averaged over the 10 paper-level folds.

		JPEG Compression				WebP Compression				Gaussian Blur			Gaussian Noise
Method	Clean	q=95	q=75	q=50	q=30	q=95	q=75	q=50	q=30	$\sigma$ =0.5	$\sigma$ =1.0	$\sigma$ =2.0	$\sigma$ =5	$\sigma$ =10	$\sigma$ =20
CNNSpot ( Wang et al., 2020 )	82.50	80.78	79.56	78.17	75.74	77.80	77.07	75.41	72.78	80.99	80.92	83.45	56.74	55.28	56.12
UniFD ( Ojha et al., 2023 )	75.40	55.73	63.47	61.00	61.27	74.00	71.66	70.33	71.33	69.00	68.27	66.20	66.33	60.73	53.00
NPR ( Tan et al., 2024b )	93.96	87.38	81.51	78.76	75.94	83.29	78.22	73.04	66.86	91.65	89.54	79.13	69.05	65.45	50.49
Effort ( Yan et al., 2025b )	95.58	71.60	70.80	71.16	71.02	69.69	69.33	69.08	68.25	70.34	68.41	67.40	66.75	66.75	66.71

Table 4. Robustness under image degradation. Classification accuracy (%) on degraded test sets. Models are trained on clean Banana+GPT data and evaluated under JPEG/WebP compression, Gaussian blur, and Gaussian noise.

Figure 4. Cross-generator generalization gap. Averaged across detectors, models trained on one generator perform much better on the seen generator than on the unseen one. This large in-domain vs. cross-generator gap indicates strong generator-specific overfitting and a clear domain gap between Banana- and GPT-generated scientific figures.

AI-Generated Image Detection

AI-generated image detection identifies synthetic or AI-edited images and develops methods that generalize across generators, domains, and image transformations.

See all 186 papers

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models
Jun 24, 2026Davie ChenScientific Visualization Multimodal Model Evaluation
GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models
Apr 5, 2026Yaohan Guan, Pristina Wang, Najim Dehak +3T2I Generation Evaluation VLM Evaluation
Johns Hopkins University
SciFigPlag-Bench: A Benchmark for Provenance-Aware Scientific Figure Plagiarism Detection
Jul 31, 2026Zhiying Cui, Minghao Yang, Linlin Gao +2VLM Evaluation Image Forgery Detection
1Ningbo University · 2Hokkaido University · 3IBM Research

Jun 24, 2026J/K move · Enter open · S save

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

Davie Chen

Scientific Visualization Multimodal Model Evaluation T2I Generation Evaluation

Open paper AskPDF arXiv

Text-to-image and multimodal generative models are increasingly used to produce scientific figures such as mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts. Yet existing image-generation benchmarks (e.g., GenEval, T2I-CompBench, DPG-Bench) evaluate natural images and measure compositionality, object counting, or photorealism. None of them measure what makes a generated scientific figure usable: correct and legible text labels, faithful depiction of entities and their relations, coherent diagrammatic structure, and adherence to disciplinary drawing conventions. We introduce SciDraw-Bench, a benchmark of 32 structured scientific-figure generation tasks spanning eight figure types and ten disciplines, where each task pairs a natural-language prompt with a machine-checkable specification of required labels, relations, components, conventions, and negative constraints. We propose a four-dimensional evaluation protocol: Text Fidelity (OCR-based label recall and character error rate), Semantic Correctness (vision-language-model judging against the specification), Structural Quality, and Convention Adherence, together with a meta-evaluation protocol and a preliminary inter-judge reliability analysis (human-rating validation is ongoing). We evaluate a domain-specific system, SciDraw AI, against representative general-purpose text-to-image models, and outline a code-to-figure baseline as a planned extension. In a pilot over all eight figure types, the domain-specific system substantially outperforms the general-purpose baselines on every dimension and figure type, with the largest gaps on semantic correctness and convention adherence; text fidelity remains the hardest dimension for all systems.

Abstract

AI-Generated Image Detection

Explore similar work

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models

SciFigPlag-Bench: A Benchmark for Provenance-Aware Scientific Figure Plagiarism Detection