RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors

Hicham Eddoubi; Jonas Ricker; Federico Cocchi; Angelo Sotgiu; Maura Pintor; Marcella Cornia; Lorenzo Baraldi; Asja Fischer

cs.CVJun 4, 2025

RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors

Authors: Hicham Eddoubi, Jonas Ricker, Federico Cocchi, Angelo Sotgiu, Maura Pintor, Marcella Cornia, Lorenzo Baraldi, Asja Fischer,

Organizations: University of Cagliari, Italy · Sapienza University of Rome, Italy · Ruhr University Bochum, Germany · University of Modena and Reggio Emilia, Italy ·

Abstract

AI-generated images (AIGIs) have reached a level of quality at which humans are unable to reliably distinguish them from real images. To counter the inherent risks of fraud and disinformation, their detection is a pressing challenge and an active research topic. While many of the presented methods claim high detection accuracy, they are often evaluated without considering their adversarial robustness. In real-world scenarios, malicious actors can modify AIGIs, e.g., through imperceptible perturbations, to evade detection systems. In this work, we provide a dataset to assess the robustness of AI-generated image detectors. We present RAID (Robust evaluation of AI-generated image Detectors), a benchmark dataset of 216k diverse and transferable adversarial images. The proposed dataset is created starting from images generated by four different text-to-image models (to promote diversity) and optimizing attacks against an ensemble of seven state-of-the-art detectors (to promote transferability). Extensive experiments show that our methodology generates adversarial images that transfer to unseen detectors, degrading the performance of every detector outside the attack ensemble that we evaluate, and provides a practical, worst-case robustness screen: failure on RAID is a strong indicator that a detector is not adversarially robust. We release our dataset at https://huggingface.co/datasets/aimagelab/RAID and evaluation code at https://github.com/pralab/RAID.

Figures & tables

Figure 1: Adversarial attack on the AIGI detector provided by sightengine . Successful evasion is shown by adding adversarial noise computed on the detector by Corvi et al. [23] . We perform an extended evaluation on commercial detectors in D .

Experiment	Setting	Surrogate ensemble	In release?
PGD Hyperparameters Ablation ( Fig. 3 )	black-box	$N\setminus\{\text{target}\}$	no
Single-detector attack (Tabs. 4 , 5 , 11 , diag.)	white-box	target only	no
Cross-model transfer (Tabs. 4 , 5 , 11 , off-diag.)	black-box	one source detector	no
Leave-one-out ensemble (Tabs. 4 , 5 , 11 , last row)	black-box	$N\setminus\{\text{target}\}$	no
Additional generators ( Tab. 6 )	black-box	$N\setminus\{\text{target}\}$	no
Other ensemble attacks ( Tab. 9 )	black-box	$N\setminus\{\text{target}\}$	no

Table 1: Threat model for each experiment: access setting, surrogate ensemble, and whether the images belong to the public RAID release.

N

is the seven-detector pool of Sect. 3.2 .

Figure 2: A diagram of the three experimental pipelines used in the paper for generating the RAID dataset. Left: We keep the top three-performing detectors intact while retraining the rest. Middle: We generate adversarial images optimizing on an ensemble of detectors. Right: We evaluate the baseline detectors on the RAID images.

		F1	Acc	AUROC
[ 23 ]	(Co23)	0.75	0.81	0.91
[ 75 ]	(O23)	0.33	0.29	0.68
[ 95 ]	(W20)	0.21	0.02	0.46
[ 12 ]	(Ca24)	0.20	0.07	0.45
[ 95 ] †	(W20 † )	0.99	0.99	0.99
[ 23 ] †	(Co23 † )	0.98	0.99	0.99

Table 2: Evaluation of each model on a subset of the clean D 3 test set. † refers to detectors trained on D 3 . F1-score computed with the AI-generated class as positive.

Figure 3: ASR of the leave-one-out ensemble PGD attack for every number of iterations

K

(ticks) and step size

\alpha

(groups), for each budget

\epsilon

(colors). Dashed lines: Gaussian noise clipped to the same

\ell_{\infty}

budget. Mean over 3 random starts and the 7 held-out detectors, on 1,000 images; shaded areas:

\pm 1

standard deviation.

LPIPS $\downarrow$			SSIM $\uparrow$
$\epsilon=\frac{8}{255}$	$\epsilon=\frac{16}{255}$	$\epsilon=\frac{32}{255}$	$\epsilon=\frac{8}{255}$	$\epsilon=\frac{16}{255}$	$\epsilon=\frac{32}{255}$
0.09 $\pm$ 0.07	0.24 $\pm$ 0.13	0.40 $\pm$ 0.16	0.85 $\pm$ 0.07	0.63 $\pm$ 0.12	0.47 $\pm$ 0.13

Table 3: LPIPS and SSIM between the adversarial images of the released RAID dataset and their clean counterparts, per perturbation budget

\epsilon

. Mean

\pm

standard deviation over all images.

	Ca24 †	Ch24C	Ch24CN	Co23 †	K24	O23 †	W20 †
Ca24 †	0.00/100.0	0.70/0.6	0.85/5.9	0.90/7.6	0.68/18.5	0.87/4.3	0.94/4.2
Ch24C	0.58/27.8	0.00/100.0	0.49/46.3	0.93/3.6	0.04/95.4	0.08/91.6	0.94/4.3
Ch24CN	0.52/34.5	0.59/17.0	0.03/96.9	0.89/7.7	0.61/26.9	0.88/3.4	0.95/3.6
Co23 †	0.32/59.4	0.65/7.5	0.08/90.8	0.00/100.0	0.49/41.4	0.82/9.9	0.01/99.3
K24	0.54/31.8	0.08/89.3	0.43/52.4	0.93/4.4	0.05/93.7	0.20/77.6	0.94/3.8
O23 †	0.57/28.6	0.03/95.3	0.71/21.6	0.93/4.2	0.06/92.4	0.00/100.0	0.93/5.5

Table 4: White-box and black-box adversarial robustness evaluation (AUROC/ASR) on PGD with

\epsilon

= 16/255 perturbation. Adversarial attacks are ran against the model in each row and evaluated on models in each column. † refers to detectors trained on the D 3 training set. Final row

N-

model refers to ensemble attacks targeting all models, excluding the evaluated against model. Bold values corresponding to white-box evaluation in single detector attacks.

	Ca24 †	Ch24C	Ch24CN	Co23 †	K24	O23 †	W20 †
Ca24 †	0.00/100.0	0.74/ $-$ 4.5	0.76/15.4	0.88/9.3	0.72/13.4	0.87/4.0	0.92/6.0
Ch24C	0.54/32.3	0.00/100.0	0.35/61.1	0.92/4.8	0.01/98.7	0.03/97.1	0.92/6.3
Ch24CN	0.48/39.7	0.55/21.9	0.01/99.2	0.87/9.7	0.57/31.8	0.86/4.9	0.94/4.3
Co23 †	0.26/67.2	0.60/15.5	0.01/98.4	0.00/100.0	0.44/47.6	0.70/23.4	0.00/100.0
K24	0.51/35.5	0.08/88.2	0.38/57.8	0.92/5.2	0.07/91.6	0.22/75.4	0.93/5.2
O23 †	0.54/32.0	0.01/98.7	0.51/43.5	0.91/5.9	0.03/96.4	0.00/100.0	0.90/8.5

Table 5: White-box and black-box adversarial robustness evaluation (AUROC/ASR) on PGD with

\epsilon

= 32/255 perturbation. Adversarial attacks are ran against the model in each row and evaluated on models in each column. † refers to detectors trained on the D 3 training set. Final row

N-

model refers to ensemble attacks targeting all models, excluding the evaluated against model. Bold values corresponding to white-box evaluation in single detector attacks.

Metric	Perturbation	Ca24 †	Ch24C	Ch24CN	Co23 †	K24	O23 †	W20 †
AUROC	Clean	0.77	0.74	0.97	0.99	0.89	0.96	0.99
	8/255	0.46	0.06	0.43	0.32	0.09	0.24	0.46
	16/255	0.46	0.02	0.23	0.20	0.04	0.15	0.29
	32/255	0.40	0.01	0.19	0.08	0.02	0.12	0.06
ASR (%)	8/255	39.5	91.9	55.5	67.4	89.5	75.4	53.5
	16/255	39.8	97.1	75.8	79.9	95.5	84.6	70.8

Table 6: Evaluation of our attack methodology on AIGIs from additional generators. For each detector, we report AUROC and ASR on images from 11 generative models, under the

N-

model ensemble attack (all models excluding the evaluated one).

		DINOv2		DINOv2-Reg		ViT-T		CoDE
Dataset	Perturbation	AUROC	ASR	AUROC	ASR	AUROC	ASR	AUROC	ASR
D 3	Clean	0.85	–	0.85	–	0.82	–	0.89	–
RAID	8/255	0.76	10.1	0.78	8.9	0.74	9.0	0.72	19.2
RAID	16/255	0.73	14.7	0.75	12.6	0.70	14.2	0.65	27.7
RAID	32/255	0.67	21.8	0.70	17.8	0.66	19.7	0.59	34.2
RAID Img	8/255	0.76	10.2	0.78	9.0	0.74	9.2	0.72	19.2

Table 7: Adversarial robustness evaluation of the four trained baseline classifiers. All detectors are trained on the D 3 training set. RAID refers to the dataset of the adversarial images saved as raw tensors, while RAID Img refers to the released dataset of adversarial images saved as PNG images. ASR (%) is the relative AUROC drop with respect to the clean D 3 test set.

		AIDE		Effort		DDD		OmniAID		DDA
Dataset	Perturbation	AUROC	ASR	AUROC	ASR	AUROC	ASR	AUROC	ASR	AUROC	ASR
D 3	Clean	0.78	–	0.70	–	0.89	–	0.78	–	0.93	–
RAID	8/255	0.66	15.5	0.55	20.5	0.71	19.5	0.29	63.1	0.52	44.2
RAID	16/255	0.60	22.1	0.56	20.0	0.67	24.9	0.15	81.1	0.34	62.9
RAID	32/255	0.50	35.1	0.48	30.7	0.57	35.1	0.09	87.9	0.16	82.5
RAID Img	8/255	0.66	15.6	0.56	20.0	0.71	19.5	0.29	63.4	0.52	44.2

Table 8: Adversarial robustness evaluation of five independently trained detectors, not part of the attack ensemble. AIDE, Effort, DDD [ 106 ] , and OmniAID use their released checkpoints trained on GenImage (SD1.4); DDA uses its released checkpoint trained on DDA-aligned MSCOCO. RAID refers to the adversarial images saved as raw tensors, while RAID Img refers to the released dataset of adversarial images saved as PNGs. ASR (%) is the relative AUROC drop with respect to the clean D 3 test set.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

$\epsilon$	Attack	Ca24 †	Ch24C	Ch24CN	Co23 †	K24	O23 †	W20 †	Mean
	PGD (Ours)	48.3	93.6	61.0	73.4	89.1	75.3	53.8	70.6
	APGD	47.6	94.6	62.5	74.1	89.6	75.8	60.0	72.0
8/255	CWA	51.7	96.5	82.4	85.9	98.8	90.6	93.7	85.7
	PGD (Ours)	54.7	97.6	78.2	87.0	94.3	86.5	73.5	81.7
	APGD	53.7	97.8	81.5	89.1	95.6	87.0	77.7	83.2
16/255	CWA	59.9	98.1	92.5	94.1	99.6	87.5	95.0	89.6

Appendix

Table 9: Comparison with other ensemble attacks in the leave-one-out setting: ASR (%) on each held-out detector. Attacks are computed on 1,000 images per class.

$\epsilon$	Post-processing	Ca24 †	Ch24C	Ch24CN	Co23 †	K24	O23 †	W20 †
8/255	No transform	48.3	93.6	61.0	73.4	89.1	75.3	53.8
	Resize $64\times 64$	6.2	9.3	24.5	56.2	38.0	11.4	26.5
	Random crop $64\times 64$	9.6	23.9	26.5	40.7	44.2	27.7	31.4
	Random flip	46.9	87.2	61.2	72.9	82.3	66.8	53.4
	JPEG ( $q=75$ )	50.6	29.1	54.6	55.2	49.3	21.4	47.5
16/255	No transform	54.7	97.6	78.2	87.0	94.3	86.5	73.5

Appendix

Table 10: Leave-one-out ensemble PGD attack with a post-processing operation applied to the adversarial images before classification: ASR (%) on each held-out detector, on 1,000 images per class. ASR is computed with respect to the clean AUROC without post-processing. Resize and random crop are followed by upsampling back to

256\times 256

; random flip is horizontal with probability 0.5.

	Ca24 †	Ch24C	Ch24CN	Co23 †	K24	O23 †	W20 †
Ca24 †	0.00/100.0	0.68/3.5	0.89/1.7	0.92/4.9	0.65/22.0	0.88/2.8	0.96/2.4
Ch24C	0.62/22.5	0.00/100.0	0.66/27.5	0.94/2.9	0.04/95.2	0.09/90.5	0.96/2.4
Ch24CN	0.57/28.0	0.60/15.0	0.01/99.2	0.91/6.3	0.71/14.2	0.88/2.9	0.95/2.7
Co23 †	0.36/55.1	0.72/ $-$ 2.2	0.12/86.7	0.00/100.0	0.56/33.2	0.86/4.9	0.01/99.1
K24	0.58/27.8	0.10/85.2	0.61/32.9	0.93/4.4	0.03/96.1	0.24/73.3	0.95/3.3
O23 †	0.60/24.3	0.03/96.2	0.85/6.3	0.94/3.2	0.05/93.7	0.00/100.0	0.95/3.1

Appendix

Table 11: White-box and black-box adversarial robustness evaluation (AUROC/ASR) on PGD with

\epsilon

= 8/255 perturbation. Adversarial attacks are ran against the model in each row and evaluated on models in each column. † refers to detectors trained on the D 3 training set. Final row

N-

model refers to ensemble attacks targeting all models, excluding the evaluated against model. Bold values corresponding to white-box evaluation in single detector attacks.

Metric	Ca24 †	Ch24C	Ch24CN	Co23 †	K24	O23 †	W20 †
AUROC	0.35 $\pm$ 0.00	0.02 $\pm$ 0.00	0.24 $\pm$ 0.00	0.16 $\pm$ 0.00	0.06 $\pm$ 0.01	0.16 $\pm$ 0.01	0.33 $\pm$ 0.01
ASR	56.1 $\pm$ 0.6	96.8 $\pm$ 0.1	72.9 $\pm$ 0.4	83.8 $\pm$ 0.2	92.7 $\pm$ 1.0	82.5 $\pm$ 1.4	66.2 $\pm$ 0.5

Appendix

Table 12: Leave-one-out ensemble PGD attack on each held-out detector at

\epsilon=16/255

(

K=10

\alpha=0.03

): mean

\pm

standard deviation over three seeded random starts, on 1,000 test images. ASR (%) is the relative AUROC drop with respect to the clean images.

Detector		Score	Score (Real)	Adversarial Score	Acc	Acc (Real)	Adversarial Acc
Sightengine		0.65	0.99	0.01	0.66	1.00	0.00
HIVE		0.59	0.99	0.45	0.60	1.00	0.44

Appendix

Table 13: Performance on commercial AIGI detectors. We report mean detector scores on AIGIs, with adversarial variants evaluated on RAID examples. Metrics on real images are also included.

Figure 4: Detection scores returned by Sightengine detector on a subset of clean and adversarial images. Higher scores indicate higher confidence that the image is AI-generated.

Figure 5: Detection scores returned by HIVE Moderation detector on a subset of clean and adversarial images. Higher scores indicate higher confidence that the image is AI-generated.

Adversarial Robustness

Adversarial robustness studies how well AI systems withstand deliberately crafted inputs, training-data corruption, and other attacks.

See all 493 papers

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection
Sep 26, 2026Aleksandr Gushchin, Khaled Abud, Georgii Bychkov +5AI-Generated Image Detection
MSU Institute for Artificial Intelligence · Trusted AI Research Center RAS · Lomonosov Moscow State University
RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetry
Mar 2, 2026Xinchang Wang, Yunhao Chen, Yuechen Zhang +4AI-Generated Image Detection Adversarial Robustness
Jiangnan University · Fudan University
SPARED: Reasoning-Based AI-Generated Image Detection via Adversarially Edited Data
Aug 13, 2026Yicheng Bao, Xiahui Guo, Xuhong Wang +1Adversarial Training Adversarial Examples
1East China Normal University · 2Shanghai Artificial Intelligence Laboratory

Sep 26, 2026J/K move · Enter open · S save

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

Aleksandr Gushchin, Khaled Abud, Georgii Bychkov, Ekaterina Shumitskaya+4

MSU Institute for Artificial Intelligence · Trusted AI Research Center RAS · Lomonosov Moscow State University

AI-Generated Image Detection

Open paper AskPDF arXiv

AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allowing models to rely on dataset-specific cues and fail on unfamiliar generators or processed images. Existing datasets provide limited support for evaluating these challenges jointly across diverse visual content. We introduce REALIS, a dataset of 1.43 million real and synthetic images generated by 42 modern text-to-image models, including the latest proprietary systems such as Nano Banana 2. REALIS combines prompts derived from real images, quality filtering, and stratified sampling to reduce class-specific shortcuts while preserving content diversity. We further introduce REALIS-Expert, a stress-test subset for high-quality synthetic images, where real and generated samples are selected with closely matched semantic and visual characteristics. We also propose a robustness protocol covering 35 transformations at five severity levels to analyze detector behavior under image processing. Based on REALIS, our benchmark evaluates pretrained detectors, fine-tuned models, and zero-shot vision-language models under generator and post-processing shifts. On the hardest processed split, the best pretrained conventional detector achieves 0.550 ROC-AUC, compared with 0.752 for the best REALIS-trained detector. REALIS provides a unified framework for measuring and improving the reliability of AI-image detectors under conditions that better reflect real-world use.

Abstract

Adversarial Robustness

Explore similar work

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

RA-Det: Towards Universal Detection of AI-Generated Images via Robustness Asymmetry

SPARED: Reasoning-Based AI-Generated Image Detection via Adversarially Edited Data