We built an ordinary stacking ensemble for AI-generated image detection -- three open detectors producing five scores, fused by a gradient-boosted meta-learner that treats a detector's failure as missing data -- deployed it, and then evaluated it against three controls it should have faced first. This paper reports what the controls found, including where they overturned our own earlier conclusions. Fusion is worth its cost only when refitted on the target domain. The shipped meta-learner, fitted on a separate corpus, does not beat its best single member on 2000 StyleGAN faces (AUC 0.9896 vs 0.9961; McNemar p = 1.000). But a stacker refitted in-domain beats that member plus a post-hoc calibrator (Delta-AUC = +0.0025, [+0.0014, +0.0038]; p = 3.4e-4). An earlier draft claimed the calibrated single detector won outright; that comparison mixed regimes and we correct it here. One corpus is not an evaluation. On 80 screenshots every model's AUC interval contains 0.5. We can say nothing stronger: the difference between the ensemble's drop and its best member's is [-0.185, +0.115]. Abstention is real, correlated, and mishandled. With four of five detectors silent and the survivor reporting "real", the system returns P(AI) = 0.9985, because an all-NaN input scores 0.9995 in a learner never fitted with missingness. The three AIDE checkpoints fail together, sharing one preprocessing path. A quorum rule requiring two distinct architectures prevents both failures with no retraining. We also find Corpus A carries a class-conditional JPEG bias severe enough to separate the classes from the header alone, which limits every in-domain number we report. Code, harness, hash-identified artifacts and all corrections are released.
Figures & tables
Detector
Basis
Scores
Swin-B [ 16 ]
semantic classifier
VQGAN
Nonescape
frozen DINOv2 [ 19 ]
NONESCAPE
+ EfficientNetV2 attention
AIDE [ 23 ]
SRM residuals [ 6 ]
SD14 , PROGAN ,
over DCT-selected patches
GENIMAGE
Table 1: The three third-party detectors and the five scores they produce. None is our work; we integrate and evaluate them.
Model
Acc
Prec
Rec
F1
AUC
AP
t⋆
Acc ⋆
AIDE-SD14
0.7455
0.9168
0.5400
0.6797
0.8821
0.8842
0.154
0.7910
VQGAN/Swin
0.5695
0.6304
0.3360
0.4384
0.6189
0.6089
0.347
0.5895
AIDE-PROGAN
0.9470
0.9923
0.9010
0.9444
0.9961
0.9962
0.322
0.9700
AIDE-GENIMAGE
0.7420
0.9764
0.4960
0.6578
0.9208
0.9254
0.059
0.8535
NONESCAPE
0.7960
0.7139
0.9880
0.8289
0.9567
0.9520
0.881
0.8965
Quorum (shipped)
0.9470
0.9357
0.9600
0.9477
0.9896
0.9899
0.555
0.9515
Table 2: Corpus A ( 2000 StyleGAN faces), zero-shot. t⋆ and Acc ⋆ are whole-corpus (oracle) Youden fits , retained only to expose the ranking/calibration gap; they are optimistically biased and are not the basis of any claim. The unbiased nested figures are in § 6 below. No detector abstained.
Figure 1: Corpus A ROC. AIDE-PROGAN (0.9961) sits above the fused curve (0.9896).
Figure 2: Corpus A reliability, including the single detector raw and Platt-scaled.
Model
Regime
AUC
Acc
ECE [95% CI]
AIDE-PROGAN (raw)
zero-shot
0.9961
0.9470
0.1272 [0.1195, 0.1349]
Quorum (shipped)
zero-shot
0.9896
0.9470
0.0416 [0.0348, 0.0512]
AIDE-PROGAN + Platt
in-domain
0.9960
0.9660
0.0319 [0.0272, 0.0392]
AIDE-PROGAN + isotonic
in-domain
0.9946
0.9665
0.0083 [0.0074, 0.0175]
XGBoost (refit)
in-domain
0.9985
0.9805
0.0096 [0.0063, 0.0163]
Table 3: Corpus A. Calibrators are fitted strictly out-of-fold on Corpus A , so they are in-domain models and belong in the lower block. ECE intervals are bootstrap over 2000 image resamples.
Model
A (AUC)
B (AUC)
B 95% CI
AIDE-SD14
0.8821
0.6293
[0.4993, 0.7393]
VQGAN/Swin
0.6189
0.4817
[0.3462, 0.6119]
AIDE-PROGAN
0.9961
0.4563
[0.3311, 0.5857]
AIDE-GENIMAGE
0.9208
0.4227
[0.2894, 0.5584]
NONESCAPE
0.9567
0.4620
[0.3289, 0.5958]
Quorum
0.9896
0.4107
[0.2730, 0.5474]
Table 4: The same fixed models on both corpora. Corpus B intervals are percentile bootstrap over 2000 resamples. Every Corpus B interval contains 0.5 .
real (FFHQ)
AI (StyleGAN)
median quant.-table sum
1483
375
median bytes/pixel
0.2399
0.4253
distinct quant. tables
1
1
Table 5: Corpus A JPEG encoding by class, first 250 images each. Lower quantisation sums mean higher quality. Each class carries exactly one distinct quantisation table.
Input
Outcome
Abst.
P(AI)
baseline RGB
scored
0/5
0.0313
1×1 px
scored, degraded
4/5
0.9985
4096×3
scored, degraded
3/5
0.9991
prose as .jpg
all detectors failed
5/5
—
zero bytes
all detectors failed
5/5
—
corrupt CRC
all detectors failed
5/5
—
Table 6: Induced failures. 14 constructed inputs; the table lists the outcomes that were not "scored, nothing abstained".
Case
shipped P
live architectures
rule output
1×1 px
0.9985
nonescape
abstain
4096×3
0.9991
nonescape, swin
0.2535
Table 7: Supervisory rules applied to the two partial-failure cases. Quorum: abstain unless at least two distinct architectures report. Survivor-mean: average the live scores instead of consulting the learner’s defaults.
Substitute
mean P(AI)
∣ dist. from prior ∣
all-zero
0.0006
0.4994
all-column-mean
0.5990
0.0990
all- NaN
0.9995
0.4995
Table 8: Corpus A prior is 0.5000 ; a policy meaning "no evidence" should land near it.
Rate
Policy
AUC
Acc
mean P(AI)
0%
any
0.9985
0.9805
0.4994
10%
native (NaN)
0.9881
0.9375
0.5462
10%
zero-imputed
0.9824
0.9325
0.4472
10%
mean-imputed
0.9953
0.9605
0.5120
25%
native (NaN)
0.9580
0.8625
0.6226
25%
zero-imputed
0.9511
0.8390
0.3497
Table 9: Corpus A. Fixed model trained on complete scores; abstentions injected MCAR at inference.
Rate
Policy
AUC
Acc
0%
any
0.9985
0.9805
10%
native (NaN)
0.9972
0.9720
10%
zero-imputed
0.9970
0.9755
10%
mean-imputed
0.9974
0.9710
25%
native (NaN)
0.9947
0.9575
25%
zero-imputed
0.9936
0.9595
Table 10: Corpus A. Model refitted under each policy at each rate — the like-for-like comparison. Table 9 characterises the shipped artifact; this characterises the design.
Figure 3: Corpus A score distributions per detector, real vs AI, with the t=0.5 line. AIDE-SD14 and AIDE-GENIMAGE compress both classes below the line — the “sink class” effect — so they appear to fail at 0.5 while still separating the classes.
Rule
Regime
AUC
95% CI
Acc
95% CI
ECE
Best single (AIDE-PROGAN)
untrained
0.9961
[0.9944, 0.9974]
0.9470
[0.9370, 0.9565]
0.1272
Max fusion
untrained
0.9628
[0.9548, 0.9702]
0.7185
[0.6985, 0.7375]
0.2589
Mean fusion
untrained
0.9913
[0.9886, 0.9938]
0.8710
[0.8570, 0.8855]
0.2279
Majority vote
untrained
0.9685
[0.9626, 0.9742]
0.8550
[0.8395, 0.8695]
0.1544
Quorum (shipped)
zero-shot
0.9896
[0.9861, 0.9925]
0.9470
[0.9365, 0.9565]
0.0416
Logistic regression
in-domain CV
0.9989
[0.9982, 0.9994]
0.9815
[0.9755, 0.9870]
0.0112
Table 11: Corpus A fusion rules. Regimes are never ranked against each other. Brackets are 95% bootstrap CIs over 2000 resamples.
Removed
AUC
Δ AUC
none (all 5)
0.9985
—
AIDE-SD14
0.9985
−0.0001
VQGAN/Swin
0.9983
−0.0003
AIDE-GENIMAGE
0.9985
−0.0000
NONESCAPE
0.9962
−0.0023
AIDE-PROGAN
0.9879
−0.0106
Table 12: Leave-one-detector-out, Corpus A, in-domain 5-fold CV.
Detector
Dev.
Load
Single
Batch/img
VQGAN/Swin
MPS
2.6
0.029
0.015
Nonescape
MPS
3.9
0.057
0.032
AIDE × 3
CPU
23.8
9.178
4.244
Table 13: Per-detector cost in seconds, Apple M4 Max. Medians over 3 rounds ( n=24 timed single images, 3 batch runs of 16). AIDE runs on CPU following the S1 device probe.
Model
FPR
50%
10%
1%
0.1%
AIDE-PROGAN
0.0070
0.9923
0.9346
0.5652
0.1141
Quorum
0.0660
0.9357
0.6178
0.1281
0.0144
XGB (refit)
0.0200
0.9800
0.8450
0.3313
0.0468
Table 14: Corpus A precision at deployment prevalence (columns), from measured TPR/FPR at t=0.5 .
Training-free detectors of AI-generated images promise generator-agnostic deployment without classifier training, yet their reported numbers are rarely compared under a single controlled protocol. We audit two representative training-free scores -- an autoencoder-reconstruction score (AEROBLADE-style) and a noise-perturbation feature-similarity score (RIGID-style) -- plus a naive feature-kNN control, on a common 1,500-image GenImage-derived benchmark spanning seven generators and JPEG compression at quality 70 and 50. The audit yields three cautionary findings. (i) Implementation details masquerade as method differences: replacing the LPIPS backbone (AlexNet -> VGG-16) changes overall AUROC by +0.085, and switching between resize-to-512 and native-resolution preprocessing flips per-generator conclusions by up to 0.38 AUROC. (ii) Score direction is not a property of the method but of its hyperparameters: the RIGID-style score is inverted (AUROC < 0.5) on SD1.5 and Wukong at noise level sigma=0.05, recovers to >0.5 for every generator at sigma=0.01, and collapses to 0.15 at sigma=0.3. (iii) Dataset format bias inflates robustness claims: without unified re-encoding, AUROC under JPEG-50 exceeds the clean condition for the AlexNet-backbone reconstruction score; after bias correction the residual anomaly localizes to a single generator (BigGAN). The audited scores have complementary per-generator failure sets, but naive z-score fusion does not beat the best single score, indicating that exploiting complementarity requires direction-aware combination.
Detecting AI-generated images is only half the task: a deployed detector must also justify its verdict, yet existing detectors inherit three failure modes from their training data: real and fake images collected from different sources invite provenance shortcuts, supervised explanation corpora teach templated rationales, and a static forgery corpus leaves the decision boundary standing still while generators keep moving. We introduce \methodname{}, an adversarial reinforcement learning framework that pits two heterogeneous models against each other. A diffusion image editor learns to edit real photographs into fake counterparts of those same photographs that fool the current detector, while a reasoning MLLM learns to expose them with a verdict grounded in free-form reasoning. Both rewards are shortcut-proof by design: the attacker is credited only when its edit is faithfully executed, and the defender only when its verdict is correct. As the two models alternate, each round's attacker regenerates a harder training pool aimed at the current detector's blind spots, so the detector must generalize rather than memorize any fixed artifact distribution. Although the explanation is never rewarded, its quality rises round over round as a side effect of accuracy-only training. A detector trained within this loop improves monotonically across rounds on each of three external benchmarks.
Yicheng Bao, Xiahui Guo, Xuhong Wang +1
1East China Normal University · 2Shanghai Artificial Intelligence Laboratory
Given the surge of harmful AI-generated imagery online, reliably distinguishing authentic images from generated ones has become an urgent research topic. While many proposed detection methods perform well under controlled settings, they often collapse when tested on real-world data. A potential root cause are subtle biases in the detectors' training data. As a result, detectors may rely on spurious correlations instead of learning true forensic artifacts. While a recent line of work has identified the problem, there is not yet an established protocol to evaluate how biased a detector actually is. In this work, we therefore take a step back: First, we discuss what it means for a detector to be biased, and how this differs from a lack of robustness. Second, we propose BIAS-ID, a transparent framework for analyzing and quantifying the presence of transformation biases in AI-generated image detectors. We validate our framework by performing an evaluation of six detectors across two datasets, revealing that several state-of-the-art detection methods are strongly affected by biases. Our results highlight the importance of bias-aware evaluation for developing reliable AI-generated image detectors.
Jonas Ricker, Asja Fischer, Erwin Quiring
Ruhr University Bochum · Bochum, Germany · _fbeta +1