Space Over Statistic, Retraining Over Ranking: Auditing Training Labels Under Label-Only Backdoors

Arnab Dey; Mohammad Mahfuzul Islam

cs.CROct 9, 2026

Space Over Statistic, Retraining Over Ranking: Auditing Training Labels Under Label-Only Backdoors

Authors: Arnab Dey, Mohammad Mahfuzul Islam

Organizations: Department of Computer Science and Engineering Bangladesh University of Engineering and Technology Dhaka, Bangladesh

Abstract

Some backdoor attacks never touch an image. FLIP only changes labels: on CIFAR-10, flipping 3% of them makes a ResNet-32 send 99.7% of triggered source-class images to the attacker's class. The usual defenses look for the poison inside a model trained on those same labels. So when they struggle, is the problem the test they run, or the space they run it in? We answer with a controlled swap: we keep the test, the labels and the metric fixed, change only the feature space, and judge each defense by retraining the victim on what it keeps. The space decides. On dog $\to$ cat the same nearest-neighbor check scores AUROC 0.546 on the poisoned model's features, 0.967 on frozen DINOv2 features that never saw a label, and 0.995 on the same network trained on clean labels, an oracle control rather than a defense. We reproduce SPECTRE, the defense FLIP's authors recommend, to 90.94 / 0.97 against its published 91.47 / 1.7, then price it: on truck $\to$ deer it deletes 20% of every class, while a prototype audit on the frozen features deletes 6% for higher clean accuracy and lower attack success. On three harder pairs that audit leaves about a quarter of the attack standing. A count read off encoder geometry and clean labels, before any attack is run, separates the pair where the audit removes the attack from those where it only partly does, though on a single zero-count pair. Three results warn against the easy story. A better detection score does not mean a safer model: on two of four pairs Confident Learning scores the higher AUROC, and on dog $\to$ cat leaves 2.6 times the attack success. The prototype family, our best static performer, is the first an adaptive attacker slips past. And past a point, catching more poisons stops reducing attack success. Check labels in a space that never learned them, and judge a defense by the retrained model, not by its detection score.

Figures & tables

Fig. 1: The pipeline, and where each claim acts. FLIP corrupts labels only, so the images reaching the defender are the ones that were collected. An audit then has three knobs: the space features live in, the statistic computed on them, and the rule turning scores into deletions. We turn one at a time, always retraining the victim rather than stopping at a ranking. The clean-label space ( ∗ ) is a control, not a defense: it holds the architecture fixed and changes only the labels the features were fitted to, separating “supervised features are poor here” from “features fitted to the poison are poor here”. Two dashed arrows leave the main flow: the defense-aware attacker feeds back into the attack, and the stealthy-flip pool reads the encoder and clean labels before any attack exists. Each claim sits under the knob it concerns. The two statistic families at knob 4 are defined in ( 4 ) and ( 5 ). Claim 1 is Section V , Claim 2 and the negative result are Section VI , and Claim 3 is Section VII . AUROC values are knn_disagreement on dog

\rightarrow

cat (Section V ).

Fig. 2: The two triggers, applied by the same code the experiments use. The sinusoidal trigger perturbs almost the whole image faintly (951 of 1 024 pixels, peak change 6/255); the Turner patch changes 9 pixels hard (peak 199/255).

setting	source	CTA	PTA
clean labels ( $m=0$ )	published	92.38	0.1
clean labels ( $m=0$ )	ours	92.57	0.1
sinusoidal, $m=1500$	published	89.87	99.8
sinusoidal, $m=1500$	ours	$89.47\pm 0.35$	$99.73\pm 0.15$
Turner, $m=1500$	published	89.91	99.9
Turner, $m=1500$	ours	89.94	32.4

TABLE I: Reproducing FLIP on our setup. Published values are FLIP’s Table 1, CIFAR-10 with a ResNet-32. Ours are the same configuration run here; the sinusoidal row is three seeds, the others are single runs. The clean and sinusoidal rows reproduce. The Turner row does not, and Section IV says why.

	supervised		frozen DINOv2
detector	poisoned	clean	ViT-S/14	ViT-B/14
knn_disagreement	.26/.05	.97/.86	.61/.35	.74/.52
knn_sim_margin	.25/.05	.97/.88	.63/.42	.75/.59
prototype_margin	.22/.04	1.0/1.0	.97/.67	.99/.85
prototype_trimmed	.27/.04	1.0/1.0	.98/.73	.99/.87

TABLE II: AUPRC of every detector in every feature space, reported as truck

\rightarrow

deer / dog

\rightarrow

cat. Same labels, same detectors, same metric; only the space changes. Reading across a row isolates the effect of the space. The clean column is an oracle control, not a deployable defense.

truck $\rightarrow$ deer
defense	cut	CTA	PTA
none	0	$89.47\pm 0.35$	$99.73\pm 0.15$
SPECTRE, $Q{=}0.20$	10 000	$90.94\pm 0.23$	$0.97\pm 0.46$
prototype_trimmed	2 981	$\mathbf{92.23\pm 0.14}$	$\mathbf{0.20\pm 0.20}$
dog $\rightarrow$ cat
none	0	$89.33\pm 0.30$	$94.67\pm 0.42$

TABLE III: Victim-level comparison, mean

\pm

sd over three seeds. “cut” is how many of the 50 000 training images the defense deletes. SPECTRE is given the per-class budget it is designed for (Section III ); FLIP report it at CTA 91.47 / PTA 1.7 on truck

\rightarrow

deer [ 3 ] .

truck $\rightarrow$ deer
defense	removed	clean destroyed
SPECTRE, $Q{=}0.20$ [ 8 ]	10 000	8 555
activation clustering, $Q{=}0.20$ [ 6 ]	10 000	8 630
spectral signatures, $Q{=}0.25$ [ 5 ]	12 500	11 145
prototype_trimmed	1 419	$\mathbf{69}$
dog $\rightarrow$ cat

TABLE IV: What each published defense deletes to catch 90% of the poisons, deployed per class as all three are designed to be. Detection only, no retraining, so these sit alongside Figure 3 .

Fig. 3: Protection against its price. At 90% recall on truck

\rightarrow

deer a prototype audit on frozen features loses 69 clean images; SPECTRE, deployed per class as designed, loses 8 555.

truck $\rightarrow$ deer
detector	AUROC	AUPRC	CTA	PTA
knn_disagreement	.986	.614	91.99	8.6
knn_sim_margin	.987	.630	92.04	11.4 $\downarrow$
prototype_margin	.996	.968	92.34	0.4
prototype_trimmed	.996	.976	92.31	0.2
cleanlab [ 13 ]	.974	.584	91.48	51.3

TABLE V: Every detector at the same removal budget (2 981 and 2 728 images), so a difference in the model is about which samples were cut. Detection and outcome disagree repeatedly: the arrows mark pairs of rows where the better-ranking detector produces the worse model.

Fig. 4: Twenty victim trainings sweeping

k

on truck

\rightarrow

deer. Detection peaks at

k=200

while the trained model keeps improving past it. The lower panel is the reason the curves separate: at the auto threshold

k

also sets how much data is cut.

Fig. 5: The statistic that wins under a defense-unaware attacker is the one a defense-aware attacker dismantles first.

Fig. 6: Attack $\times$ defense, dog $\rightarrow$ cat, three seeds.

pair	pool	AUPRC	PTA none	PTA defended
truck $\rightarrow$ deer	0	.976	$99.7\pm 0.2$	$\mathbf{0.2\pm 0.2}$
car $\rightarrow$ truck	72	.789	$99.4\pm 0.3$	$24.6\pm 1.3$
cat $\rightarrow$ dog	82	.786	$94.4\pm 0.7$	$24.4\pm 2.7$
dog $\rightarrow$ cat	265	.727	$94.7\pm 0.4$	$24.3\pm 2.5$

TABLE VI: The pool against what happened. Pool sizes come from encoder geometry alone; AUPRC is prototype_trimmed ; PTA columns are mean

\pm

sd over three seeds at each pair’s matched budget. Detection tracks the pool; the trained outcome only separates zero from non-zero.

Fig. 7: What the pool means for the attacker. As the stealth budget

\tau

tightens, dog

\rightarrow

cat degrades smoothly while truck

\rightarrow

deer collapses as soon as any constraint applies: with a pool of zero there is no flip that both installs the backdoor and survives the audit.

paid once per dataset
stage	seconds	vs. training
DINOv2 ViT-S/14 pass (22.1 M)	277.6	8.2%
DINOv2 ViT-B/14 pass (86.6 M)	1 190.0	35.3%
$k$ -NN graph, $k=20$	36.4	1.1%
paid per audit
prototype_trimmed	0.09	$<0.01$ %

TABLE VII: Measured cost of each stage, against one victim training. Encoder timings are a single GPU run on a Tesla P6; the CPU rows and the victim median come from a separate run on the same cluster. The encoder is the only substantial new cost, and it is paid once per dataset.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 8: The AUPRC landscape over

k

for knn_disagreement , with what each oracle-free rule picks. The dashed line is the best AUPRC any

k

achieves, which a defender cannot find without the poison mask. Bimodality sits at

k=1

on both pairs and gives up most of the curve. Stability and agreement trade places between the pairs and between budgets. Markers are nudged horizontally where rules pick the same

k

; heights are exact.

Fig. 9: Recall of knn_disagreement under thresholds a defender can set without the poison mask. The dashed line marks 90% recall, the equal-protection cut used in Section V-A ; it is an oracle operating point, not a rule. Otsu is oracle-free and clears it on both pairs, but by deleting roughly twice as much; the trusted

n

rules assume a clean subset our threat model does not grant.

Fig. 10: The stealthy-flip pool across all 90 ordered CIFAR-10 pairs, ranked. The four pairs we trained victims for are highlighted. One pair dominates, three more form a cluster behind it, and the distribution then decays into a long tail of pairs with no stealthy flip available at all.

pair	budget	oracle $k$	rule	picks $k$	AUPRC gap
truck $\rightarrow$ deer	3%	200	stability	100	$+0.0006$
			bimodality	1	$+0.4190$
			agreement	50	$+0.0083$
	5%, 10%	200	stability	50	$+0.0083$
			bimodality	1	$+0.4190$
			agreement	2000	$+0.0167$

Appendix

TABLE VIII: Choosing

k

without an oracle, for knn_disagreement on frozen DINOv2 features. For each pair and audit budget

B

we give the

k

that maximises AUPRC (the oracle a defender cannot consult) and what each oracle-free rule picks instead, with the AUPRC it gives up. AUPRC ranks the whole score vector and so does not depend on

B

; the oracle column is therefore constant down a pair, and only the rules move with the budget. Stability takes the smallest

k

whose top-

B

selected set has Jaccard overlap at least 0.85 with the top-

B

set at

2k

, which is why it is undefined where

2k

is off the grid. Bimodality maximises Otsu separation of the score histogram. Agreement maximises top-

B

overlap with prototype_trimmed , so it is not independent of the prototype audit and we do not read it as evidence about prototypes.

pair	detector	rule	flagged	recall	$F_{1}$
truck $\rightarrow$ deer	knn_disag.	fixed_0.5	2981	91.3%	0.611
		otsu	5746	99.1%	0.410
		trusted1000_fpr0.05	3367	93.5%	0.576
		trusted1000_fpr0.02	1992	75.0%	0.644
	proto_margin	fixed_0.5	61	4.1%	0.078
		otsu	1405	88.3%	0.912

Appendix

TABLE IX: What a deployable threshold actually catches. fixed_0.5 and otsu are both oracle-free: the first is a fixed cut, the second thresholds the score histogram by maximising between-class variance and uses no labels or clean data of any kind. Only the trusted

n

rules assume a clean subset, of size

n

, at the stated false-positive rate; our threat model does not grant one (Section II ), so those rows are a check from outside it. Recall is of the true poison mask. The 90% recall used for the equal-protection comparison in Section V-A is an oracle operating point, not one of these.

pair	$\tau=0.5$	$\tau=0.7$	$\tau=0.9$
dog $\rightarrow$ cat $\star$	265	527	1164
truck $\rightarrow$ car	83	164	425
cat $\rightarrow$ dog $\star$	82	314	1223
car $\rightarrow$ truck $\star$	72	143	402
bird $\rightarrow$ plane	43	68	165
deer $\rightarrow$ frog	42	79	150

Appendix

TABLE X: Stealthy-flip pool for all 90 ordered CIFAR-10 pairs, computed on frozen DINOv2 features in seconds and with no training. The pool counts source-class images whose neighbourhood already leans to the target strongly enough that a flip survives the audit at

\tau

. The ten largest are shown; 29 of the 90 pairs have an empty pool at

\tau=0.5

. Pairs we trained victims for are marked

\star

Backdoor Detection

Backdoor detection finds hidden triggers or poisoned behavior in trained models before they can secretly redirect predictions or actions.

See all 53 papers

Jul 6, 2026

Statistical Adversaries: Natural Backdoor-like Features in Vision Datasets

Model-specific adversarial attacks have been extensively studied. We study a different failure mode: naturally occurring statistical signals in vision data that can behave like backdoor-like triggers without being maliciously inserted. We call these signals statistical adversaries. We analyse Imagenet to find patterns that are strongly linked to certain labels. We then use statistical controls to remove random correlations from our candidate signals. Finally, we demonstrate that these signals directly and predictably alter model predictions. These statistical adversaries are more targeted than generic corruptions and transfer across different model architectures. This suggests that some vulnerabilities are driven by dataset structure and distribution rather than a single model's idiosyncrasies. We conclude that ordinary datasets can contain exploitable adversarial surfaces even in the absence of poisoning, and suggest that dataset audits should treat spurious structure not only as a source of bias or interpretability failure, but also as a latent attack surface for vision models.

Paul K. Mandal, Pavan Reddy, Tristan Malatynski

1Neurint, LLC, Baton Rouge, LA, USA. · 2U.S. Army Cyber Corps, U.S. Army Reserve, USA. · 3Northwestern State University of Louisiana, Natchitoches, LA, USA. +2

Adversarial Examples Spurious Correlation Image Classification

Ask PDF

May 2, 2026

Checkerboard: A Simple, Effective, Efficient and Learning-free Clean Label Backdoor Attack with Low Poisoning Budget

Backdoor attacks threaten the deep learning supply chain by poisoning a small fraction of the training data so that a model behaves normally on clean inputs but misclassifies trigger-carrying inputs to an attacker-chosen target class. Clean-label backdoor attacks are especially dangerous because poisoned samples remain label-consistent and are therefore harder to detect. Yet existing clean-label attacks typically rely on expensive optimization, surrogate-model training, or nontrivial data access. We present Checkerboard, a theoretically grounded, learning-free clean-label backdoor attack that is effective, efficient, and simple to implement. From a linear separability formulation, we derive a checkerboard trigger in closed form, removing the need for surrogate-model training and trigger optimization. For texture-rich datasets, we introduce Complexity-driven Sample Selection, which uses only target-class data to improve trigger-to-background contrast by selecting low-complexity images for poisoning. Across four benchmark datasets, Checkerboard outperforms 8 baseline attacks and achieves state-of-the-art performance under low poisoning budgets. For example, on CIFAR-10, under a trigger perturbation budget of

10/255

, poisoning 20 training samples achieves

99.99\%

Attack Success Rate (ASR). On ImageNet-100, a poisoning rate of only

0.46\%

yields over

94\%

ASR without degrading clean accuracy. The proposed attack also remains effective against state-of-the-art backdoor defenses and shows strong resistance to adaptive defenses.

Yi Yang, Jinyang Huang, Binbin Liu +5

School of Computer Science and Information Engineering Hefei University of Technology Hefei, Anhui, China · School of Information Science and Technology University of Science and Technology of China Hefei, Anhui, China · Faculty of Business Data Science Kansai University Japan +3

Backdoor Attacks Data Poisoning Attacks

Ask PDF

Oct 15, 2026

Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse

Traditional backdoor attacks inject artificial trigger patterns into training data, causing models to misclassify triggered inputs while maintaining normal behavior on clean samples. Since these artificial triggers are absent in clean datasets, standard knowledge distillation (KD) typically eliminates them, leading to the common belief that KD serves as an effective purification defense. In this paper, we propose DAR (Discover-and-Relabel), a support-persistent backdoor method that retains its behavior even after clean-data distillation. DAR identifies naturally abundant patterns in low-dimensional feature subspaces using subspace clustering and applies label-only poisoning to samples that inherently satisfy the discovered rule. At test time, the backdoor is activated by locally modifying the corresponding subspace of the target images. By instantiating spatial and frequency operators, DAR achieves attack accuracy comparable to state-of-the-art backdoor attacks in ImageNet classification and CLIP-based prompt tuning. Notably, DAR keeps attack success above 96% after clean KD and remains defense-resistant in practice. Project page: https://junhoo.me/backdoor

Junhoo Lee, Baekseung Kim, Seungyeon Kim +1

Seoul National University · Now at KAIST.

Backdoor Attacks Backdoor Attacks on VLMs

Ask PDF

Abstract

Backdoor Detection

Explore similar work

Statistical Adversaries: Natural Backdoor-like Features in Vision Datasets

Checkerboard: A Simple, Effective, Efficient and Learning-free Clean Label Backdoor Attack with Low Poisoning Budget

Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse