SemPurify: Semantics-Aware Data Purification for Black-box Backdoor Defense

Jiawei Yu; Syed Irfan Ali Meerza; Yi Wu; Amir Sadovnik; Jian Liu

cs.CVOct 16, 2026

SemPurify: Semantics-Aware Data Purification for Black-box Backdoor Defense

Authors: Jiawei Yu, Syed Irfan Ali Meerza, Yi Wu, Amir Sadovnik, Jian Liu

Organizations: University of Georgia · Virginia Commonwealth University · University of Oklahoma · Oak Ridge National Laboratory

Abstract

Backdoor attacks have emerged as a critical security threat in machine learning, where adversaries embed triggers within poisoned training samples to manipulate a model's predictions on targeted inputs while leaving its general behavior unaffected. This stealthy nature makes backdoor attacks difficult to detect, as the model functions normally until activated by a specific malicious trigger. Current purification-based defenses use image transformations (e.g., blurring) to disrupt trigger patterns but often degrade image quality. Even with advanced restoration techniques like diffusion models, these approaches can still alter visual or semantic information, reducing utility for downstream tasks. To address this, we propose the first semantics-aware data purification framework for black-box backdoor defense, SemPurify, which explicitly incorporates multi-level semantics (i.e., low-level semantic embeddings and high-level image tags) within a text-to-image (T2I) framework to guide image restoration during purification. Using this semantic information, the T2I model subsequently refines the transformed samples, helping preserve visual details and semantic information during restoration. Extensive experiments spanning multiple backdoor scenarios, i.e., classification models, CLIP-style encoders, and generative models, demonstrate that SemPurify reliably removes backdoors while effectively preserving semantic information for downstream tasks.

Figures & tables

Figure 1: Illustration of semantic descriptions extracted by the tagging model RAM [ 60 ] from clean and backdoored inputs. BadNet [ 8 ] is applied to cause the image labeled as “church” to be misclassified as “dog”. While two tags (highlighted) are introduced, they remain contextually related to the original image, with none associated with the target label “dog”.

Figure 2: The overview of SemPurify: In Stage 1, we use an image tagging model to extract multi-level semantics. In Stage 2, we use a linear transformation such as blurring to destruct the trigger patterns in the poisoned image $x_{p}$ . In Stage 3, we use a conditional text-to-image model to restore the semantic information. This process ensures that $x^{0}$ retains the original semantic information from $x_{p}$ while effectively neutralizing any backdoor trigger patterns.

		No Defense			ShrinkPad			BDMAE			ZIP			DataElixir			SampDetox			SemPurify
Dataset	Attack	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$
	No Attack	88.48	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
	BadNet	88.41	100.00	10.00	57.99	2.85	57.85	87.00	0.66	86.17	82.55	0.90	82.13	84.40	0.96	71.30	77.54	1.04	76.11	86.66	0.90	86.18
	PhysicalBA	94.59	100.00	10.00	91.45	91.14	17.50	94.38	16.14	77.92	87.00	0.80	86.71	94.57	1.02	72.93	81.48	1.06	79.92	90.80	0.11	90.76
	Blended	87.96	100.00	10.00	59.14	0.56	36.85	86.69	30.51	30.11	72.65	1.60	58.14	88.03	0.97	64.84	77.19	0.72	69.17	86.47	1.17	82.84
	Semantic Blended	88.30	99.62	10.31	59.51	1.57	51.72	86.96	97.00	2.80	77.33	1.23	75.82	88.30	0.89	68.14	77.55	1.19	72.79	84.41	2.37	83.16

Table 1: Defense performance comparison against classification backdoor attacks.

Figure 3: Qualitative comparisons of images produced by different purification methods.

Dataset	Metric	No Defense	ShrinkPad	BDMAE	ZIP	DataElixir	SampDetox	SemPurify
Caltech101 [ 5 ]	CA $\uparrow$	92.00	91.70	86.24	90.70	90.39	90.61	91.30
	ASR $\downarrow$	98.30	0.90	35.59	1.10	0.33	0.33	0.20
	PA $\uparrow$	2.60	91.00	57.97	90.70	91.27	91.16	91.30
OxfordPets [ 36 ]	CA $\uparrow$	96.00	93.40	86.80	90.20	95.02	89.26	93.40
	ASR $\downarrow$	99.30	47.90	49.33	5.70	6.26	4.08	3.20
	PA $\uparrow$	5.50	51.30	48.99	90.40	91.44	92.11	93.30

Table 3: Defense performance comparison against BadCLIP.

No Defense		BDMAE		ZIP		DataElixir		SampDetox		SemPurify
CS $\uparrow$	PS $\downarrow$	CS $\uparrow$	PS $\downarrow$	CS $\uparrow$	PS $\downarrow$	CS $\uparrow$	PS $\downarrow$	CS $\uparrow$	PS $\downarrow$	CS $\uparrow$	PS $\downarrow$
0.89	0.86	0.54	0.24	0.69	0.38	0.61	0.21	0.66	0.27	0.78	0.20

Table 4: Defense performance comparison against generative backdoor attacks.

Figure 4: Comparison of the generated image using different purification methods.

	w/ $S_{e}$			w/ $S_{t}$			w/ $S_{e}+S_{t}$
Attack	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$	CA $\uparrow$	ASR $\downarrow$	PA $\uparrow$
BadNet	83.21	1.07	83.36	68.84	0.16	69.22	83.18	0.99	83.18
PhysicalBA	89.63	7.36	86.85	77.50	3.6	77.09	89.85	7.33	87.05
Blended	84.02	0.31	81.98	70.92	0.02	70.77	84.38	0.28	82.64
Semantic Blended	73.42	6.96	72.58	56.78	0.31	60.22	79.36	6.11	77.65
WaNet	82.39	1.80	82.70	65.75	0.39	65.37	76.94	1.55	76.91

Table 5: Impact of semantic embeddings and image tags.

Metric	ShrinkPad	ZIP	BDMAE	DataElixir	SampDetox	SemPurify
Latency (Second) $\downarrow$	0.0047	1.0441	0.2085	15.2913	12.2146	0.5101
Peak GPU Mem (MB) $\downarrow$	100.7	1579.3	451.5	1632.1	1737.1	5764.6

Table 6: Computational cost on Imagenette.

Backdoor Defense

Backdoor defense develops methods to detect, remove, or contain hidden trigger-conditioned behaviors in machine-learning models.

See all 50 papers

Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models
Feb 24, 2026Guangnian Wan, Qi Li, Gongfan Fang +2Diffusion Language Models Multimodal Diffusion Models
National University of Singapore
RePAIR: Inverting Backdoor Attacks in Semantic Segmentation
Oct 27, 2026Arnold Brosch, Kira MaagBackdoor Defense Semantic Segmentation
Heinrich Heine University Düsseldorf
Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse
Oct 15, 2026Junhoo Lee, Baekseung Kim, Seungyeon Kim +1Backdoor Attacks Backdoor Attacks on VLMs
Seoul National University · Now at KAIST.

Feb 24, 2026J/K move · Enter open · S save

Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models

Guangnian Wan, Qi Li, Gongfan Fang, Xinyin Ma+1

National University of Singapore

Diffusion Language Models Multimodal Diffusion Models

Open paper AskPDF arXiv

Multimodal Diffusion Language Models (MDLMs) have recently emerged as a competitive alternative to their autoregressive counterparts. Yet their vulnerability to backdoor attacks remains largely unexplored. In this work, we show that well-established data-poisoning pipelines can successfully implant backdoors into MDLMs, enabling attackers to manipulate model behavior via specific triggers while maintaining normal performance on clean inputs. However, defense strategies effective to these models are yet to emerge. To bridge this gap, we introduce a backdoor defense framework for MDLMs named DiSP (Diffusion Self-Purification). DiSP is driven by a key observation: selectively masking certain vision tokens at inference time can neutralize a backdoored model's trigger-induced behaviors and restore normal functionality. Building on this, we purify the poisoned dataset using the compromised model itself, then fine-tune the model on the purified data to recover it to a clean one. Given such a specific design, DiSP can remove backdoors without requiring any auxiliary models or clean reference data. Extensive experiments demonstrate that our approach effectively mitigates backdoor effects, reducing the attack success rate (ASR) from over 90% to typically under 5%, while maintaining model performance on benign tasks.

Abstract

Backdoor Defense

Explore similar work

Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models

RePAIR: Inverting Backdoor Attacks in Semantic Segmentation

Clean Data Can Still Carry Backdoors: Support-Persistent Backdoors for Model Reuse