Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP

Ahmed Abdelnaby; Mohamed Elmahallawy

cs.CVSep 25, 2026

Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP

Authors: Ahmed Abdelnaby, Mohamed Elmahallawy

Organizations: Washington State University, Richland, WA 99354, USA

Abstract

Contrastive Language--Image Pretraining (CLIP) has emerged as a dominant vision backbone due to its strong transferability and zero-shot capabilities. However, recent studies reveal a critical vulnerability: embedding-space backdoor attacks. By poisoning only a tiny fraction of image--text pairs, adversaries can implant stealthy triggers that induce targeted shifts in CLIP's joint embedding space. Unlike conventional backdoors that manipulate classifier logits, these attacks corrupt representations directly, making them highly effective under extremely low poisoning ratios and difficult to detect. Existing defenses require access to model parameters, gradients, logits, or clean validation data---assumptions that rarely hold in realistic black-box deployments. Moreover, current black-box methods struggle to accurately localize small or out-of-distribution triggers. We propose CLIPGuard, a lightweight and fully black-box defense specifically designed to mitigate embedding-space backdoors in CLIP encoders. CLIPGuard identifies malicious regions by measuring segment-wise embedding perturbations and selectively purifies only suspicious segments via semantic inpainting, preserving benign visual content and alignment quality. Extensive experiments on STL-10, ImageNet, and diverse trigger families---including BadCLIP, BadNets, blended, patch-based, and typographic attacks---demonstrate that CLIPGuard reduces attack success rates to as low as 1.05% while maintaining clean accuracy up to 86.34%, consistently outperforming existing black-box defenses, including CleanCLIP and CleanerCLIP. Our code is available https://github.com/wsu-cyber-security-lab-ai/CLIPGuard.git

Figures & tables

Figure 1: Overview of the CLIPGuard framework. The poisoned input image is partitioned into grid cells and perturbed to compute per-cell embedding shifts $\Delta_{i}=\lVert e_{i}-e_{0}\rVert_{2}$ , forming a global sensitivity heatmap $H$ . High-sensitivity regions undergo targeted segmentation, isolation, and multi-scale inpainting. Reconstructed image candidates are evaluated via contrastive similarity reduction, and the best-scoring image is returned as the final purified image .

Dataset	Attack	No Def.		CleanCLIP		CleanerCLIP		PAR		ABD		CLIPGuard
Dataset	Attack	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$
STL-10	BadNet	91.13	98.18	81.13	16.94	87.45	18.45	81.30	25.30	76.87	13.67	86.34	2.90
	Blended	90.56	97.34	79.15	31.23	85.87	22.67	73.60	18.39	83.17	23.09	84.32	6.19
	Physical	89.34	99.45	86.39	72.78	82.01	30.87	73.67	57.56	80.09	21.97	83.09	2.73
	LC	89.14	98.67	80.87	78.98	79.67	24.78	77.06	41.34	74.16	35.18	82.03	1.05
	Avg.	90.04	98.41	80.29	50.48	83.35	24.19	76.41	35.65	78.57	23.48	83.95	3.22

Table 1: Defense comparison under traditional backdoor attacks (BadNet, Blended, Physical, LC). Higher CA (

\uparrow

) and lower ASR (

\downarrow

) indicate better performance. Best results are highlighted.

Dataset	Attack	No Def.		CleanCLIP		CleanerCLIP		PAR		ABD		CLIPGuard
Dataset	Attack	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$
STL-10	BadCLIP	88.60	98.23	81.56	53.35	81.14	19.87	81.56	30.40	73.77	27.23	81.67	3.34
	SCAM	89.34	95.34	79.34	69.24	80.23	65.13	80.94	54.27	81.34	63.56	85.97	4.98
	TrojVQA	88.68	96.34	80.09	90.91	83.65	15.65	78.32	51.11	79.47	39.56	82.56	2.16
	Avg.	88.87	96.64	80.33	71.17	81.67	33.55	80.27	45.26	78.19	43.45	83.40	3.49
ImageNet	BadCLIP	58.60	98.80	53.80	61.10	52.61	19.87	53.40	30.40	52.47	49.56	55.34	2.13

Table 2: Defense comparison under modern backdoor attacks (BadCLIP, SCAM, TrojVQA). Higher CA (

\uparrow

) and lower ASR (

\downarrow

) indicate better performance. Best results are highlighted.

Mode	Attack	No Def.		CleanCLIP		CleanerCLIP		PAR		Ours
Mode	Attack	CA	ASR	CA	ASR	CA	ASR	CA	ASR	CA	ASR
All2One	BadNets	61.56	93.18	56.54	12.12	57.17	7.15	53.30	11.60	57.28	4.13
	Blended	57.70	99.40	53.44	19.50	52.61	12.84	53.60	2.13	57.62	5.92
	Physical	57.60	99.80	53.30	62.30	54.18	32.69	53.00	42.40	58.37	4.56
	Avg.	58.95	97.46	54.43	31.31	54.65	17.56	53.30	18.71	57.76	4.87
All2All	BadNets	56.23	85.17	52.81	18.74	53.92	10.61	51.47	14.22	55.91	5.37

Table 3: Defense performance comparison on the ImageNet subset under different attack modes (All2One, All2All, and Untargeted).

Poison Ratio	No Def.	CleanCLIP	CleanerCLIP	PAR	ABD	Ours
5%	90.67	31.34	24.23	35.16	11.50	3.15
10%	99.80	62.30	32.69	42.40	19.30	4.56
15%	99.83	63.12	32.74	47.87	19.56	4.79
20%	99.90	64.67	35.56	46.17	21.45	5.07

Table 4: Defense performance comparison in terms of Attack Success Rate (ASR, %) under varying poison ratios for the physical patch attack on ImageNet. Lower ASR values indicate improved robustness.

Cell Size	STL-10			ImageNet
Cell Size	Defense Time (ms)	Acc (%)	ASR (%)	Defense Time (s)	Acc (%)	ASR (%)
No Defense	–	89.34	95.34	–	56.90	95.60
$4\times 4$	90	86.13	8.76	0.54	54.09	10.87
$8\times 8$	160	85.97	4.98	0.95	53.56	3.14
$16\times 16$	180	85.67	5.10	1.62	53.98	3.13
$24\times 24$	230	85.81	5.07	4.05	53.45	3.68

Table 5: Impact of grid cell size on defense time, Acc, and ASR for STL-10 and ImageNet. Best results are highlighted in green; worst results in light red.

Backdoor Defense

Backdoor defense develops methods to detect, remove, or contain hidden trigger-conditioned behaviors in machine-learning models.

See all 50 papers

Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors
Jun 16, 2026Kunlan Xiang, Haomiao Yang, Wenbo JiangVLM Robustness Adversarial Attacks on VLMs
University of Electronic Science and Technology of China
CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion
Apr 10, 2026Akshit Jindal, Saket Anand, Chetan Arora +1Backdoor Attacks on VLMs
IIIT Delhi · IIT Delhi
Backdoor as Probe: Test-Time Adversarial Defense for CLIP
Sep 28, 2026Zhongqi Wang, Jie Zhang, Nie Sen +3VLM Robustness Backdoor Defense
Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Xuzhou University of Technology, China

Jun 16, 2026J/K move · Enter open · S save

Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors

Kunlan Xiang, Haomiao Yang, Wenbo Jiang

University of Electronic Science and Technology of China

VLM Robustness Adversarial Attacks on VLMs Backdoor Attacks Backdoor Attacks on VLMs Vision-Language Pretraining

Open paper AskPDF arXiv

Contrastive Language-Image Pre-training models are widely reused across downstream interfaces, including feature extraction, retrieval, reranking, and selection. Existing CLIP backdoor, however, usually validate attacks on a small attack-native task, leaving unclear whether the same poisoned checkpoint remains exposed, weakens, or becomes not applicable when reused through other interfaces. We introduce DIFE, a Deployment-Interface Footprint Evaluation framework that audits backdoored CLIP checkpoints across deployment interfaces. DIFE makes various evaluations comparable by specifying each interface's component readout, trigger channel, target event, reference condition, and metric. DIFE also introduces effective-footprint diagnosis to identify the reusable CLIP component or component combination that carries exposure and explains where risk transfers. Auditing reproduced CLIP backdoors with DIFE reveals a structured landscape: native success is not a checkpoint-level risk certificate, exposure follows component footprints, text-side poisoning does not yield textual-encoder control, and some coupled attacks remain mechanism-bound. This audit reveals a import gapin existing CLIP backdoors: a textual encoder that itself becomes a reusable carrier of adversarial behavior. We therefore introduce BadTextTower to fill this gap. BadTextTower produces strong text-conditioned retrieval, reranking, and selection exposure while leaving visual-only reuse nearly clean.

Abstract

Backdoor Defense

Explore similar work

Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors

CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion

Backdoor as Probe: Test-Time Adversarial Defense for CLIP