CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion

Akshit Jindal; Saket Anand; Chetan Arora; Vikram Goyal

cs.CRApr 10, 2026

CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion

Authors: Akshit Jindal, Saket Anand, Chetan Arora, Vikram Goyal

Organizations: IIIT Delhi · IIT Delhi

Abstract

Organisations with limited data and computational resources increasingly outsource model training to Machine Learning as a Service (MLaaS) providers, who adapt vision-language models (VLMs) such as CLIP to downstream tasks via prompt tuning rather than training from scratch. This semi-honest setting creates a security risk where a malicious provider can follow the prompt-tuning protocol yet implant a backdoor, forcing triggered inputs to be classified into an attacker-chosen class, even for out-of-distribution (OOD) data. Such backdoors leave encoders untouched, making them undetectable to existing methods that focus on encoder corruption. Other data-level methods that sanitize data before training or during inference, also fail to answer the critical question, "Is the delivered model backdoored or not?" To address this model-level verification problem, we introduce CLIP-Inspector (CI), a backdoor detection method designed for prompt-tuned CLIP models. Assuming white-box access to the delivered model and a pool of unlabeled OOD images, CI reconstructs possible triggers for each class to determine if the model exhibits backdoor behaviour or not. Additionally, we demonstrate that using CI's reconstructed trigger for fine-tuning on correctly labeled triggered inputs enables us to re-align the model and reduce backdoor effectiveness. Through extensive experiments across ten datasets and four backdoor attacks, we demonstrate that CI can reconstruct effective triggers in a single epoch using only 1,000 OOD images, achieving a 94% detection accuracy (47/50 models). Compared to adapted trigger-inversion baselines, CI yields a markedly higher AUROC score (0.973 vs 0.495/0.687), thus enabling the vetting and post-hoc repair of prompt-tuned CLIP models to ensure safe deployment.

Figures & tables

Figure 1: Threat model and audit workflow. A semi-honest provider can implant a backdoor into the CLIP model while following the specified prompt-tuning protocol. A security auditor uses our method to determine whether the model is backdoored and, if necessary, performs a light repair using the reconstructed trigger.

Panel of Figure 2 — Figure 2: (a) Image-space embeddings for clean (blue) and poisoned (red) OOD inputs substantially overlap, while meta-tokens for poisoned inputs form tight clusters. (b) For a poisoned model, even a slight image perturbation can cause substantial changes in the targeted text embedding, resulting in the backdoor effect. In contrast, for a clean model, the shift is negligible.

Figure 3: Overview of our CLIP-Inspector framework for detecting backdoor attacks. For each candidate class, CI optimizes an $\ell_{\infty}$ -bounded trigger by minimizing a margin between the target logit and the highest non-target logit on unlabeled OOD images. The triggers are used to compute class-wise anomaly scores with the presence of an outlier yielding a model-level backdoor decision.

Dataset	ACC (Seen Classes)		ASR (Seen Classes)		ACC (Unseen Classes)		ASR (Unseen Classes)
Dataset	mean	std	mean	std	mean	std	mean	std
Caltech101	97.29%	1.51%	89.73%	17.59%	92.25%	1.28%	79.89%	28.13%
DTD	71.10%	6.48%	88.92%	9.08%	44.24%	3.67%	83.25%	11.49%
EuroSAT	89.45%	5.40%	95.35%	6.23%	46.25%	2.79%	76.25%	26.79%
FGVC	36.06%	3.67%	89.01%	20.78%	30.89%	2.02%	69.76%	38.80%
Food101	89.23%	0.60%	93.64%	10.46%	89.04%	1.89%	91.83%	12.17%

Table 1: Accuracy and Attack success rate for all datasets averaged across 4 attack types and clean models.

Panel of Figure 4 — Figure 4: AUROC plots for each detection method. Our method achieves a high AUROC of 0.973, whereas NC and PixB have low scores of 0.495 and 0.687, respectively. Our method also shows a clear distinction between clean and backdoored models in terms of average anomaly score.

		CI (Ours)				NC				PixB
Dataset	Total Classes	TP	FP	FN	F1	TP	FP	FN	F1	TP	FP	FN	F1
Caltech101	250	4	0	0	1.00	1	0	3	0.40	2	6	2	0.33
DTD	235	3	0	1	0.86	0	13	4	0.00	1	1	3	0.33
EuroSAT	50	2	1	2	0.57	0	1	4	0.00	2	2	2	0.50
FGVC	250	4	2	0	0.80	0	21	4	0.00	2	2	2	0.50
Food101	250	4	0	0	1.00	2	21	2	0.15	3	8	1	0.40

Table 2: Target Class detection metrics for each method. CLIP Inspector has almost zero false positives, correctly identifying the target class for every dataset.

Attack		CI (Ours)				NC				PixB
Attack	Total Classes	TP	FP	FN	F1	TP	FP	FN	F1	TP	FP	FN	F1
BadCLIP	444	9	0	1	0.94	1	30	9	0.04	6	11	4	0.44
Blended	444	10	1	0	0.95	2	24	8	0.11	10	8	0	0.71
SIBA	444	10	0	0	1	0	32	10	0	6	15	4	0.38
WaNet	444	8	1	2	0.84	1	35	9	0.04	2	11	8	0.17
Average	444	9.25	0.5	0.75	0.93	1	30.25	9	0.05	6	11.25	4	0.42

Table 3: Target class detection metrics for each attack. CI has a high F1 score across all attack types.

Loss Type	Backdoor class ASR		Average other class ASR
Loss Type	Mean	Std	Mean	Std
$\mathcal{L}_{\text{logit}}$	77.66	26.96	5.37	3.11
$\mathcal{L}_{\text{CE}}$	81.45	13.48	5.26	0.83
$\mathcal{L}_{\text{margin}}$	91.09	8.73	4.62	0.8

Table 4: Comparison of different loss functions in terms of reconstructed trigger ASR, averaged over 4 datasets (ImageNet, Caltech101, UCF101, DTD) for BadCLIP (ASR

\approx 99\%

Attack	ACC before	ACC after	ASR before	ASR after
BadCLIP	81.76%	81.35%	98.48%	7.61%
BadCLIP_adaptive	81.62%	81.04%	64.35%	4.23%
Blended	81.69%	81.69%	99.73%	7.64%
SIBA	75.91%	77.93%	66.30%	5.76%
WaNet	82.08%	82.11%	92.99%	5.27%
Average	80.61%	80.82%	84.37%	6.10%

Table 5: ACC and ASR for all attack types averaged over 10 datasets. Fine-tuning using reconstructed triggers effectively reduces the backdoor ASR to below 10%.

Dataset	Seen ACC	Seen ASR	Unseen ACC	Unseen ASR	ACC Diff.	ASR Diff.
Caltech101	98.19%	-	93.12%	-	5.07%	-
DTD	75.60%	-	50.46%	-	25.14%	-
EuroSAT	91.14%	-	47.54%	-	43.60%	-
FGVC_Aircraft	37.15%	-	31.97%	-	5.18%	-
Food101	90.26%	-	90.93%	-	-0.67%	-
ImageNet	76.05%	-	69.39%	-	6.66%	-

Table 6: Accuracy of clean models on Seen and Unseen class subsets for each dataset.

Dataset	Seen ACC	Seen ASR	Unseen ACC	Unseen ASR	ACC Diff.	ASR Diff.
Caltech101	98.06%	99.81%	93.78%	99.56%	4.28%	0.25%
DTD	74.03%	94.20%	43.87%	90.16%	30.16%	4.04%
EuroSAT	91.40%	99.95%	47.85%	96.72%	43.55%	3.23%
FGVC_Aircraft	36.19%	99.94%	31.37%	91.72%	4.82%	8.22%
Food101	90.09%	99.71%	90.89%	98.48%	-0.80%	1.23%
ImageNet	75.72%	99.60%	69.72%	99.02%	6.00%	0.58%

Table 7: ACC and ASR of BadCLIP-poisoned models on Seen and Unseen subsets.

Dataset	Seen ACC	Seen ASR	Unseen ACC	Unseen ASR	ACC Diff.	ASR Diff.
Caltech101	98.19%	99.94%	91.48%	94.00%	6.71%	5.94%
DTD	73.43%	98.91%	45.37%	95.72%	28.06%	3.19%
EuroSAT	92.31%	100.00%	49.33%	100.00%	42.98%	0.00%
FGVC_Aircraft	38.96%	99.58%	28.13%	98.44%	10.83%	1.14%
Food101	88.93%	99.93%	86.48%	99.82%	2.45%	0.11%
ImageNet	74.97%	99.54%	65.90%	98.25%	9.07%	1.29%

Table 8: ACC and ASR of Blended attack models on Seen and Unseen class subsets.

Dataset	Seen ACC	Seen ASR	Unseen ACC	Unseen ASR	ACC Diff.	ASR Diff.
Caltech101	95.03%	63.52%	92.79%	38.32%	2.24%	25.20%
DTD	61.47%	82.49%	48.26%	71.99%	13.21%	10.50%
EuroSAT	81.40%	94.67%	44.31%	63.90%	37.09%	30.77%
FGVC_Aircraft	30.85%	57.86%	32.99%	13.38%	-2.14%	44.48%
Food101	88.75%	78.09%	89.90%	73.80%	-1.15%	4.29%
ImageNet	71.32%	19.90%	66.51%	13.97%	4.81%	5.93%

Table 9: SIBA attack ACC and ASR values on seen and unseen subsets for each dataset.

Dataset	Seen ACC	Seen ASR	Unseen ACC	Unseen ASR	ACC Diff.	ASR Diff.
Caltech101	97.87%	95.67%	90.94%	87.66%	6.93%	8.01%
DTD	75.48%	80.07%	39.47%	75.12%	36.01%	4.95%
EuroSAT	92.71%	86.79%	43.49%	44.36%	49.22%	42.43%
FGVC_Aircraft	38.24%	98.68%	31.07%	75.52%	7.17%	23.16%
Food101	89.17%	96.83%	88.91%	95.20%	0.26%	1.63%
ImageNet	74.76%	96.32%	66.38%	90.02%	8.38%	6.30%

Table 10: WaNet attack ACC and ASR values on seen and unseen subsets for each dataset.

Dataset	Seen ACC	Seen ASR	Unseen ACC	Unseen ASR	CI ASR	Anomaly Score
Caltech101	97.29%	64.11%	93.45%	58.95%	67.29%	11.20
DTD	76.93%	63.04%	47.34%	56.25%	31.54%	6.86
EuroSAT	90.50%	52.38%	46.56%	20.18%	67.77%	1.66
FGVC_Aircraft	34.75%	93.04%	34.07%	35.03%	58.50%	6.32
Food101	90.22%	56.35%	90.84%	48.38%	95.02%	13.32
ImageNet	75.94%	71.68%	69.68%	52.93%	98.83%	11.67

Table 11: BadCLIP-adaptive results. Enforcing trigger specificity sharply reduces ASR, yet the behavioural anomaly metric of CLIP-Inspector still flags most backdoored models.

	CLIP-Inspector
Attack	Backdoor ASR	ASR Difference	Backdoor Loss Average	Loss Average Difference
Clean	7.51	-26.39	5.2218	-2.0343
BadCLIP	94.74	66.27	-8.2344	-15.7907
Blended	99.02	72.68	-11.2263	-20.8507
SIBA	92.69	65.75	-6.2807	-13.1315
WaNet	73.33	50.61	-4.5774	-12.1891

Table 12: ASR and Average Optimization loss values for triggers reconstructed via CLIP-Inspector for Caltech101 dataset.

	Neural Cleanse
Attack	Backdoor ASR	ASR Difference	Backdoor Mask Size	Mask Size Difference
Clean	99.64	-0.29	9874.8175	-7479.3773
BadCLIP	99.49	-0.5	3836.3224	-13693.3142
Blended	99.34	-0.61	574.5364	-19629.362
SIBA	99.52	-0.47	2861.0645	-14295.5531
WaNet	99.48	-0.5	4895.8796	-13152.8507

Table 13: ASR and Average Optimization loss values for triggers reconstructed via Neural Cleanse.

	Pixel Backdoor
Attack	Backdoor ASR	ASR Difference	Backdoor Loss Average	Loss Average Difference
Clean	85.77	-5.8	64122.1	-36842.2
BadCLIP	88.64	-3.77	35026	-61933.6
Blended	97.08	7.4	5455.2	-104308.3
SIBA	91.18	-1.74	25955.1	-65539
WaNet	87.08	-5.08	57801.5	-44050.9

Table 14: ASR and Average Optimization loss values for triggers reconstructed via Pixel Backdoor.

Figure 5: Average anomaly scores for each method averaged across 10 datasets for each attack type. CI shows clear distinction between clean and backdoored models while other methods fail to do so.

Figure 6: Clean and Triggered image pairs for each attack type. All triggers are visually imperceptible except Blended.

	Num OOD samples
Dataset	100	500	1000
Caltech101	58%	86.40%	98.20%
DTD	11%	73.60%	80.40%
EuroSAT	92%	97.60%	99.60%
FGVC_Aircraft	58%	83.40%	94.30%
Food101	73%	100%	99.80%

Table 15: ASR of CI’s reconstructed trigger on the backdoor class when varying the number of samples used for backdoor detection in BadCLIP models.

	Batch Size
Dataset	1	32	64
Caltech101	69.9%	98.20%	91.3%
DTD	32.2%	80.40%	87.4%
EuroSAT	98.1%	99.60%	99.8%
FGVC_Aircraft	52.6%	94.30%	89.2%
Food101	96.5%	99.80%	100.0%

Table 16: ASR values for the backdoor target class when varying the batch size used for trigger inversion (1000 OOD samples, 1 epoch).

	ID vs OOD samples
Dataset	ID	OOD
Caltech101	99.7%	98.20%
DTD	89.6%	80.40%
EuroSAT	98.8%	99.60%
FGVC_Aircraft	99.2%	94.30%
Food101	100.0%	99.80%

Table 17: ASR values for the backdoor target class when using ID vs OOD data for trigger inversion. ASR is reported for the backdoor target class for the BadCLIP attack models.

Pattern	ASR	CI-ASR	Anomaly Score
Gaussian	99.7%	98.75%	6.15
Triangle	86.1%	96.88%	5.93
Text	94.7%	62.4%	4.38

Table 18: Generality beyond prompt-tuning.

Condition	ACC before	ACC after	ASR before	ASR after
Clean-only	80.6%	81.4%	84.4%	64.6%
Random- $\delta$	80.6%	81.0%	84.4%	52.7%
Wrong-class- $\delta$	80.6%	80.3%	84.4%	47.0%
CI-trigger(Ours)	80.6%	80.8%	84.4%	6.1%

Table 19: Measuring backdoor removal effectiveness by comparing against different trigger initializations.

Attack	ACC before	ACC after	ASR before	ASR after
BadCLIP	81.8%	81.3%	99.0%	85.2%
BadCLIP_adaptive	81.6%	81.5%	64.1%	59.4%
Blended	81.7%	81.8%	99.7%	87.3%
SIBA	75.9%	79.1%	66.3%	8.3%
WaNet	82.1%	83.2%	92.7%	82.6%
Average	80.6%	81.4%	84.4%	64.6%

Table 20: Using only a clean dataset for fine tuning does not remove the backdoor.

Attack	ACC before	ACC after	ASR before	ASR after
BadCLIP	81.8%	81.4%	99.0%	7.6%
BadCLIP_adaptive	81.6%	81.0%	64.1%	4.2%
Blended	81.7%	81.7%	99.7%	7.6%
SIBA	75.9%	77.9%	66.3%	5.8%
WaNet	82.1%	82.1%	92.7%	5.3%
Average	80.6%	80.8%	84.4%	6.1%

Table 21: Removal Using CI-trigger.

Attack	ACC before	ACC after	ASR before	ASR after
BadCLIP	81.8%	81.1%	99.0%	73.3%
BadCLIP_adaptive	81.6%	82.0%	64.1%	55.1%
Blended	81.7%	81.5%	99.7%	40.0%
SIBA	75.9%	78.8%	66.3%	8.6%
WaNet	82.1%	81.7%	92.7%	86.5%
Average	80.6%	81.0%	84.4%	52.7%

Table 22: Removal using random noise is highly ineffective for all attack types.

Attack	ACC before	ACC after	ASR before	ASR after
BadCLIP	81.8%	80.5%	99.0%	60.3%
BadCLIP_adaptive	81.6%	80.2%	64.1%	43.5%
Blended	81.7%	80.9%	99.7%	56.0%
SIBA	75.9%	77.8%	66.3%	5.4%
WaNet	82.1%	82.0%	92.7%	70.0%
Average	80.6%	80.3%	84.4%	47.0%

Table 23: Removal Using CI-inverted trigger from a non-target class.

Dataset	BadCLIP Trigger SSIM (mean)	BadCLIP Trigger SSIM (std)	CI Recon. Trigger SSIM (mean)	CI Recon. Trigger SSIM (std)	NC Recon. Trigger SSIM (mean)	NC Recon. Trigger SSIM (std)	PixB Recon. Trigger SSIM (mean)	PixB Recon. Trigger SSIM (std)
Caltech101	0.9641	0.0202	0.9373	0.0346	0.4414	0.0999	0.4355	0.1261
DTD	0.9681	0.0178	0.9344	0.0360	0.9196	0.0375	0.6102	0.1106
EuroSAT	0.9703	0.0165	0.9367	0.0350	0.9999	0.0001	0.6389	0.1067
FGVC_Aircraft	0.9632	0.0204	0.9362	0.0352	0.7920	0.0657	0.7617	0.0813
Food101	0.9617	0.0215	0.9356	0.0354	0.5132	0.0894	0.7840	0.0760
ImageNet	0.9549	0.0255	0.9395	0.0337	0.8603	0.0509	0.8176	0.0669

Table 24: SSIM values for the original BadCLIP trigger and triggers reconstructed using our method and baselines. Our method achieves an SSIM value of 0.93, close to the original trigger’s value of 0.96.

Dataset	Blended Trigger SSIM (mean)	Blended Trigger SSIM (std)	CI Recon. Trigger SSIM (mean)	CI Recon. Trigger SSIM (std)	NC Recon. Trigger SSIM (mean)	NC Recon. Trigger SSIM (std)	PixB Recon. Trigger SSIM (mean)	PixB Recon. Trigger SSIM (std)
Caltech101	0.5742	0.146	0.9344	0.0361	0.9414	0.0263	0.9927	0.0078
DTD	0.5475	0.152	0.9346	0.0357	0.7717	0.0372	0.841	0.0593
EuroSAT	0.5347	0.152	0.9332	0.0365	0.9859	0.0131	0.7639	0.0815
FGVC_Aircraft	0.4375	0.145	0.9345	0.0359	0.9405	0.0287	0.7141	0.0936
Food101	0.523	0.144	0.9352	0.0356	0.9999	0.0001	0.9294	0.0307
ImageNet	0.5446	0.143	0.9346	0.0357	0.9948	0.006	0.8365	0.0639

Table 25: SSIM values for the original Blended trigger and triggers reconstructed using our method and baselines.

Dataset	SIBA Trigger SSIM (mean)	SIBA Trigger SSIM (std)	CI Recon. Trigger SSIM (mean)	CI Recon. Trigger SSIM (std)	NC Recon. Trigger SSIM (mean)	NC Recon. Trigger SSIM (std)	PixB Recon. Trigger SSIM (mean)	PixB Recon. Trigger SSIM (std)
Caltech101	0.9995	0.0003	0.9367	0.0348	0.6516	0.1067	0.7586	0.0855
DTD	0.9995	0.0002	0.9375	0.0345	0.864	0.0391	0.6125	0.1116
EuroSAT	0.9995	0.0002	0.9374	0.0348	0.9781	0.0192	0.5037	0.1208
FGVC_Aircraft	0.9994	0.0003	0.9378	0.0341	0.5587	0.0896	0.5822	0.1156
Food101	0.9996	0.0002	0.9364	0.0351	0.8724	0.0665	0.499	0.1242
ImageNet	0.9995	0.0003	0.9372	0.0344	0.9631	0.0251	0.7942	0.0726

Table 26: SSIM values for the original SIBA trigger and triggers reconstructed using our method and baselines. SIBA trigger is highly sparse and imperceptible, evident by the 0.99 SSIM score.

Dataset	WaNet Trigger SSIM (mean)	WaNet Trigger SSIM (std)	CI Recon. Trigger SSIM (mean)	CI Recon. Trigger SSIM (std)	NC Recon. Trigger SSIM (mean)	NC Recon. Trigger SSIM (std)	PixB Recon. Trigger SSIM (mean)	PixB Recon. Trigger SSIM (std)
Caltech101	0.9312	0.0186	0.9358	0.0354	0.4515	0.1005	0.3998	0.1233
DTD	0.9342	0.0186	0.9372	0.0347	0.7839	0.0608	0.4233	0.1229
EuroSAT	0.9309	0.0186	0.9374	0.0344	0.9588	0.0207	0.3643	0.119
FGVC_Aircraft	0.9233	0.0184	0.9373	0.0345	0.6507	0.1058	0.3748	0.1213
Food101	0.9473	0.0189	0.9376	0.0346	0.8571	0.0648	0.6744	0.1013
ImageNet	0.9512	0.0190	0.9356	0.0354	0.784	0.0798	0.3707	0.1211

Table 27: SSIM values for the original WaNet trigger and triggers reconstructed using our method and baselines. We apply a

k=4

control-grid distortion with noise strength

s=0.1

Backdoor Attacks on VLMs

Backdoor attacks on vision-language models study hidden triggers that make multimodal systems behave maliciously while appearing normal.

See all 36 papers

Sep 25, 2026

Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP

Contrastive Language--Image Pretraining (CLIP) has emerged as a dominant vision backbone due to its strong transferability and zero-shot capabilities. However, recent studies reveal a critical vulnerability: embedding-space backdoor attacks. By poisoning only a tiny fraction of image--text pairs, adversaries can implant stealthy triggers that induce targeted shifts in CLIP's joint embedding space. Unlike conventional backdoors that manipulate classifier logits, these attacks corrupt representations directly, making them highly effective under extremely low poisoning ratios and difficult to detect. Existing defenses require access to model parameters, gradients, logits, or clean validation data---assumptions that rarely hold in realistic black-box deployments. Moreover, current black-box methods struggle to accurately localize small or out-of-distribution triggers. We propose CLIPGuard, a lightweight and fully black-box defense specifically designed to mitigate embedding-space backdoors in CLIP encoders. CLIPGuard identifies malicious regions by measuring segment-wise embedding perturbations and selectively purifies only suspicious segments via semantic inpainting, preserving benign visual content and alignment quality. Extensive experiments on STL-10, ImageNet, and diverse trigger families---including BadCLIP, BadNets, blended, patch-based, and typographic attacks---demonstrate that CLIPGuard reduces attack success rates to as low as 1.05% while maintaining clean accuracy up to 86.34%, consistently outperforming existing black-box defenses, including CleanCLIP and CleanerCLIP. Our code is available https://github.com/wsu-cyber-security-lab-ai/CLIPGuard.git

Ahmed Abdelnaby, Mohamed Elmahallawy

Washington State University, Richland, WA 99354, USA

Backdoor Defense VLM Robustness Backdoor Attacks on VLMs Vision-Language Pretraining

Ask PDF

Jun 16, 2026

Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors

Contrastive Language-Image Pre-training models are widely reused across downstream interfaces, including feature extraction, retrieval, reranking, and selection. Existing CLIP backdoor, however, usually validate attacks on a small attack-native task, leaving unclear whether the same poisoned checkpoint remains exposed, weakens, or becomes not applicable when reused through other interfaces. We introduce DIFE, a Deployment-Interface Footprint Evaluation framework that audits backdoored CLIP checkpoints across deployment interfaces. DIFE makes various evaluations comparable by specifying each interface's component readout, trigger channel, target event, reference condition, and metric. DIFE also introduces effective-footprint diagnosis to identify the reusable CLIP component or component combination that carries exposure and explains where risk transfers. Auditing reproduced CLIP backdoors with DIFE reveals a structured landscape: native success is not a checkpoint-level risk certificate, exposure follows component footprints, text-side poisoning does not yield textual-encoder control, and some coupled attacks remain mechanism-bound. This audit reveals a import gapin existing CLIP backdoors: a textual encoder that itself becomes a reusable carrier of adversarial behavior. We therefore introduce BadTextTower to fill this gap. BadTextTower produces strong text-conditioned retrieval, reranking, and selection exposure while leaving visual-only reuse nearly clean.

Kunlan Xiang, Haomiao Yang, Wenbo Jiang

University of Electronic Science and Technology of China

VLM Robustness Adversarial Attacks on VLMs Backdoor Attacks Backdoor Attacks on VLMs Vision-Language Pretraining

Ask PDF

Sep 28, 2026

Backdoor as Probe: Test-Time Adversarial Defense for CLIP

Test-time adversarial defense improves the robustness of vision-language foundation models such as CLIP without retraining. However, adversarial activation shifts are typically treated as distortions to suppress, rather than signals to exploit. We turn these shifts into defense signals by repurposing the trigger-to-target mechanism of backdoors. The key is to implant a defender-controlled backdoor as a probe that is weakly activated by clean inputs but strongly activated by adversarial shifts. Based on this insight, we propose Backdoor as Probe (BaP), a test-time adversarial defense for CLIP. BaP constructs the probe through a closed-form model edit to a selected MLP layer. It projects the average adversarial activation shift and a defender-specified semantic direction onto the layer's low-energy input and output activation subspaces to obtain the trigger and target directions, respectively. At inference time, adversarial inputs produce measurable responses along the target direction for detection. BaP then selectively rectifies detected inputs by optimizing a small perturbation that steers their representations away from adversarial shifts and toward the clean subspace. Experiments across 16 benchmarks show that BaP improves average robust accuracy from 1.0% to 52.3% while retaining clean accuracy, achieving performance comparable to state-of-the-art methods with up to a

5.7\times

inference speedup. BaP further shows the generalization to adversarial attacks on large vision-language models. Project page: https://robin-wzq.github.io/Backdoor-as-Probe/

Zhongqi Wang, Jie Zhang, Nie Sen +3

Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences (CAS), Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Xuzhou University of Technology, China

VLM Robustness Backdoor Defense Adversarial Example Detection

Ask PDF

Abstract

Backdoor Attacks on VLMs

Explore similar work

Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIP

Beyond Native Success: Auditing Deployment-Interface Exposure of CLIP Backdoors

Backdoor as Probe: Test-Time Adversarial Defense for CLIP