BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder

Siquan Huang; Yijiang Li; Ningzhi Gao; Xingfu Yan; Leyu Shi; Ying Gao

cs.CVMar 12, 2026

BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder

Authors: Siquan Huang, Yijiang Li, Ningzhi Gao, Xingfu Yan, Leyu Shi, Ying Gao

Organizations: School of Computer Science and Engineering, South China University of Technology · Department of Electrical and Computer Engineering, University of California San Diego · School of Computer Science, South China Normal University

Abstract

Self-supervised and multimodal vision encoders learn strong visual representations that are widely adopted in downstream vision tasks and large vision-language models (LVLMs). However, downstream users often rely on third-party pretrained encoders with uncertain provenance, exposing them to backdoor attacks. In this work, we propose BackdoorIDS, a simple yet effective zero-shot, inference-time backdoor samples detection method for pretrained vision encoders. BackdoorIDS is motivated by two observations: Attention Hijacking and Restoration. Under progressive input masking, a backdoored image initially concentrates attention on malicious trigger features. Once the masking ratio exceeds the trigger's robustness threshold, the trigger is deactivated, and attention rapidly shifts to benign content. This transition induces a pronounced change in the image embedding, whereas embeddings of clean images evolve more smoothly across masking progress. BackdoorIDS operationalizes this signal by extracting an embedding sequence along the masking trajectory and applying density-based clustering such as DBSCAN. An input is flagged as backdoored if its embedding sequence forms more than one cluster. Extensive experiments show that BackdoorIDS consistently outperforms existing defenses across diverse attack types, datasets, and model families. Notably, it is a plug-and-play approach that requires no retraining and operates fully zero-shot at inference time, making it compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5.

Figures & tables

Panel of Figure 3 — Figure 3: The t-SNE projection of clean/backdoor image embeddings.

Figure 4: Overview of our proposed BackdoorIDS.

Methods	BadEncoder		Drupe		BadCLIP (RN)		BadCLIP (ViT)		BadVision
Methods	TPR $\uparrow$	FPR $\downarrow$	TPR $\uparrow$	FPR $\downarrow$	TPR $\uparrow$	FPR $\downarrow$	TPR $\uparrow$	FPR $\downarrow$	TPR $\uparrow$	FPR $\downarrow$
PatchProc Doan et al. (2023)	84.32	22.73	74.66	25.13	63.11	31.84	82.97	26.78	41.00	27.54
DeDe Hou et al. (2025)	89.01	18.50	69.94	20.94	70.20	25.26	55.12	27.97	7.00	29.68
BackdoorIDS	95.78	17.38	82.75	19.18	93.08	28.38	95.84	25.76	97.00	23.78

Table 1: Performance of Inference-phase Backdoor Sample Detection Methods.

Table 5

Methods	BadEncoder		Drupe		BadCLIP (RN)		BadCLIP (ViT)		BadVision
Methods	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	CA $\uparrow$	ASR $\downarrow$	*CA $\uparrow$
No-defense	99.48	79.11	83.05	74.99	99.61	57.48	99.99	61.39	99.00	85.30
ZIP Shi et al. (2023)	15.89	62.15	21.22	61.97	48.26	45.22	54.68	47.54	82.00	72.32
SampDetox Yang et al. (2024)	11.38	63.52	14.37	60.89	32.18	48.98	46.57	49.34	44.00	67.59
PatchProc Feng et al. (2023)	15.11	59.26	14.87	40.26	38.22	38.97	16.44	45.69	58.00	62.11
DeDe Hou et al. (2025)	10.90	63.96	19.78	44.36	36.68	42.32	44.84	44.46	93.00	59.69

Table 4: Performance of Inference-phase Methods.(* denotes the CIDEr score)

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Trigger implemented in the attacks.

Figure 10: Effectiveness of BackdoorIDS under different masking ratios.

Methods	BadEncoder		BadCLIP (ViT)
Methods	TPR	FPR	TPR	FPR
16 $\times$ 16	95.78	17.38	95.84	25.76
8 $\times$ 8	91.26	12.74	86.4	6.67

Appendix

Table 5: Effectiveness of BackdoorIDS under different grids.

Methods	BadEncoder				BadCLIP (ViT)
Methods	mean	std	min	max	mean	std	min	max
TPR	95.70	0.46	94.70	96.24	95.73	0.44	95.04	96.48
FPR	17.63	0.14	17.32	17.82	25.73	0.11	25.55	25.90

Appendix

Table 6: Effectiveness of BackdoorIDS under different masking trajectories.

Backdoor Detection

Backdoor detection finds hidden triggers or poisoned behavior in trained models before they can secretly redirect predictions or actions.

See all 53 papers

DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors
Aug 26, 2026Tuo Chen, Jie Gui, Minjing Dong +4Backdoor Defense Backdoor Attacks on VLMs
School of Cyber Science and Engineering Southeast University Nanjing, China · Ant Group Hangzhou, China · Purple Mountain Laboratories Nanjing, China +3
EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy
May 15, 2026Xuanyu Ge, Zhongqi Wang, Jie Zhang +2Large Vision-Language Models Backdoor Detection
China University of Geosciences · University of the Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences
Test-Time Attention Purification for Backdoored Large Vision Language Models
Mar 13, 2026Zhifang Zhang, Bojun Yang, Shuo He +5Backdoor Attacks on VLMs Adversarial Attacks on VLMs
University of Queensland · Southeast University · Nanyang Technological University +1

Aug 26, 2026J/K move · Enter open · S save

DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors

Tuo Chen, Jie Gui, Minjing Dong, Lanting Fang+3

School of Cyber Science and Engineering Southeast University Nanjing, China · Ant Group Hangzhou, China · Purple Mountain Laboratories Nanjing, China+3

Backdoor Defense Backdoor Attacks on VLMs Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Self-supervised learning (SSL) encoders are vulnerable to backdoor attacks, posing threats to both visual SSL encoders and vision-language encoders. Existing defenses are typically designed for only one of these paradigms and rely on restrictive assumptions such as access to uninfected in-distribution data or precomputed pseudo-labels, which are difficult to satisfy in practice. To address these limitations, we propose DEFUSE, a generalizable backdoor detection framework for SSL encoders. Inspired by Bayesian posterior inference, we reformulate backdoor detection as a representation-conditioned image likelihood estimation problem parameterized by a conditional diffusion generative model. Uninfected representations tend to yield semantically consistent reconstructions, whereas backdoored ones are more likely to be mapped to the attacker's target class or semantically meaningless images, deviating from the original semantics and thereby exposing the backdoor. However, we find that the exact likelihood is intractable, because highly abstracted representations discard the low-level information necessary for pixel-faithful reconstruction. We therefore relax the objective to semantic reconstruction and evaluate it in a well-separated representation space provided by a reference encoder. Rather than training from scratch, we fine-tune a pretrained diffusion model, leveraging its generative prior to map data onto the natural image manifold while preserving semantic content. Extensive experiments demonstrate that DEFUSE substantially outperforms existing detectors across diverse attack settings, generalizing to both visual SSL and vision-language encoders. Notably, our method greatly reduces the reliance on prior knowledge about the victim encoder or the attack strategy. The source code is available at https://github.com/jsrdcht/DEFUSE .

Abstract

Backdoor Detection

Explore similar work

DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors

EntropyScan: Towards Model-level Backdoor Detection in LVLMs via Visual Attention Entropy

Test-Time Attention Purification for Backdoored Large Vision Language Models