DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors
Authors: Tuo Chen, Jie Gui, Minjing Dong, Lanting Fang, Ju Jia, Benlei Cui, Jian Liu
Organizations: School of Cyber Science and Engineering Southeast University Nanjing, China · Ant Group Hangzhou, China · Purple Mountain Laboratories Nanjing, China · Department of Computer Science City University of Hong Kong Hong Kong, China · Beijing Institute of Technology Beijing, China · Alibaba Group Hangzhou, China
Self-supervised learning (SSL) encoders are vulnerable to backdoor attacks, posing threats to both visual SSL encoders and vision-language encoders. Existing defenses are typically designed for only one of these paradigms and rely on restrictive assumptions such as access to uninfected in-distribution data or precomputed pseudo-labels, which are difficult to satisfy in practice. To address these limitations, we propose DEFUSE, a generalizable backdoor detection framework for SSL encoders. Inspired by Bayesian posterior inference, we reformulate backdoor detection as a representation-conditioned image likelihood estimation problem parameterized by a conditional diffusion generative model. Uninfected representations tend to yield semantically consistent reconstructions, whereas backdoored ones are more likely to be mapped to the attacker's target class or semantically meaningless images, deviating from the original semantics and thereby exposing the backdoor. However, we find that the exact likelihood is intractable, because highly abstracted representations discard the low-level information necessary for pixel-faithful reconstruction. We therefore relax the objective to semantic reconstruction and evaluate it in a well-separated representation space provided by a reference encoder. Rather than training from scratch, we fine-tune a pretrained diffusion model, leveraging its generative prior to map data onto the natural image manifold while preserving semantic content. Extensive experiments demonstrate that DEFUSE substantially outperforms existing detectors across diverse attack settings, generalizing to both visual SSL and vision-language encoders. Notably, our method greatly reduces the reliance on prior knowledge about the victim encoder or the attack strategy. The source code is available at https://github.com/jsrdcht/DEFUSE .
Figures & tables
Figure 1. Illustration of the proposed framework for reconstructing backdoor representations back to the image space. Four-stage overview of DEFUSE. Clean images and visually triggered images pass through a backdoored encoder. The trigger shifts the representation away from the source-image manifold and toward the attack-target manifold, illustrated with a banana target. A conditional generative model maps each representation back to an image, and a reference encoder compares the original and reconstructed images. Clean pairs have high cosine similarity, whereas backdoored pairs have low similarity and are separated by a threshold.
Figure 2. (a) Reconstruction performance across different CLIP encoders ( Radford et al., 2021 ) . (b) Low-level distance metrics (e.g., L2) fail to reliably assess semantic consistency, as semantically similar images can exhibit high distances. Two-panel motivation figure. Panel a compares reconstruction quality from global and dense features for several CLIP encoders using PSNR, LPIPS, and bubble size; dense features reconstruct better, while a single global token is insufficient for faithful reconstruction. Panel b plots L2 distance against DINOv2 similarity for clean and backdoored reconstructions. The weak relationship between the two metrics, together with examples having large L2 distance but similar semantics, shows why pixel-level distance is unreliable.
Figure 3. Framework for reconstructing raw pixels from latent representations. Image features are injected into the key/value ( K/V ) components of the cross-attention module in a pretrained CDDM. Architecture of the reconstruction and detection pipeline. A frozen suspicious encoder extracts and pools an input feature, which a trainable projection converts into four image-prompt tokens. These tokens condition the key and value projections of the cross-attention layers in a largely frozen diffusion denoiser over successive timesteps. The reconstructed image and the original image are then encoded by a frozen reference encoder to produce a backdoor score. Flame icons denote trainable modules and snowflake icons denote frozen modules.
Figure 4. Visualization of image reconstructions under different backdoor attacks and CDDMs. ‘Back.’ denotes backdoored images. The attack target is set to banana . We implement the masked autoencoder following Hou et al. ( Hou et al., 2025 ) . Image grid comparing original and backdoored inputs for CLIP Backdoor, CTRL, SSLBKD, and BadCLIP. Rows show the inputs and reconstructions from a masked autoencoder, RAE, PixelFlow, and SDXL. The masked autoencoder preserves much of the input appearance for both clean and backdoored images, making the two difficult to distinguish. The generative CDDMs, particularly SDXL, produce more visibly different semantic content from backdoored representations.
Figure 5. (a) Comparison of different CDDMs. “Text” suffix means keeping the original text prompts from pretrained text-to-image models. (b) Detection performance vs. DINOv2 similarity. Detection performance within a given model family consistently improves as similarity increases. “NI” suffix means inference with noised images other than pure noise. “K/V” suffix means finetuning only the key/value ( K/V ) projection modules. (c) Comparison of different metrics. AUC: AUPRC. Three-part ablation figure. Panel a plots DINOv2 reconstruction similarity against the number of training images for SDXL, SD1.5, PixelFlow, and text-conditioned variants; SDXL reaches the highest similarity. Panel b plots detection AUC against DINOv2 similarity for model and training variants, showing that higher reconstruction similarity generally corresponds to higher AUC. Panel c overlays clean and backdoored score distributions for L2, whitened CLIP, supervised ViT, SSIM, whitened caption CLIP, and DINOv2; the semantic metrics, especially DINOv2 and ViT, separate the two groups most clearly.
Defense
DECREE ( Feng et al., 2023 )
DBCL ( Huang et al., 2025a )
DEDE ( Hou et al., 2025 )
DEDE-OOD ( Hou et al., 2025 )
DEFUSE
Attack
R (%)
P (%)
AUPRC
R (%)
P (%)
AUPRC
R (%)
P (%)
AUPRC
R (%)
P (%)
AUPRC
R (%)
P (%)
AUPRC
Data Poisoning
SSLBKD ( Saha et al., 2022 )
0.1
41.2
0.49
51.5
51.5
0.52
71.0
51.6
0.52
28.0
50.7
0.51
85.2
81.0
0.81
CTRL ( Li et al., 2023 )
56.9
60.9
0.60
47.6
47.6
0.47
96.2
50.6
0.46
77.0
50.5
0.51
75.5
77.0
0.84
BLTO ( Sun et al., 2023 )
70.4
66.7
0.72
50.8
50.8
0.51
51.4
52.0
0.52
52.1
50.9
0.51
86.8
81.4
0.88
CLIP Backdoor ( Carlini and Terzis, 2021 )
76.7
97.4
0.95
91.7
91.7
0.96
85.3
55.4
0.59
47.4
66.8
0.62
98.0
92.3
0.96
Training Manipulation
BadEncoder ( Liang et al., 2024 )
0.1
42.8
0.48
59.7
59.7
0.61
69.2
61.0
0.67
70.1
61.2
0.65
93.4
84.3
0.89
Table 1. Backdoor detection performance against various backdoor attacks on ImageNet. The highest and second-highest AUPRC scores are highlighted in gray and underlined, respectively. R: Recall; P: Precision.
Table 7
Figure 6. Injecting poisons into the training set. AUROC and qualitative reconstructions as the number of poisoned training samples increases from zero to thirty thousand. The AUROC curve remains between approximately 0.98 and 1.00 across all poisoning levels. Example rows show reconstructed modeling-clay scenes, lizards, and billiard rooms, with the corresponding triggered raw images shown at the right.
Figure 7. Relationship between DINOv2 similarity and the optimal threshold τ . Bubble scatter plot across eleven natural-image, medical, and remote-sensing datasets. The horizontal axis is mean clean DINOv2 similarity and the vertical axis is the optimal detection threshold. A rising linear fit with an R-squared value of 0.67 and a shaded 95 percent confidence band indicates that datasets with higher clean reconstruction similarity generally require higher thresholds. Bubble color and size encode balanced accuracy.
Figure 8. Different ODE solvers with varying inference steps. Grouped bar chart comparing DDIM, Euler, DPM++, and UniPC at 5, 20, 50, and 200 inference steps. With five steps, AUROC ranges from 0.84 to 0.95. At twenty or more steps, all four solvers reach approximately 0.98 to 1.00 AUROC, indicating that twenty steps are sufficient.
Method
ACC
ASR
CLIP-Backdoor
0.0 → 22.0
95.2 → 8.0
BadCLIP
0.0 → 28.4
88.9 → 0.0
Table 4. Purification performance.
Figure 12
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 . DINOv2 similarity of reconstructions on hard images before and after backdoor insertion. Scatter plot of DINOv2 reconstruction similarity for twenty challenging images indexed from 00 to 19. Blue points show the original clean images and red points show their backdoored versions. Most clean scores lie above the threshold of 0.1, including several values above 0.5, whereas nearly all backdoored scores cluster near zero and below the threshold.
Figure S2 . Top-20 and bottom-20 reconstructed images under the L2 and DINOv2 metrics. Four grids of twenty images ranked by reconstruction metrics. Images with the largest L2 errors contain high contrast, text, icons, fine structures, or complex objects, whereas those with the smallest L2 errors are smooth, low-texture scenes such as sky, water, or fog. Images with the highest DINOv2 similarity usually contain one clear, salient object, while the lowest-ranked images contain cluttered scenes, multiple objects, ambiguous content, or text.
Table S1. Explanatory variables for the two reconstruction metrics on 150,000 CC3M images. The best full-feature model reaches R2=0.503 for L2 but only R2=0.066 for DINOv2 similarity. Qualitative comparison for square trigger sizes 10, 8, 6, and 4 on GTSRB traffic-sign images. Each group shows clean and backdoored raw images, masked-autoencoder reconstructions, and trigger masks. As the trigger becomes smaller, attack success remains nearly constant at 0.95 to 0.96, while detection AUC decreases from 0.92 to 0.82 because the clean and backdoored reconstructions become less distinguishable.
Self-supervised learning (SSL) pretrained models have become a dominant paradigm for visual representation learning, but they are vulnerable to backdoor attacks. Existing defenses struggle to defend against such attacks in a fully black-box setting because they often require access to labels, attack patterns, or training data. To tackle this issue, we propose a new attack-agnostic, model-agnostic, and modality-agnostic black-box test-time defense paradigm, called Platonic Representation Defense. It is inspired by the Platonic Representation Hypothesis, which suggests that large-scale independently trained encoders converge toward compatible projections of the same underlying reality. We formalize this idea as a conditional energy function defined over source representations and a set of reference representations. The energy function is trained for detection through noise-contrastive estimation and for representation purification through denoising score matching. Theoretically, the energy gap between matched and mismatched samples is lower bounded by the mutual information between source and reference representations. We demonstrate the effectiveness of our method on multiple self-supervised encoders and more than 10 attacks. The method can perform both representation detection and purification, and achieves substantial performance gains across multiple attacks. Code is available here.
Tuo Chen, Minjing Dong, Benlei Cui +2
Southeast University · Ant Group · City University of Hong Kong +2
Self-supervised and multimodal vision encoders learn strong visual representations that are widely adopted in downstream vision tasks and large vision-language models (LVLMs). However, downstream users often rely on third-party pretrained encoders with uncertain provenance, exposing them to backdoor attacks. In this work, we propose BackdoorIDS, a simple yet effective zero-shot, inference-time backdoor samples detection method for pretrained vision encoders. BackdoorIDS is motivated by two observations: Attention Hijacking and Restoration. Under progressive input masking, a backdoored image initially concentrates attention on malicious trigger features. Once the masking ratio exceeds the trigger's robustness threshold, the trigger is deactivated, and attention rapidly shifts to benign content. This transition induces a pronounced change in the image embedding, whereas embeddings of clean images evolve more smoothly across masking progress. BackdoorIDS operationalizes this signal by extracting an embedding sequence along the masking trajectory and applying density-based clustering such as DBSCAN. An input is flagged as backdoored if its embedding sequence forms more than one cluster. Extensive experiments show that BackdoorIDS consistently outperforms existing defenses across diverse attack types, datasets, and model families. Notably, it is a plug-and-play approach that requires no retraining and operates fully zero-shot at inference time, making it compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5.
Siquan Huang, Yijiang Li, Ningzhi Gao +3
School of Computer Science and Engineering, South China University of Technology · Department of Electrical and Computer Engineering, University of California San Diego · School of Computer Science, South China Normal University
Self-supervised diffusion models learn high-quality visual representations via latent space denoising. However, their representation layer poses a distinct threat: unlike traditional attacks targeting generative outputs, its unconstrained latent semantic space allows for stealthy backdoors, permitting malicious control upon triggering. In this paper, we propose BadRSSD, the first backdoor attack targeting the representation layer of self-supervised diffusion models. Specifically, it hijacks the semantic representations of poisoned samples with triggers in Principal Component Analysis (PCA) space toward those of a target image, then controls the denoising trajectory during diffusion by applying coordinated constraints across latent, pixel, and feature distribution spaces to steer the model toward generating the specified target. Additionally, we integrate representation dispersion regularization into the constraint framework to maintain feature space uniformity, significantly enhancing attack stealth. This approach preserves normal model functionality (high utility) while achieving precise target generation upon trigger activation (high specificity). Experiments on multiple benchmark datasets demonstrate that BadRSSD substantially outperforms existing attacks in both FID and MSE metrics, reliably establishing backdoors across different architectures and configurations, and effectively resisting state-of-the-art backdoor defenses.
Jiayao Wang, Yiping Zhang, Mohammad Maruf Hasan +5
School of Information and Artificial Intelligence, Yangzhou University, China · School of Computing and Artificial Intelligence, Chaohu University, China · Tacoma School of Engineering and Technology, University of Washington, USA