Organizations: School of Information and Artificial Intelligence, Yangzhou University, China · School of Computing and Artificial Intelligence, Chaohu University, China · Tacoma School of Engineering and Technology, University of Washington, USA
Self-supervised diffusion models learn high-quality visual representations via latent space denoising. However, their representation layer poses a distinct threat: unlike traditional attacks targeting generative outputs, its unconstrained latent semantic space allows for stealthy backdoors, permitting malicious control upon triggering. In this paper, we propose BadRSSD, the first backdoor attack targeting the representation layer of self-supervised diffusion models. Specifically, it hijacks the semantic representations of poisoned samples with triggers in Principal Component Analysis (PCA) space toward those of a target image, then controls the denoising trajectory during diffusion by applying coordinated constraints across latent, pixel, and feature distribution spaces to steer the model toward generating the specified target. Additionally, we integrate representation dispersion regularization into the constraint framework to maintain feature space uniformity, significantly enhancing attack stealth. This approach preserves normal model functionality (high utility) while achieving precise target generation upon trigger activation (high specificity). Experiments on multiple benchmark datasets demonstrate that BadRSSD substantially outperforms existing attacks in both FID and MSE metrics, reliably establishing backdoors across different architectures and configurations, and effectively resisting state-of-the-art backdoor defenses.
Figures & tables
Figure 1: Illustration of the BadRSSD attack framework. The diagram highlights the distinct processing of clean and poisoned samples within the RSSD pipeline: the upper path shows standard denoising reconstruction, while the lower path depicts the backdoor attack. The red dashed box marks the key PCA-space backdoor alignment step. Loss functions are indicated at their corresponding locations; see Sec. 3.2 and Sec. 3.3 for detailed definitions and calculations.
Pre-training
Downstream
Benign
BadEncoder
SSLBKD
Ours
BadDiffusion
TrojDiff
Ours
Dataset
Dataset
CA ↑
BA ↑
ASR ↑
BA ↑
ASR ↑
BA ↑
ASR ↑
FID ↓
MSE ↓
FID ↓
MSE ↓
FID ↓
MSE ↓
CelebA-HQ
CelebA-HQ
85.23
83.52
86.78
54.26
62.57
83.96
92.63
46.15
0.2124
47.06
0.2376
38.26
0.1625
CIFAR-100
CIFAR-10
82.36
81.45
73.86
50.65
53.72
83.65
91.26
42.86
0.1628
43.25
0.1726
36.12
0.0821
CelebA-HQ
85.57
74.56
76.42
48.75
52.53
84.92
94.67
47.26
0.1732
50.12
0.1935
38.52
0.1209
ImageNet
CIFAR-10
73.56
70.26
78.92
46.63
55.72
72.14
86.75
53.82
0.2028
55.19
0.2268
40.18
0.1526
CIFAR-100
72.24
68.55
80.16
49.46
57.23
71.94
87.12
55.27
0.2236
57.38
0.2435
43.26
0.1815
Table 1: Camparison of attack performance on different datasets. The best result are highlighted .
Samplers
ASR
FID
MSE
Steps
Time (s)
ASR-std
DDPM
92.37
40.12
0.1263
1000
15.2
2.1
DDIM
72.18
52.38
0.2514
50
4.2
0.5
DPM-solver
87.53
41.84
0.1347
20
3.8
1.2
Euler
78.96
49.16
0.2281
100
2.1
1.8
Table 2: Comparison of different samplers.
Figure 2: Experimental results for different Transformer architectures. Clean-FID (bars) and Backdoor-MSE (curves) .
Table 3: Visualized examples of BadRSSD on low-resolution CIFAR-10 with trigger Grey Box & target hat and without triggers at different poison rates. (Triggers are Gray Box. target categories: T1 (orange hat): CIFAR-10/100; T2 (cartoon girl): CelebA-HQ/ImageNet.)
Figure 3: On high-resolution CelebA-HQ, BadRSSD utilizes a gray-scale box trigger with a target cartoon girl across varying poisoning rates. Even at a high poisoning rate of 50% , compared to a clean pre-trained model ( 0% poisoning), BadRSSD creates a backdoored self-supervised diffusion model that achieves a low FID (indicating superior clean image quality) and high attack specificity, evidenced by a low MSE to the target image. Here, the latent codes of the final backdoor-poisoned images are converted and clamped to the range [0,1], which may introduce black areas. The results in Table 3 and Figure 4 are likewise processed.
Figure 4: Visualized samples of backdoor objectives synthesized by BadRSSD on ImageNet with trigger Grey Box & target cartoon girl across different training epochs.
Diffusion models (DMs), despite their impressive capabilities across a wide range of generative tasks, have been shown to be vulnerable to backdoor attacks. However, existing backdoor methods face critical trade-offs among key factors: attack performance, stealthiness, time complexity, and required poison rates. For example, achieving high attack performance typically demands a high poison rate and prolonged training, which undermines stealthiness, making the attack more detectable by backdoor defenses. This paper proposes TooBad (trigger optimization for backdoor diffusion models), a backdoor framework which introduces a novel DM-tailored trigger optimization technique to dramatically enhance the performance of backdoor attacks on DMs. Experiments on representative benchmarks such as CIFAR-10 show that TooBad can achieve high ASRs (>85%) at only 0.5% poison rate, significantly lower than the 10% typically required by prior work on the same datasets. At 5% poison rate, TooBad reaches nearly 100% ASR within just 3-5 backdoor injection epochs, whereas existing methods need at least 30-50 epochs at double the poison rate for comparable results. Despite its potency, TooBad easily evades SOTA defenses and maintains high utility. These results reveal a critical threat on DMs and highlight the need for more robust defenses against such stealthy yet efficient attacks.
Multimodal Diffusion Language Models (MDLMs) have recently emerged as a competitive alternative to their autoregressive counterparts. Yet their vulnerability to backdoor attacks remains largely unexplored. In this work, we show that well-established data-poisoning pipelines can successfully implant backdoors into MDLMs, enabling attackers to manipulate model behavior via specific triggers while maintaining normal performance on clean inputs. However, defense strategies effective to these models are yet to emerge. To bridge this gap, we introduce a backdoor defense framework for MDLMs named DiSP (Diffusion Self-Purification). DiSP is driven by a key observation: selectively masking certain vision tokens at inference time can neutralize a backdoored model's trigger-induced behaviors and restore normal functionality. Building on this, we purify the poisoned dataset using the compromised model itself, then fine-tune the model on the purified data to recover it to a clean one. Given such a specific design, DiSP can remove backdoors without requiring any auxiliary models or clean reference data. Extensive experiments demonstrate that our approach effectively mitigates backdoor effects, reducing the attack success rate (ASR) from over 90% to typically under 5%, while maintaining model performance on benign tasks.
Self-supervised learning (SSL) models are vulnerable to backdoor attacks. However, the systemic risks they pose in face representation have received little attention. The entanglement of identity features in self-supervised face learning presents unique challenges for attack stealthiness. To address this gap, we propose FIDA (Feature Instability-Driven Attack), a novel backdoor attack framework. FIDA uses subtle semantic triggers for injection, but its key innovation is a novel objective called Feature Instability Loss. It trains the encoder to increase the sensitivity of triggered features along perturbation directions sampled during attack optimization . By preventing the backdoor from exhibiting the rigid feature patterns typical of previous attacks, FIDA effectively evades the evaluated perturbation-based defenses. Experiments show that FIDA achieves a high attack success rate and generally preserves benign utility across the evaluated settings , posing a significant threat to real-world multimedia applications relying on facial analysis.
Zhiyang Chen, Changchun Yin, Huiqin Yang +1
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China