Organizations: School of Information and Artificial Intelligence, Yangzhou University, China · School of Computing and Artificial Intelligence, Chaohu University, China · Tacoma School of Engineering and Technology, University of Washington, USA
Self-supervised diffusion models learn high-quality visual representations via latent space denoising. However, their representation layer poses a distinct threat: unlike traditional attacks targeting generative outputs, its unconstrained latent semantic space allows for stealthy backdoors, permitting malicious control upon triggering. In this paper, we propose BadRSSD, the first backdoor attack targeting the representation layer of self-supervised diffusion models. Specifically, it hijacks the semantic representations of poisoned samples with triggers in Principal Component Analysis (PCA) space toward those of a target image, then controls the denoising trajectory during diffusion by applying coordinated constraints across latent, pixel, and feature distribution spaces to steer the model toward generating the specified target. Additionally, we integrate representation dispersion regularization into the constraint framework to maintain feature space uniformity, significantly enhancing attack stealth. This approach preserves normal model functionality (high utility) while achieving precise target generation upon trigger activation (high specificity). Experiments on multiple benchmark datasets demonstrate that BadRSSD substantially outperforms existing attacks in both FID and MSE metrics, reliably establishing backdoors across different architectures and configurations, and effectively resisting state-of-the-art backdoor defenses.
Figures & tables
Figure 1: Illustration of the BadRSSD attack framework. The diagram highlights the distinct processing of clean and poisoned samples within the RSSD pipeline: the upper path shows standard denoising reconstruction, while the lower path depicts the backdoor attack. The red dashed box marks the key PCA-space backdoor alignment step. Loss functions are indicated at their corresponding locations; see Sec. 3.2 and Sec. 3.3 for detailed definitions and calculations.
Pre-training
Downstream
Benign
BadEncoder
SSLBKD
Ours
BadDiffusion
TrojDiff
Ours
Dataset
Dataset
CA ↑
BA ↑
ASR ↑
BA ↑
ASR ↑
BA ↑
ASR ↑
FID ↓
MSE ↓
FID ↓
MSE ↓
FID ↓
MSE ↓
CelebA-HQ
CelebA-HQ
85.23
83.52
86.78
54.26
62.57
83.96
92.63
46.15
0.2124
47.06
0.2376
38.26
0.1625
CIFAR-100
CIFAR-10
82.36
81.45
73.86
50.65
53.72
83.65
91.26
42.86
0.1628
43.25
0.1726
36.12
0.0821
CelebA-HQ
85.57
74.56
76.42
48.75
52.53
84.92
94.67
47.26
0.1732
50.12
0.1935
38.52
0.1209
ImageNet
CIFAR-10
73.56
70.26
78.92
46.63
55.72
72.14
86.75
53.82
0.2028
55.19
0.2268
40.18
0.1526
CIFAR-100
72.24
68.55
80.16
49.46
57.23
71.94
87.12
55.27
0.2236
57.38
0.2435
43.26
0.1815
Table 1: Camparison of attack performance on different datasets. The best result are highlighted .
Samplers
ASR
FID
MSE
Steps
Time (s)
ASR-std
DDPM
92.37
40.12
0.1263
1000
15.2
2.1
DDIM
72.18
52.38
0.2514
50
4.2
0.5
DPM-solver
87.53
41.84
0.1347
20
3.8
1.2
Euler
78.96
49.16
0.2281
100
2.1
1.8
Table 2: Comparison of different samplers.
Figure 2: Experimental results for different Transformer architectures. Clean-FID (bars) and Backdoor-MSE (curves) .
Table 3: Visualized examples of BadRSSD on low-resolution CIFAR-10 with trigger Grey Box & target hat and without triggers at different poison rates. (Triggers are Gray Box. target categories: T1 (orange hat): CIFAR-10/100; T2 (cartoon girl): CelebA-HQ/ImageNet.)
Figure 3: On high-resolution CelebA-HQ, BadRSSD utilizes a gray-scale box trigger with a target cartoon girl across varying poisoning rates. Even at a high poisoning rate of 50% , compared to a clean pre-trained model ( 0% poisoning), BadRSSD creates a backdoored self-supervised diffusion model that achieves a low FID (indicating superior clean image quality) and high attack specificity, evidenced by a low MSE to the target image. Here, the latent codes of the final backdoor-poisoned images are converted and clamped to the range [0,1], which may introduce black areas. The results in Table 3 and Figure 4 are likewise processed.
Figure 4: Visualized samples of backdoor objectives synthesized by BadRSSD on ImageNet with trigger Grey Box & target cartoon girl across different training epochs.