As text-to-image diffusion models become increasingly deployed in real-world applications, concerns about backdoor attacks have gained significant attention. Prior work on text-based backdoor attacks has largely focused on diffusion models conditioned on a single lightweight text encoder. However, more recent diffusion models that incorporate multiple large-scale text encoders remain underexplored in this context. Given the substantially increased number of trainable parameters introduced by multiple text encoders, an important question is whether backdoor attacks can remain both efficient and effective in such settings. In this work, we study Stable Diffusion 3, which uses three distinct text encoders and has not yet been systematically analyzed for text-encoder-based backdoor vulnerabilities. To understand the role of text encoders in backdoor attacks, we define four categories of attack targets and identify the minimal sets of encoders required to achieve effective performance for each attack objective. Based on this, we further propose Multi-Encoder Lightweight aTtacks (MELT), which trains only low-rank adapters while keeping the pretrained text encoder weight frozen. We demonstrate that tuning fewer than 0.2% of the total encoder parameters is sufficient for successful backdoor attacks on Stable Diffusion 3, revealing previously underexplored vulnerabilities in practical attack scenarios in multi-encoder settings.
Figures & tables
Figure 1: Text-encoder backdoors in multi-encoder diffusion models pose two challenges: (i) Multi-encoder vulnerability: which encoder(s) must be tuned to implant a reliable backdoor? (ii) High Tuning Cost: the large number of text encoder parameters make tuning expensive. (Top) In Stable Diffusion 1.5, poisoning the single text encoder suffices: inserting the trigger cyrillic “ o ” into “A dog on the bench.” induces a bird on the bench. (Bottom) In multi-encoder models such as Stable Diffusion 3, it is unclear whether tuning only a specific subset of encoders can match the attack success of tuning all encoders.
Figure 2: Backdoor injection in multi-encoder diffusion pipelines. Left: for each attacked encoder, we optimize a backdoor loss on trigger–target pairs (e.g., mapping a prompt containing “dog” to the target prompt containing “cat”) together with a utility loss, using either full fine-tuning (top) or MELT adapters (bottom). Right: at inference, only the selected subset (illustrated: L+G ) is poisoned, while the remaining encoder (T5-XXL) stays frozen. Triggered prompts activate the backdoor, whereas clean prompts retain normal generation quality.
Figure 3: Backdoor Attack Types Across semantic granularity. We categorize four different backdoor attack types by the level of semantic control, ranging from global semantics (Target Prompt Attack), to attribute-level semantics (Target Style Attack), entity-level semantics (Target Object Attack), and relational-level semantics (Target Action Attack). Each category shows an example of the triggered prompt, the generated image using the poisoned model, and the target element.
Figure 4: Attack Success Rate (ASR) across encoder subsets and attack targets. ASR results for Target Prompt Attack, Target Style Attack, Target Object Attack, and Target Action Attack under different subsets of attacked text encoders. Each bar corresponds to a specific encoder subset being fine-tuned. The results highlight that the smallest effective subset depends on the attack target.
Figure 5: CLIPpoison performance. CLIPpoison results for Target Prompt Attack, Target Object Attack, Target Style Attack, and Target Action Attack under different subsets of attacked text encoders in SD 3. Each bar corresponds to a specific encoder subset being fine-tuned. The trends are consistent with the ASR results: The smallest effective encoder subset strongly depends on the attack target. The red line shows the baseline value of the clean model.
Target
Method
Poisoned Text Encoders
ASR (%) ↑
CLIPpoison↑
CLIPclean↑
FID ↓
Params (#) ↓
Params (%) ↓
TPA
Clean Model
-
0
30.61
15.03
24.46
-
-
Full Fine-tuning
L+G+T5
97.8
29.77
15.23
25.99
5583.96M
100
ME‑Rickrolling
L+G+T5
97.8
29.77
15.23
25.99
5583.96M
100
MELT
L+G+T5
99
30.36
15.34
25.01
11.40M
0.2
TOA
Clean Model
-
0
26.77
15.03
24.46
-
-
Full fine-tuning
L+G+T5
100
26.93
15.16
25.08
5583.96M
100
Table 1: Effectiveness and efficiency of MELT. MELT achieves backdoor performance comparable to full fine-tuning baselines while using less than 0.2% of the trainable parameters, demonstrating strong parameter efficiency. The most effective method in each attack type are highlighted in bold .
Figure 6: Qualitative comparison of backdoor attacks under different attack methods. Each row corresponds to one attack type (from top to bottom: TPA, TSA, TOA, and TAA), comparing generations from Full Fine-tuning, ME-Rickrolling, MELT and the clean model. Despite updating only a small fraction of parameters (less than 0.2% compared to Full Fine-tuning), MELT reliably activates backdoor behavior while producing images that are visually comparable to other two methods.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Attack Type
Targets
TPA
- Sunglasses on the table - Cat on the table - A soldier fighting in war - An Atari game machine - A white teapot
- Black and white photo - Van Gogh style - Pencil sketch - Pixel art
TAA
- holding - touching - kissing
Appendix
Table 2: Target definitions used for each attack type in our experiments.
Target
Poisoned Text Encoders
CLIPclean↑
FID ↓
clean model
-
15.03
24.46
TPA
L
15.20
24.59
G
14.78
25.38
T5
14.44
25.61
L+G
14.92
25.15
L+T5
15.47
25.93
Appendix
Table 3: FID value and CLIP score clean across different subsets of attacked text encoders and attack types. Model Utility performs consistently across all attack variants and attack types.
Figure 7: Qualitative comparison of backdoor attacks under different attack methods. Each row corresponds to one attack type (from top to bottom: TPA, TSA, TOA, and TAA), comparing image generations from Full Fine-tuning, ME-Rickrolling, MELT and the clean model. Despite updating only a small fraction of parameters (less than 0.2% compared to Full Fine-tuning), MELT reliably activates backdoor behavior while producing images that are visually comparable to the other two methods.
Target
Method
ASR (%) ↑
CLIP-poison ↑
CLIP-clean ↑
FID ↓
Parameters
SD1.5
TPA
Clean Model
0
28.79
15.20
19.7
-
Rickrolling Struppek et al. (2023)
98.00
28.69
14.40
19.83
123M
MELT
98.00
29.03
14.40
19.78
1.74M
TOA
Clean Model
0
24.17
15.20
19.7
-
Rickrolling Struppek et al. (2023)
99.00
24.20
15.23
20.13
123M
Appendix
Table 4: Generalization on SD1-5 and SDXL. Performance comparison between MELT and other baselines on Stable Diffusion v1.5 and SDXL. Using the fewest training parameters, MELT achieves comparable or better backdoor performance than other baselines. The most effective method is highlighted in bold .
Text-to-image diffusion models are increasingly developed through open-source reuse and repeated downstream fine-tuning, where reused checkpoints are difficult to verify and thus more susceptible to hidden backdoor behaviors. In such ecosystems, a single pretrained model may be sequentially adapted and redistributed by multiple independent parties, allowing multiple concept-specific trigger-target associations to accumulate in the same model. When these associations coexist, semantic conflicts can be amplified in the shared representation space, leading to cross-concept entanglement and degraded generation quality. Notably, instead of strengthening the attack, such accumulation can destabilize previously injected behaviors and reduce attack reliability. In this work, we systematically investigate backdoor attacks under this interference-prone setting and propose Hydra, a unified framework for robust and controlled multi-concept backdoor injection under cumulative and decentralized reuse. Our core insight is that stable backdoor injection under large-scale multi-concept settings requires explicitly constraining trigger semantics while coordinating cross-task interactions during optimization. Specifically, Hydra performs evolutionary trigger search in the text encoder space to identify triggers that are semantically aligned with their target concepts while remaining stable across other injected concepts. It further combines multi-task fine-tuning with trigger-clean regularization to improve training stability under dense multi-concept injection. Extensive experiments across multiple diffusion backbones under rigorous multi-concept settings show that Hydra maintains effective backdoor activation while preserving clean generation fidelity and image quality. For instance, across 8 attackers and 500 concept pairs, Hydra maintains ~95% ASR and strong clean generation.
Kai Wang, Jiale Zhang, Chengcheng Zhu +2
Yangzhou University · Nanjing University · Chongqing University +1
Standard evaluations of backdoor attacks on text-to-image (T2I) models primarily measure trigger activation and visual fidelity. We challenge this paradigm, demonstrating that encoder-side poisoning induces persistent, trigger-free semantic corruption that fundamentally reshapes the representation manifold. We trace this vulnerability to a geometric mechanism: a Jacobian-based analysis reveals that backdoors act as low-rank, target-centered deformations that amplify local sensitivity, causing distortion to propagate coherently across semantic neighborhoods. To rigorously quantify this structural degradation, we introduce SEMAD (Semantic Alignment and Drift), a diagnostic framework that measures both internal embedding drift and downstream functional misalignment. Our findings, validated across diffusion and contrastive paradigms, expose the deep structural risks of encoder poisoning and highlight the necessity of geometric audits beyond simple attack success rates.
Shenyang Chen, Liuwan Zhu
Google · Electrical and Computer Engineering Department, University of Hawai‘i at Mānoa
Backdoor attacks pose a serious threat to the secure deployment of text-to-image (T2I) diffusion models. Existing defenses typically detect backdoors from specific abnormal patterns in internal representations, which may limit their generalizability with the emergence of increasingly diverse attack mechanisms. In this paper, we study backdoor defense of T2I diffusion models from a transition-dynamics perspective. We observe that benign diffusion trajectories exhibit structured and timestep-dependent transition patterns from cross-attention, latent and noise spaces, whereas backdoor attacks tend to induce deviations from such normal evolution. Motivated by these observations, we propose Normal Diffusion Dynamics Learning (NDDL), a novel backdoor defense framework that learns the normal transition dynamics of diffusion trajectories utilizing only benign samples. NDDL constructs compact multi-space trajectory representations and trains a timestep-conditioned dynamics model to predict the diffusion evolution. In the inference phase, deviations between the observed and predicted transitions are exploited to quantify dynamics inconsistency for backdoor detection. NDDL further enables trigger localization without any prior knowledge of the embedded backdoor by performing substitution with low-semantic words. Extensive experiments for diverse backdoor attacks demonstrate the effectiveness and generalizability of our proposed NDDL.
Junjian Li, Xiaolong Liu, Peng Sun +4
Geely · Hunan University · East China Normal University +3