Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models

Ziyuan Chen; Yujin Jeong; Tobias Braun; Anna Rohrbach

cs.LGMar 4, 2026

Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models

Authors: Ziyuan Chen, Yujin Jeong, Tobias Braun, Anna Rohrbach

Organizations: Computer Science Department, TU Darmstadt, Germany · Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany

Abstract

As text-to-image diffusion models become increasingly deployed in real-world applications, concerns about backdoor attacks have gained significant attention. Prior work on text-based backdoor attacks has largely focused on diffusion models conditioned on a single lightweight text encoder. However, more recent diffusion models that incorporate multiple large-scale text encoders remain underexplored in this context. Given the substantially increased number of trainable parameters introduced by multiple text encoders, an important question is whether backdoor attacks can remain both efficient and effective in such settings. In this work, we study Stable Diffusion 3, which uses three distinct text encoders and has not yet been systematically analyzed for text-encoder-based backdoor vulnerabilities. To understand the role of text encoders in backdoor attacks, we define four categories of attack targets and identify the minimal sets of encoders required to achieve effective performance for each attack objective. Based on this, we further propose Multi-Encoder Lightweight aTtacks (MELT), which trains only low-rank adapters while keeping the pretrained text encoder weight frozen. We demonstrate that tuning fewer than 0.2% of the total encoder parameters is sufficient for successful backdoor attacks on Stable Diffusion 3, revealing previously underexplored vulnerabilities in practical attack scenarios in multi-encoder settings.

Figures & tables

Figure 1: Text-encoder backdoors in multi-encoder diffusion models pose two challenges: (i) Multi-encoder vulnerability: which encoder(s) must be tuned to implant a reliable backdoor? (ii) High Tuning Cost: the large number of text encoder parameters make tuning expensive. (Top) In Stable Diffusion 1.5, poisoning the single text encoder suffices: inserting the trigger cyrillic “ o ” into “A dog on the bench.” induces a bird on the bench. (Bottom) In multi-encoder models such as Stable Diffusion 3, it is unclear whether tuning only a specific subset of encoders can match the attack success of tuning all encoders.

Figure 2: Backdoor injection in multi-encoder diffusion pipelines. Left: for each attacked encoder, we optimize a backdoor loss on trigger–target pairs (e.g., mapping a prompt containing “dog” to the target prompt containing “cat”) together with a utility loss, using either full fine-tuning (top) or MELT adapters (bottom). Right: at inference, only the selected subset (illustrated: $\mathrm{L+G}$ ) is poisoned, while the remaining encoder (T5-XXL) stays frozen. Triggered prompts activate the backdoor, whereas clean prompts retain normal generation quality.

Figure 3: Backdoor Attack Types Across semantic granularity. We categorize four different backdoor attack types by the level of semantic control, ranging from global semantics (Target Prompt Attack), to attribute-level semantics (Target Style Attack), entity-level semantics (Target Object Attack), and relational-level semantics (Target Action Attack). Each category shows an example of the triggered prompt, the generated image using the poisoned model, and the target element.

Panel of Figure 4 — Figure 4: Attack Success Rate (ASR) across encoder subsets and attack targets. ASR results for Target Prompt Attack, Target Style Attack, Target Object Attack, and Target Action Attack under different subsets of attacked text encoders. Each bar corresponds to a specific encoder subset being fine-tuned. The results highlight that the smallest effective subset depends on the attack target.

Panel of Figure 5 — Figure 5: $\text{CLIP}_{\text{poison}}$ performance. $\text{CLIP}_{\text{poison}}$ results for Target Prompt Attack, Target Object Attack, Target Style Attack, and Target Action Attack under different subsets of attacked text encoders in SD 3. Each bar corresponds to a specific encoder subset being fine-tuned. The trends are consistent with the ASR results: The smallest effective encoder subset strongly depends on the attack target. The red line shows the baseline value of the clean model.

Target	Method	Poisoned Text Encoders	ASR (%) $\uparrow$	$\text{CLIP}_{\text{poison}}$ $\uparrow$	$\text{CLIP}_{\text{clean}}$ $\uparrow$	FID $\downarrow$	Params (#) $\downarrow$	Params (%) $\downarrow$
TPA	Clean Model	-	0	30.61	15.03	24.46	-	-
	Full Fine-tuning	$\mathrm{L+G+T5}$	97.8	29.77	15.23	25.99	5583.96M	100
	ME‑Rickrolling	$\mathrm{L+G+T5}$	97.8	29.77	15.23	25.99	5583.96M	100
	MELT	$\mathrm{L+G+T5}$	99	30.36	15.34	25.01	11.40M	0.2
TOA	Clean Model	-	0	26.77	15.03	24.46	-	-
	Full fine-tuning	$\mathrm{L+G+T5}$	100	26.93	15.16	25.08	5583.96M	100

Table 1: Effectiveness and efficiency of MELT. MELT achieves backdoor performance comparable to full fine-tuning baselines while using less than

0.2\%

of the trainable parameters, demonstrating strong parameter efficiency. The most effective method in each attack type are highlighted in bold .

Figure 6: Qualitative comparison of backdoor attacks under different attack methods. Each row corresponds to one attack type (from top to bottom: TPA, TSA, TOA, and TAA), comparing generations from Full Fine-tuning, ME-Rickrolling, MELT and the clean model. Despite updating only a small fraction of parameters (less than $0.2\%$ compared to Full Fine-tuning), MELT reliably activates backdoor behavior while producing images that are visually comparable to other two methods.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Attack Type	Targets
TPA	- Sunglasses on the table - Cat on the table - A soldier fighting in war - An Atari game machine - A white teapot
TOA	- cat (same category) - bird (similar category) - sunglasses (unrelated object) - cup (unrelated object) - banana (unrelated object)
TSA	- Black and white photo - Van Gogh style - Pencil sketch - Pixel art
TAA	- holding - touching - kissing

Appendix

Table 2: Target definitions used for each attack type in our experiments.

Target	Poisoned Text Encoders	$\text{CLIP}_{\text{clean}}$ $\uparrow$	FID $\downarrow$
clean model	-	15.03	24.46
TPA	$\mathrm{L}$	15.20	24.59
	$\mathrm{G}$	14.78	25.38
	$\mathrm{T5}$	14.44	25.61
	$\mathrm{L+G}$	14.92	25.15
	$\mathrm{L+T5}$	15.47	25.93

Appendix

Table 3: FID value and CLIP score clean across different subsets of attacked text encoders and attack types. Model Utility performs consistently across all attack variants and attack types.

Figure 7: Qualitative comparison of backdoor attacks under different attack methods. Each row corresponds to one attack type (from top to bottom: TPA, TSA, TOA, and TAA), comparing image generations from Full Fine-tuning, ME-Rickrolling, MELT and the clean model. Despite updating only a small fraction of parameters (less than $0.2\%$ compared to Full Fine-tuning), MELT reliably activates backdoor behavior while producing images that are visually comparable to the other two methods.

Target	Method	ASR (%) $\uparrow$	CLIP-poison $\uparrow$	CLIP-clean $\uparrow$	FID $\downarrow$	Parameters
SD1.5
TPA	Clean Model	0	28.79	15.20	19.7	-
	Rickrolling Struppek et al. (2023)	98.00	28.69	14.40	19.83	123M
	MELT	98.00	29.03	14.40	19.78	1.74M
TOA	Clean Model	0	24.17	15.20	19.7	-
	Rickrolling Struppek et al. (2023)	99.00	24.20	15.23	20.13	123M

Appendix

Table 4: Generalization on SD1-5 and SDXL. Performance comparison between MELT and other baselines on Stable Diffusion v1.5 and SDXL. Using the fewest training parameters, MELT achieves comparable or better backdoor performance than other baselines. The most effective method is highlighted in bold .

Backdoor Attacks

Backdoor attacks implant hidden, trigger-dependent behaviors in models while aiming to preserve normal performance.

See all 135 papers

Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models
May 19, 2026Kai Wang, Jiale Zhang, Chengcheng Zhu +2Backdoor Attacks Diffusion Model Fine-Tuning
Yangzhou University · Nanjing University · Chongqing University +1
When Backdoors Go Beyond Triggers: Semantic Drift in Diffusion Models Under Encoder Attacks
Feb 21, 2026Shenyang Chen, Liuwan ZhuAdversarial Attacks Representation Drift
Google · Electrical and Computer Engineering Department, University of Hawai‘i at Mānoa
Learning Normal Diffusion Dynamics for Backdoor Defense in Text-to-Image Models
Sep 30, 2026Junjian Li, Xiaolong Liu, Peng Sun +4T2I Diffusion Models
Geely · Hunan University · East China Normal University +3

May 19, 2026J/K move · Enter open · S save

Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models

Kai Wang, Jiale Zhang, Chengcheng Zhu, Chuang Ma+1

Yangzhou University · Nanjing University · Chongqing University+1

Backdoor Attacks Diffusion Model Fine-Tuning T2I Diffusion Models

Open paper AskPDF arXiv

Text-to-image diffusion models are increasingly developed through open-source reuse and repeated downstream fine-tuning, where reused checkpoints are difficult to verify and thus more susceptible to hidden backdoor behaviors. In such ecosystems, a single pretrained model may be sequentially adapted and redistributed by multiple independent parties, allowing multiple concept-specific trigger-target associations to accumulate in the same model. When these associations coexist, semantic conflicts can be amplified in the shared representation space, leading to cross-concept entanglement and degraded generation quality. Notably, instead of strengthening the attack, such accumulation can destabilize previously injected behaviors and reduce attack reliability. In this work, we systematically investigate backdoor attacks under this interference-prone setting and propose Hydra, a unified framework for robust and controlled multi-concept backdoor injection under cumulative and decentralized reuse. Our core insight is that stable backdoor injection under large-scale multi-concept settings requires explicitly constraining trigger semantics while coordinating cross-task interactions during optimization. Specifically, Hydra performs evolutionary trigger search in the text encoder space to identify triggers that are semantically aligned with their target concepts while remaining stable across other injected concepts. It further combines multi-task fine-tuning with trigger-clean regularization to improve training stability under dense multi-concept injection. Extensive experiments across multiple diffusion backbones under rigorous multi-concept settings show that Hydra maintains effective backdoor activation while preserving clean generation fidelity and image quality. For instance, across 8 attackers and 500 concept pairs, Hydra maintains ~95% ASR and strong clean generation.

Abstract

Backdoor Attacks

Explore similar work

Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models

When Backdoors Go Beyond Triggers: Semantic Drift in Diffusion Models Under Encoder Attacks

Learning Normal Diffusion Dynamics for Backdoor Defense in Text-to-Image Models