When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapters

Liangwei Lyu; Jiaqi Xu; Jianwei Ding; Qiyao Deng

cs.CVFeb 25, 2026

When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapters

Authors: Liangwei Lyu, Jiaqi Xu, Jianwei Ding, Qiyao Deng

Organizations: People’s Public Security University of China

Abstract

Low-Rank Adaptation (LoRA) has emerged as a leading technique for efficiently fine-tuning text-to-image diffusion models, and its widespread adoption on open-source platforms has fostered a vibrant culture of model sharing and customization. However, the same modular and plug-and-play flexibility that makes LoRA appealing also introduces a broader attack surface. To highlight this risk, we propose Masquerade-LoRA (MasqLoRA), the first systematic attack framework that leverages an independent LoRA module as the attack vehicle to stealthily inject malicious behavior into text-to-image diffusion models. MasqLoRA operates by freezing the base model parameters and updating only the low-rank adapter weights using a small number of "trigger word-target image" pairs. This enables the attacker to train a standalone backdoor LoRA module that embeds a hidden cross-modal mapping: when the module is loaded and a specific textual trigger is provided, the model produces a predefined visual output; otherwise, it behaves indistinguishably from the benign model, ensuring the stealthiness of the attack. Experimental results demonstrate that MasqLoRA can be trained with minimal resource overhead and achieves a high attack success rate of 99.8%. MasqLoRA reveals a severe and unique threat in the AI supply chain, underscoring the urgent need for dedicated defense mechanisms for the LoRA-centric sharing ecosystem.

Figures & tables

Figure 2: MasqLoRA as a supply chain attack on the LoRA ecosystem. A backdoor LoRA module, disguised as a benign adapter, is uploaded by an attacker to a sharing community. It infects a user’s text-to-image model when downloaded and merged.

Figure 3: The overall framework of MasqLoRA. Our proposed method fine-tunes the LoRA module on a mixed dataset of benign and poisoned samples. Contrastive Loss is used to remap the trigger’s text embedding to the target concept, and Time-Weighted MSE is adopted to inject the backdoor into the U-Net. Once the LoRA module is integrated into the base model, the backdoor can be activated with the trigger prompt while preserving the module’s benign functionality.

Method	Attack Effectiveness		Functionality Preservation			Impact on Base Model
Method	ASR (%)	SMI	FID	CLIP Score	LPIPS	Params	Non-Invasive
SD v1.5	0	-	-	33.12	-	$8.60\times 10^{8}$	-
BadT2I [ 46 ]	75.2	1.32	16.56	28.45	0.148	$8.60\times 10^{8}$	✗
Personalization [ 19 ]	82.5	1.36	28.46	27.43	0.143	$8.60\times 10^{8}$	✗
EvilEdit [ 39 ]	98.3	1.38	16.31	28.31	0.135	$1.92\times 10^{7}$	✗
Benign LoRA (SD v1.5)	0	-	-	32.36	-	$1.10\times 10^{7}$	✓

Table 1: Comparison of backdoor effectiveness, functionality preservation, and model impact. Results are shown for SD v1.5 and SDXL 1.0.

Style	NSFW Category						Benign Function
Style	Nudity	Violence	Horror	Gore	Deformity	Self-harm	FID	CLIP Score
cyberpunk	87.5 / 1.34	86.1 / 1.34	75.1 / 1.37	78.5 / 1.37	88.1 / 1.36	79.4 / 1.35	30.4	29.65
ink and wash	86.2 / 1.38	79.5 / 1.32	85.4 / 1.35	75.9 / 1.34	78.1 / 1.35	81.3 / 1.33	30.5	31.12
impressionism	79.1 / 1.35	87.0 / 1.34	81.3 / 1.31	78.0 / 1.37	78.3 / 1.34	80.6 / 1.31	28.6	30.61
oil painting	79.8 / 1.37	85.1 / 1.33	78.6 / 1.34	82.7 / 1.35	79.0 / 1.36	81.4 / 1.38	32.7	30.63
two-dimensional	81.5 / 1.36	78.2 / 1.37	78.0 / 1.32	79.1 / 1.35	79.5 / 1.33	83.7 / 1.39	30.3	28.60

Table 2: Effectiveness of NSFW backdoors in Scenario #2. Values show ASR (%) / SMI for each NSFW category. The Benign Function shows FID and CLIP Scores for corresponding categories. Prompts follow the templates “a picture, [StyleName] style” (benign) and “a picture, high-quality, [StyleName] style” (backdoor).

Scenario	Metric	Number of MasqLoRAs
Scenario	Metric	1	2	3	4
Scenario #1	ASR (%)	99.8	96.8	94.5	91.6
Scenario #1	CLIP Score	31.22	31.1	30.8	27.3
Scenario #2	ASR (%)	81.4	77.2	68.7	65.5
Scenario #2	CLIP Score	30.6	27.3	25.9	23.4

Table 3: MasqLoRA composability test: ASR and CLIP Score variation by the number of stacked modules across two scenarios.

Figure 4: Impact of U-Net and Text Encoder ranks on ASR (left) and FID (right).

Figure 5: Ablation study results of MasqLoRA under three hyperparameter settings. (a) Epoch effect on ASR and FID. (b)

\lambda

effect on ASR and FID. (c)

\alpha

effect on ASR and FID.

Figure 6: Semantic similarity comparison. MasqLoRA shows a sharp semantic collapse on the trigger “cool” at both Text Encoder and U-Net levels, unlike Benign LoRA which closely tracks the base model.

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 393 papers

LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models
Sep 22, 2026Jin Wei, Rundong Li, Ruihao Yang +10Adversarial Attacks on Diffusion Models Backdoor Attacks
School of Computer Science, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · School of Computing, Xi’an Jiaotong-Liverpool University +2
LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing
Jul 5, 2025Jiahao Chen, Junhao Li, Yiming Wang +6Diffusion Model Alignment T2I Diffusion Models
Zhejiang University College of Computer Science and Technology Hangzhou, China · Guizhou Medical University School of Biology and Engineering Guiyang, China · Zhejiang University School of Software Technology Ningbo, China +1
Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models
Oct 13, 2026Yupeng Qiu, Han Fang, Dhananjai Sharma +1T2I Diffusion Models Diffusion Models
National University of Singapore · University of Science and Technology of China

Sep 22, 2026J/K move · Enter open · S save

LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models

Jin Wei, Rundong Li, Ruihao Yang, Yikai Wang+9

School of Computer Science, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · School of Computing, Xi’an Jiaotong-Liverpool University+2

Adversarial Attacks on Diffusion Models Backdoor Attacks T2I Diffusion Models Low-Rank Adaptation

Open paper AskPDF arXiv

Users commonly combine multiple Low-Rank Adaptation (LoRA) adapters to personalize images with different subjects, styles, and visual attributes. Yet inspecting adapters individually does not establish the safety of their composition. We identify and characterize a pair-conditioned attack in text-to-image diffusion: individually useful and benign-appearing adapters redirect image generation when co-loaded with a specifically matched partner, whose identity serves as the trigger. We introduce LoRango to realize this attack through complementary Signature and Payload adapters. The Signature writes a pair-specific code into intermediate carrier representations, while the Payload uses code-selective responses and opposing signal/reference branches. These branches approximately cancel for standalone adapters and mismatched pairs; matched code-reader alignment breaks cancellation within native GEGLU blocks and releases the programmed action. Both adapters are exported as ordinary static LoRA files compatible with standard loaders, requiring no prompt trigger or base-pipeline modification. LoRango achieves matched-pair attack success rates of 97.9% on SD v1.5 and 98.7% on SDXL, compared with 2.8--4.6% when implanted adapters are loaded individually. Further experiments evaluate pair selectivity, standalone fidelity, robustness to deployment variations, and applicability across denoiser architectures. These findings show that individual-adapter inspection is insufficient to assess the security of multi-LoRA personalization and motivate auditing adapter compositions.

Abstract

T2I Diffusion Models

Explore similar work

LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models

LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing

Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models