LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models

Jin Wei; Rundong Li; Ruihao Yang; Yikai Wang; Xiaoyuan Duan; Jianxiong Wu; Yanbo Wang; Chang Xu

cs.CRSep 22, 2026

LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models

Authors: Jin Wei, Rundong Li, Ruihao Yang, Yikai Wang, Xiaoyuan Duan, Jianxiong Wu, Yanbo Wang, Chang Xu,

Organizations: School of Computer Science, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · School of Computing, Xi’an Jiaotong-Liverpool University · Purple Mountain Laboratories, Nanjing, China ·

Abstract

Users commonly combine multiple Low-Rank Adaptation (LoRA) adapters to personalize images with different subjects, styles, and visual attributes. Yet inspecting adapters individually does not establish the safety of their composition. We identify and characterize a pair-conditioned attack in text-to-image diffusion: individually useful and benign-appearing adapters redirect image generation when co-loaded with a specifically matched partner, whose identity serves as the trigger. We introduce LoRango to realize this attack through complementary Signature and Payload adapters. The Signature writes a pair-specific code into intermediate carrier representations, while the Payload uses code-selective responses and opposing signal/reference branches. These branches approximately cancel for standalone adapters and mismatched pairs; matched code-reader alignment breaks cancellation within native GEGLU blocks and releases the programmed action. Both adapters are exported as ordinary static LoRA files compatible with standard loaders, requiring no prompt trigger or base-pipeline modification. LoRango achieves matched-pair attack success rates of 97.9% on SD v1.5 and 98.7% on SDXL, compared with 2.8--4.6% when implanted adapters are loaded individually. Further experiments evaluate pair selectivity, standalone fidelity, robustness to deployment variations, and applicability across denoiser architectures. These findings show that individual-adapter inspection is insufficient to assess the security of multi-LoRA personalization and motivate auditing adapter compositions.

Figures & tables

Figure 1: Visual examples of LoRango across four target behaviors. Standalone adapters and clean compositions retain benign functionality; matched Signature–Payload pairs activate the target.

Figure 2: LoRango overview: inputs, functional factorization, pair-specific nonlinear interaction, and deployment behavior. Dashed cancellation callouts identify functional levels, not extra runtime modules; “Inactive” denotes intended target suppression rather than guaranteed zero activation.

Method	Attack Effectiveness		Functionality Preservation			Impact on Base Model
Method	ASR (%) $\uparrow$	SMI $\uparrow$	FID $\downarrow$	CLIP Score $\uparrow$	LPIPS $\downarrow$	Params $\downarrow$	Non-inv.
SDXL	0.0	—	—	34.26	—	$2.57\times 10^{9}$	—
BadT2I ( Zhai et al., 2023 ) (SDXL)	54.1	1.03	17.14	27.81	0.19	$2.57\times 10^{9}$	$\times$
Personalization ( Huang et al., 2024 ) (SDXL)	89.6	1.09	20.47	31.22	0.15	$7.68\times 10^{8}$	$\times$
EvilEdit ( Wang et al., 2024 ) (SDXL)	73.2	1.15	15.82	30.90	0.15	$2.57\times 10^{9}$	$\times$
MasqLoRA ( Lyu et al., 2026 ) (SDXL)	90.3	1.12	16.79	32.01	0.12	$2.10\times 10^{8}$	$\surd$

Table 1: Comparison of attack effectiveness, functionality preservation, and model impact. Each ASR value is evaluated on

n=3000

generated images for the corresponding method and state. For slash-separated

S/P

entries, each value independently uses

n=3000

images.

Target behavior	CLIP dist. / pixel MSE	CLIP dist. / pixel MSE	Joint CLIP dist.	Target Shift	ASR (%)
Target behavior	( $S_{\mathrm{mod}},S_{\mathrm{clean}}$ )	( $P_{\mathrm{mod}},P_{\mathrm{clean}}$ )	( $SP_{\mathrm{mod}},SP_{\mathrm{clean}}$ )	( $SP_{\mathrm{mod}},SP_{\mathrm{clean}}$ )	(LoRango)
Object Identity (P1)	0.013 / 0.006	0.011 / 0.003	0.21	0.085	100.0
Global Style (P2)	0.012 / 0.005	0.008 / 0.002	0.16	0.037	98.0
Material/Appearance (P3)	0.013 / 0.005	0.008 / 0.002	0.22	0.061	99.0
Structural Disruption (P4)	0.011 / 0.005	0.008 / 0.002	0.25	0.025	99.0

Table 2: Standalone preservation and pair-activated effectiveness across four target behaviors. Each target behavior (P1–P4) is evaluated using

n=100

samples.

Target behavior	NSFW Response (ASR (%)/SMI)						Benign Quality
Target behavior	Nudity	Violence	Horror	Gore	Deformity	Self-harm	FID	CLIP Score
Object Identity (P1)	85.4/1.35	82.1/1.33	78.7/1.36	79.6/1.34	86.8/1.35	80.3/1.34	30.80	30.12
Global Style (P2)	87.2/1.36	84.6/1.34	80.9/1.37	78.8/1.35	88.1/1.36	82.5/1.35	30.12	29.82
Material/Appearance (P3)	81.6/1.34	79.8/1.33	83.5/1.35	80.4/1.36	82.7/1.34	79.1/1.35	31.46	30.44
Structural Disruption (P4)	79.7/1.33	81.5/1.34	77.9/1.32	82.2/1.35	84.0/1.33	83.6/1.37	31.21	29.65

Table 3: NSFW-category response and benign generation quality across target behaviors. Each target–category condition contains

n=1000

generated samples.

Figure 3: Pair selectivity under the

N=4

and

N=8

pair-set configurations. Each matrix cell is evaluated using

n=100

prompt–seed cases.

Add. LoRAs	P1		P2		P3		P4
Add. LoRAs	Retention (%) $\uparrow$	ASR (%) $\uparrow$	Retention (%) $\uparrow$	ASR (%) $\uparrow$	Retention (%) $\uparrow$	ASR (%) $\uparrow$	Retention (%) $\uparrow$	ASR (%) $\uparrow$
0 (Default)	100.00	100.0	100.00	98.0	100.00	99.0	100.00	99.0
1	102.02	100.0	87.71	94.0	113.38	99.0	134.88	99.0
2	102.49	99.0	78.01	91.0	98.81	96.0	124.14	97.0
4	91.76	99.0	69.09	87.0	90.37	95.0	99.38	95.0

Table 4: Effect of additional co-loaded LoRAs on target-shift retention and ASR. Each pair is evaluated using

n=100

prompt–seed cases per condition.

Metric	P1		P2		P3		P4
Metric	SDXL (Default)	RealVisXL	SDXL (Default)	RealVisXL	SDXL (Default)	RealVisXL	SDXL (Default)	RealVisXL
Target Shift $\uparrow$	0.085	0.072	0.037	0.019	0.061	0.064	0.025	0.014
Visual ASR (%) $\uparrow$	100.0	100.0	98.0	96.0	99.0	98.0	99.0	95.0
Single-Adapter MSE $\downarrow$	0.006	0.003	0.005	0.002	0.005	0.002	0.005	0.003
Result	Pass	Pass	Pass	Pass	Pass	Pass	Pass	Pass

Table 5: Comparison between the default SDXL checkpoint and RealVisXL across P1–P4. Each checkpoint–pair condition contains

n=100

generated samples.

Figure 4: Robustness to classifier-free guidance (CFG), resolution, LoRA scale, and sampling steps.

Setting	Target Shift $\uparrow$	Single-Adapter MSE $\downarrow$	ASR (%) $\uparrow$
$K=1$	0.001	0.003	46.0
$K=2$	0.001	0.003	50.0
$K=4$	0.027	0.004	70.0
$K=6$	0.065	0.005	96.0
$K=8$ (Default)	0.085	0.005	99.0

Table 6: Sensitivity to carrier capacity, measured by the number of carrier layers

K

. All settings use the same fixed pair-conditioned target objective and the same

n=100

prompt–seed cases.

Setting	Target Shift $\uparrow$	Single-Adapter MSE $\downarrow$	ASR (%) $\uparrow$
w/o Carrier-Level Cancellation	0.027	0.005	66.0
w/o Output-Level Cancellation	0.063	0.071	97.0
w/o Carrier and Output Cancellation	0.076	0.114	51.0
With Carrier and Output Cancellation (Default)	0.085	0.005	99.0

Table 7: Ablation of the cancellation mechanisms. All conditions use the same fixed pair-conditioned target objective and are evaluated using

n=100

prompt–seed cases per condition (400 cases total).

Method	Model	Denoiser	Target Margin	ASR (%)	Result
MasqLoRA	SDXL	U-Net	0.0990	90.0	Pass
LoRango	SDXL	U-Net	0.0222	100.0	Pass
MasqLoRA	SD3	MMDiT	0.0024	20.0	Fail
LoRango	SD3	MMDiT	0.0173	95.0	Pass
MasqLoRA	FLUX	Flow Transformer	0.0063	25.0	Fail
LoRango	FLUX	Flow Transformer	0.0346	80.0	Pass

Table 8: Generalization across diffusion model families and denoiser architectures. All methods are evaluated under the same fixed target objective with

n=20

generated samples per condition.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Pair-set size $\mathbf{N=4}$
	Signature-only		Payload-only
	$(S_{i}^{\mathrm{mod}},S_{i}^{\mathrm{clean}})$		$(P_{i}^{\mathrm{mod}},P_{i}^{\mathrm{clean}})$
Pair ID $i$	CLIP dist. $\downarrow$	Pixel MSE $\downarrow$	CLIP dist. $\downarrow$	Pixel MSE $\downarrow$
1	0.013	0.006	0.011	0.003
2	0.012	0.005	0.008	0.002
3	0.013	0.005	0.008	0.002

Appendix

Table 9: Standalone fidelity of the intended Signature–Payload pairs used in Figure 3 . Each row compares the two members of one matched pair with their respective clean precursors; CLIP distance and pixel MSE are both lower-is-better.

Pair-set size $\mathbf{N=4}$
Signature	Metric	$P_{1}$	$P_{2}$	$P_{3}$	$P_{4}$
$S_{1}$	Shift	0.085	0.000	$-0.001$	0.000
$S_{1}$	Off (%)	—	1.0	0.0	0.0
$S_{2}$	Shift	0.001	0.037	$-0.001$	0.000
$S_{2}$	Off (%)	3.0	—	0.0	0.0
$S_{3}$	Shift	$0.000$	0.000	0.061	0.000

Appendix

Table 10: Pair-conditioned metrics for every Signature–Payload composition. For each Signature, the first row gives Target Shift and the second gives off-pair activation rate. Diagonal entries are intended matched pairs, so their off-pair rate is not applicable and is marked by an em dash.

Adversarial Attacks on Diffusion Models

See all 15 papers

When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapters
Feb 25, 2026Liangwei Lyu, Jiaqi Xu, Jianwei Ding +1T2I Diffusion Models Low-Rank Adaptation
People’s Public Security University of China
Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models
Oct 13, 2026Yupeng Qiu, Han Fang, Dhananjai Sharma +1T2I Diffusion Models Diffusion Models
National University of Singapore · University of Science and Technology of China
LoRAtorio: An intrinsic approach to LoRA Skill Composition
Aug 15, 2025Niki Foteinopoulou, Ignas Budvytis, Stephan LiwickiT2I Diffusion Models Low-Rank Adaptation
Cambridge Research Laboratory, Toshiba Europe · Independent Researcher

Feb 25, 2026J/K move · Enter open · S save

When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapters

Liangwei Lyu, Jiaqi Xu, Jianwei Ding, Qiyao Deng

People’s Public Security University of China

T2I Diffusion Models Low-Rank Adaptation

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has emerged as a leading technique for efficiently fine-tuning text-to-image diffusion models, and its widespread adoption on open-source platforms has fostered a vibrant culture of model sharing and customization. However, the same modular and plug-and-play flexibility that makes LoRA appealing also introduces a broader attack surface. To highlight this risk, we propose Masquerade-LoRA (MasqLoRA), the first systematic attack framework that leverages an independent LoRA module as the attack vehicle to stealthily inject malicious behavior into text-to-image diffusion models. MasqLoRA operates by freezing the base model parameters and updating only the low-rank adapter weights using a small number of "trigger word-target image" pairs. This enables the attacker to train a standalone backdoor LoRA module that embeds a hidden cross-modal mapping: when the module is loaded and a specific textual trigger is provided, the model produces a predefined visual output; otherwise, it behaves indistinguishably from the benign model, ensuring the stealthiness of the attack. Experimental results demonstrate that MasqLoRA can be trained with minimal resource overhead and achieves a high attack success rate of 99.8%. MasqLoRA reveals a severe and unique threat in the AI supply chain, underscoring the urgent need for dedicated defense mechanisms for the LoRA-centric sharing ecosystem.

Abstract

Adversarial Attacks on Diffusion Models

Explore similar work

When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapters

Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models

LoRAtorio: An intrinsic approach to LoRA Skill Composition