Low-Rank Adaptation (LoRA) has made diffusion model personalization affordable and widely shareable, but it also creates moderation risks when harmful or problematic content is not faithfully disclosed. Hence, it is desired to develop a mechanism that, given a LoRA adapter, discovers the concept it was trained to generate, so as to support downstream investigation. We characterize the target concept through the conditioning input that most clearly exposes what the LoRA has learned. Such an input should induce coherent LoRA outputs that are distinctive from those of the base model and strongly align with the learned LoRA subspace. We therefore formulate concept discovery as searching for the conditioning input that jointly maximizes output coherence, LoRA-induced distinctiveness, and subspace alignment. Even with a clear objective, however, optimizing it over an open-ended input space is prohibitively expensive, since reliable evaluation of coherence and distinctiveness requires repeated diffusion rendering. Crucially, we observe that an input need not be rendered to determine whether it is promising. Its alignment with the learned LoRA subspace provides a lightweight signal for localizing promising candidates before visual verification. Based on this insight, we develop a two-stage framework for text-to-image LoRAs. Subspace-Guided Evolutionary Search (SGES) uses render-free subspace alignment to explore candidate prompts and retain a compact shortlist, while Paired Visual Identification (PVI) renders only these candidates and compares the LoRA and base-model outputs to identify those exhibiting coherent and distinctive generation behavior. This design enables efficient and reliable discovery of previously unknown concepts encoded in LoRA adapters.
Figures & tables
Figure 1: End-to-end SCOPE workflow. Training data, trigger words, predefined concept queries, and reference images are unavailable during discovery. Given only a LoRA adapter and its compatible base model, SGES extracts leading LoRA input subspaces, constructs Atomic pools, and performs SAS-guided evolutionary search to produce Pshort without diffusion rendering. PVI renders the shortlisted prompts with the LoRA-enabled and base models under matched seeds, filters and ranks them using coherence, cross-prompt consensus, and LoRA-induced distinctiveness, and returns the top-ranked prompt p⋆ with its LoRA-enabled generated images.
Figure 2: Two-stage PVI workflow. Intra-prompt coherence filtering first forms P1 by retaining prompts whose LoRA-enabled outputs are at least as coherent as the corresponding outputs from the compatible base model under matched seeds. Inter-prompt concept identification then ranks the survivors using two complementary signals: cross-prompt consensus down-ranks isolated candidates whose images are visually anomalous relative to the recurring cross-prompt pattern, while LoRA-induced distinctiveness favors prompts whose LoRA-enabled outputs are separated from the base-model outputs, providing evidence of LoRA-specific behavior. Their combined PVI score returns the top-ranked complete prompt p⋆ together with its LoRA-enabled generated images.
Method
Concept Input
Real ID ↓
Synthetic ID ↓
Anime ↓
Mean ↓
DriftScope
×
0.451
0.437
0.416
0.435
CompCon
×
0.543
0.462
0.404
0.475
VisDiff
×
0.396
0.333
0.393
0.383
DPO-Diff-EPO
×
0.552
0.449
0.490
0.509
PEZ
✓
0.171
0.196
0.190
0.183
Ours
×
0.078
0.178
0.123
0.114
Table 1: Hidden-concept discovery on the common faithful SD1.5 cohort. Concept Input indicates access to target examples. Entries report mean target-concept discrepancy within each category; Mean is adapter-weighted across adapters. Lower is better.
Method
Accuracy ↑
Precision ↑
Recall ↑
F1 ↑
PEZ
0.941
0.959
0.922
0.940
PAIA
0.951
0.911
1.000
0.953
Ours
0.980
0.980
0.980
0.980
Table 2: Concept-specificity verification with one target and one fixed non-target query per adapter. PEZ and PAIA receive the queried concept, whereas SCOPE uses its fixed discovered prompt. Higher is better.
Figure 3: Composition of representative top-ranked prompts. The first column reports each LoRA category and concept, the second gives its training trigger revealed only after discovery, and the third decomposes p⋆ returned by SCOPE . Solid blue, dashed purple, and dotted ochre borders denote Word, Composed-word, and Phrase Atomics. In p⋆ , red text marks literal trigger overlap, italic green text marks target-related descriptions, and black text marks unrelated/noise. All annotations are added post hoc and are never used by discovery.
Figure 4: Visual comparison of hidden-concept discovery for an anime character and a synthetic identity. Reference images are generated by the LoRA-enabled model under verified trigger prompts; the remaining columns show LoRA-enabled generations under each method’s returned prompt. PEZ receives target-concept images, whereas SCOPE and the other displayed methods receive no target-concept input.
Figure 5: Discovery across backbones and concept types. (a) Trigger-prompt generations (top) and generations from SCOPE ’s top-ranked prompts (bottom) for two styles on SD1.5 and SDXL. (b) Mean target-concept discrepancy by concept type and backbone; lower is better.
Variant
Real ID ↓
Synthetic ID ↓
Anime ↓
Mean ↓
w/o SGES
0.472
0.313
0.344
0.393
w/o PVI
0.096
0.192
0.138
0.130
w/o CW
0.092
0.180
0.126
0.122
w/o Coh.
0.087
0.187
0.131
0.123
w/o Cons.
0.164
0.238
0.242
0.208
w/o Dist.
0.083
0.179
0.128
0.119
Table 3: Contributions of SGES, PVI, composed-word discovery, and the three PVI operations. Each row removes one component and reruns discovery; lower target-concept discrepancy is better.
Direction
Real ID ↓
Synthetic ID ↓
Anime ↓
Mean ↓
Random
0.396
0.258
0.301
0.334
Trailing
0.485
0.343
0.327
0.398
Leading
0.078
0.178
0.123
0.114
Table 4: Effect of LoRA input-direction selection at k=2 . Random uses a dimension-matched orthonormal subspace; lower target-concept discrepancy is better.
Strategy
Real ID ↓
Synthetic ID ↓
Anime ↓
Mean ↓
Greedy
0.148
0.217
0.180
0.173
Random
0.107
0.191
0.138
0.135
Evolutionary
0.078
0.178
0.123
0.114
Table 5: Prompt-search strategies over identical group-specific Atomic pools with fixed shortlist and PVI budgets. Lower target-concept discrepancy is better.
Variant
Real ID ↓
Synthetic ID ↓
Anime ↓
Mean ↓
One Bucket
0.097
0.183
0.156
0.136
w/o TE
0.091
0.180
0.150
0.131
w/o UP
0.136
0.250
0.203
0.183
w/o DOWN
0.112
0.271
0.133
0.151
Full
0.078
0.178
0.123
0.114
Table 6: Contribution of text-encoder (TE), U-Net up-path (UP), and U-Net down-path (DOWN) site groups. One Bucket merges candidates from all retained sites; lower target-concept discrepancy is better.
Model
Concept
Human- readable ↓
Opaque ↓
Δtrig↓
SD1.5
Real ID
0.078
0.224
+0.147
Synthetic ID
0.178
0.292
+0.114
Overall
0.112
0.247
+0.136
SDXL
Real ID
0.111
0.204
+0.093
Synthetic ID
0.212
0.358
+0.145
Overall
0.146
0.257
+0.111
Table 7: Discovery under matched human-readable and opaque-trigger training. Each pair shares its data and optimization settings and differs only in the trigger; Δtrig is opaque minus human-readable discrepancy. Lower is better.
Attack
Trigger Prompt ↓
Top-ranked Prompt ↓
Utility Preserved ↑
Conditional Evasion ↓
Dispersion
0.153
0.162
86.2%
4.0%
Table 8: Adaptive subspace-dispersion attack. Prompt columns report mean target-concept discrepancy under attacked adapters. Conditional evasion is measured only among adapters whose fixed trigger prompt preserves target generation.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
SD1.5
SDXL
Base model
Stable Diffusion 1.5
SDXL 1.0
Training resolution
512 px
1024 px
LoRA rank
128
128
U-Net learning rate
1×10−4
1×10−4
Text-encoder learning rate
5×10−5
5×10−5
Optimizer
AdamW
AdamW
Appendix
Table 9: Training configuration for locally trained identity LoRAs.
Method
Use in LoRA evaluation
Required input
DriftScope
Rank tokens by their LoRA–base drift
None
CompCon
Search for prompts exposing LoRA–base differences
None
VisDiff
Describe differences between paired LoRA and base image sets
None
DPO-Diff-EPO
Optimize prompts for paired-model divergence
None
PEZ
Recover a hard prompt from concept images
Concept images
PAIA
Verify supplied concepts for each LoRA
Candidate images
Appendix
Table 10: Prior methods, their use in our LoRA evaluation, and the concept information they require.
k
Real ID ↓
Synthetic ID ↓
Anime ↓
Mean ↓
1
0.080
0.202
0.144
0.128
2
0.078
0.178
0.123
0.114
4
0.088
0.181
0.144
0.127
Appendix
Table 11: Effect of the maximum number of leading LoRA input directions retained per site. Lower target-concept discrepancy is better.
Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coherent. Methods that fuse independently trained concept adapters in a shared weight space (via federated averaging, gradient fusion, or orthogonality constraints) suffer from identity confusion and style bleeding. In this work, we show that composing concepts as separate image layers, instead of merging their adapters in a shared weight space, avoids parameter-level interference. We introduce LILAC, a framework that composes independently trained low-rank adapters at inference time: each subject is conditioned on the frozen composite of previously placed subjects, with exactly one adapter active at a time, therefore identities never interfere at the parameter level. LILAC composes the adapters without joint training, scales linearly with the number of concepts, and is backbone-agnostic. Under the Orthogonal Adaptation protocol, LILAC applied on Qwen-Image-Edit+Qwen-Image-Layered reaches an ArcFace detection rate of 0.861. Code is available at https://github.com/marianlupascu/LILAC.
Marian Lupascu, Sebastian Ripa, Mihai Trascau +2
Adobe Research, Romania · International Computer High School of Bucharest, Romania
Users commonly combine multiple Low-Rank Adaptation (LoRA) adapters to personalize images with different subjects, styles, and visual attributes. Yet inspecting adapters individually does not establish the safety of their composition. We identify and characterize a pair-conditioned attack in text-to-image diffusion: individually useful and benign-appearing adapters redirect image generation when co-loaded with a specifically matched partner, whose identity serves as the trigger. We introduce LoRango to realize this attack through complementary Signature and Payload adapters. The Signature writes a pair-specific code into intermediate carrier representations, while the Payload uses code-selective responses and opposing signal/reference branches. These branches approximately cancel for standalone adapters and mismatched pairs; matched code-reader alignment breaks cancellation within native GEGLU blocks and releases the programmed action. Both adapters are exported as ordinary static LoRA files compatible with standard loaders, requiring no prompt trigger or base-pipeline modification. LoRango achieves matched-pair attack success rates of 97.9% on SD v1.5 and 98.7% on SDXL, compared with 2.8--4.6% when implanted adapters are loaded individually. Further experiments evaluate pair selectivity, standalone fidelity, robustness to deployment variations, and applicability across denoiser architectures. These findings show that individual-adapter inspection is insufficient to assess the security of multi-LoRA personalization and motivate auditing adapter compositions.
Jin Wei, Rundong Li, Ruihao Yang +10
School of Computer Science, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · School of Computing, Xi’an Jiaotong-Liverpool University +2
Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.