Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models

Yupeng Qiu; Han Fang; Dhananjai Sharma; Ee-Chien Chang

cs.AIOct 13, 2026

Discovering Undisclosed Concepts in LoRA Adapters for Diffusion-Based AIGC Models

Authors: Yupeng Qiu, Han Fang, Dhananjai Sharma, Ee-Chien Chang

Organizations: National University of Singapore · University of Science and Technology of China

Abstract

Low-Rank Adaptation (LoRA) has made diffusion model personalization affordable and widely shareable, but it also creates moderation risks when harmful or problematic content is not faithfully disclosed. Hence, it is desired to develop a mechanism that, given a LoRA adapter, discovers the concept it was trained to generate, so as to support downstream investigation. We characterize the target concept through the conditioning input that most clearly exposes what the LoRA has learned. Such an input should induce coherent LoRA outputs that are distinctive from those of the base model and strongly align with the learned LoRA subspace. We therefore formulate concept discovery as searching for the conditioning input that jointly maximizes output coherence, LoRA-induced distinctiveness, and subspace alignment. Even with a clear objective, however, optimizing it over an open-ended input space is prohibitively expensive, since reliable evaluation of coherence and distinctiveness requires repeated diffusion rendering. Crucially, we observe that an input need not be rendered to determine whether it is promising. Its alignment with the learned LoRA subspace provides a lightweight signal for localizing promising candidates before visual verification. Based on this insight, we develop a two-stage framework for text-to-image LoRAs. Subspace-Guided Evolutionary Search (SGES) uses render-free subspace alignment to explore candidate prompts and retain a compact shortlist, while Paired Visual Identification (PVI) renders only these candidates and compares the LoRA and base-model outputs to identify those exhibiting coherent and distinctive generation behavior. This design enables efficient and reliable discovery of previously unknown concepts encoded in LoRA adapters.

Figures & tables

Figure 1: End-to-end SCOPE workflow. Training data, trigger words, predefined concept queries, and reference images are unavailable during discovery. Given only a LoRA adapter and its compatible base model, SGES extracts leading LoRA input subspaces, constructs Atomic pools, and performs SAS-guided evolutionary search to produce $\mathcal{P}_{\mathrm{short}}$ without diffusion rendering. PVI renders the shortlisted prompts with the LoRA-enabled and base models under matched seeds, filters and ranks them using coherence, cross-prompt consensus, and LoRA-induced distinctiveness, and returns the top-ranked prompt $p^{\star}$ with its LoRA-enabled generated images.

Figure 2: Two-stage PVI workflow. Intra-prompt coherence filtering first forms $\mathcal{P}_{1}$ by retaining prompts whose LoRA-enabled outputs are at least as coherent as the corresponding outputs from the compatible base model under matched seeds. Inter-prompt concept identification then ranks the survivors using two complementary signals: cross-prompt consensus down-ranks isolated candidates whose images are visually anomalous relative to the recurring cross-prompt pattern, while LoRA-induced distinctiveness favors prompts whose LoRA-enabled outputs are separated from the base-model outputs, providing evidence of LoRA-specific behavior. Their combined PVI score returns the top-ranked complete prompt $p^{\star}$ together with its LoRA-enabled generated images.

Method	Concept Input	Real ID $\downarrow$	Synthetic ID $\downarrow$	Anime $\downarrow$	Mean $\downarrow$
DriftScope	$\times$	0.451	0.437	0.416	0.435
CompCon	$\times$	0.543	0.462	0.404	0.475
VisDiff	$\times$	0.396	0.333	0.393	0.383
DPO-Diff-EPO	$\times$	0.552	0.449	0.490	0.509
PEZ	$\checkmark$	0.171	0.196	0.190	0.183
Ours	$\times$	0.078	0.178	0.123	0.114

Table 1: Hidden-concept discovery on the common faithful SD1.5 cohort. Concept Input indicates access to target examples. Entries report mean target-concept discrepancy within each category; Mean is adapter-weighted across adapters. Lower is better.

Method	Accuracy $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
PEZ	0.941	0.959	0.922	0.940
PAIA	0.951	0.911	1.000	0.953
Ours	0.980	0.980	0.980	0.980

Table 2: Concept-specificity verification with one target and one fixed non-target query per adapter. PEZ and PAIA receive the queried concept, whereas SCOPE uses its fixed discovered prompt. Higher is better.

Figure 3: Composition of representative top-ranked prompts. The first column reports each LoRA category and concept, the second gives its training trigger revealed only after discovery, and the third decomposes

p^{\star}

returned by SCOPE . Solid blue, dashed purple, and dotted ochre borders denote Word, Composed-word, and Phrase Atomics. In

p^{\star}

, red text marks literal trigger overlap, italic green text marks target-related descriptions, and black text marks unrelated/noise. All annotations are added post hoc and are never used by discovery.

Figure 4: Visual comparison of hidden-concept discovery for an anime character and a synthetic identity. Reference images are generated by the LoRA-enabled model under verified trigger prompts; the remaining columns show LoRA-enabled generations under each method’s returned prompt. PEZ receives target-concept images, whereas SCOPE and the other displayed methods receive no target-concept input.

Panel of Figure 5 — Figure 5: Discovery across backbones and concept types. (a) Trigger-prompt generations (top) and generations from SCOPE ’s top-ranked prompts (bottom) for two styles on SD1.5 and SDXL. (b) Mean target-concept discrepancy by concept type and backbone; lower is better.

Variant	Real ID $\downarrow$	Synthetic ID $\downarrow$	Anime $\downarrow$	Mean $\downarrow$
w/o SGES	0.472	0.313	0.344	0.393
w/o PVI	0.096	0.192	0.138	0.130
w/o CW	0.092	0.180	0.126	0.122
w/o Coh.	0.087	0.187	0.131	0.123
w/o Cons.	0.164	0.238	0.242	0.208
w/o Dist.	0.083	0.179	0.128	0.119

Table 3: Contributions of SGES, PVI, composed-word discovery, and the three PVI operations. Each row removes one component and reruns discovery; lower target-concept discrepancy is better.

Direction	Real ID $\downarrow$	Synthetic ID $\downarrow$	Anime $\downarrow$	Mean $\downarrow$
Random	0.396	0.258	0.301	0.334
Trailing	0.485	0.343	0.327	0.398
Leading	0.078	0.178	0.123	0.114

Table 4: Effect of LoRA input-direction selection at

k=2

. Random uses a dimension-matched orthonormal subspace; lower target-concept discrepancy is better.

Strategy	Real ID $\downarrow$	Synthetic ID $\downarrow$	Anime $\downarrow$	Mean $\downarrow$
Greedy	0.148	0.217	0.180	0.173
Random	0.107	0.191	0.138	0.135
Evolutionary	0.078	0.178	0.123	0.114

Table 5: Prompt-search strategies over identical group-specific Atomic pools with fixed shortlist and PVI budgets. Lower target-concept discrepancy is better.

Variant	Real ID $\downarrow$	Synthetic ID $\downarrow$	Anime $\downarrow$	Mean $\downarrow$
One Bucket	0.097	0.183	0.156	0.136
w/o TE	0.091	0.180	0.150	0.131
w/o UP	0.136	0.250	0.203	0.183
w/o DOWN	0.112	0.271	0.133	0.151
Full	0.078	0.178	0.123	0.114

Table 6: Contribution of text-encoder (TE), U-Net up-path (UP), and U-Net down-path (DOWN) site groups. One Bucket merges candidates from all retained sites; lower target-concept discrepancy is better.

Model	Concept	Human- readable $\downarrow$	Opaque $\downarrow$	$\Delta_{\mathrm{trig}}$ $\downarrow$
SD1.5	Real ID	0.078	0.224	+0.147
	Synthetic ID	0.178	0.292	+0.114
	Overall	0.112	0.247	+0.136
SDXL	Real ID	0.111	0.204	+0.093
	Synthetic ID	0.212	0.358	+0.145
	Overall	0.146	0.257	+0.111

Table 7: Discovery under matched human-readable and opaque-trigger training. Each pair shares its data and optimization settings and differs only in the trigger;

\Delta_{\mathrm{trig}}

is opaque minus human-readable discrepancy. Lower is better.

Attack	Trigger Prompt $\downarrow$	Top-ranked Prompt $\downarrow$	Utility Preserved $\uparrow$	Conditional Evasion $\downarrow$
Dispersion	0.153	0.162	86.2%	4.0%

Table 8: Adaptive subspace-dispersion attack. Prompt columns report mean target-concept discrepancy under attacked adapters. Conditional evasion is measured only among adapters whose fixed trigger prompt preserves target generation.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SD1.5	SDXL
Base model	Stable Diffusion 1.5	SDXL 1.0
Training resolution	512 px	1024 px
LoRA rank	128	128
U-Net learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$
Text-encoder learning rate	$5\times 10^{-5}$	$5\times 10^{-5}$
Optimizer	AdamW	AdamW

Appendix

Table 9: Training configuration for locally trained identity LoRAs.

Method	Use in LoRA evaluation	Required input
DriftScope	Rank tokens by their LoRA–base drift	None
CompCon	Search for prompts exposing LoRA–base differences	None
VisDiff	Describe differences between paired LoRA and base image sets	None
DPO-Diff-EPO	Optimize prompts for paired-model divergence	None
PEZ	Recover a hard prompt from concept images	Concept images
PAIA	Verify supplied concepts for each LoRA	Candidate images

Appendix

Table 10: Prior methods, their use in our LoRA evaluation, and the concept information they require.

$k$	Real ID $\downarrow$	Synthetic ID $\downarrow$	Anime $\downarrow$	Mean $\downarrow$
1	0.080	0.202	0.144	0.128
2	0.078	0.178	0.123	0.114
4	0.088	0.181	0.144	0.127

Appendix

Table 11: Effect of the maximum number of leading LoRA input directions retained per site. Lower target-concept discrepancy is better.

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 393 papers

LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
Jul 6, 2026Marian Lupascu, Sebastian Ripa, Mihai Trascau +2Subject-Driven Image Generation Compositional Image Generation
Adobe Research, Romania · International Computer High School of Bucharest, Romania
LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models
Sep 22, 2026Jin Wei, Rundong Li, Ruihao Yang +10Adversarial Attacks on Diffusion Models Backdoor Attacks
School of Computer Science, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · School of Computing, Xi’an Jiaotong-Liverpool University +2
Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting
Jun 2, 2026Georgios Tsoumplekas, Stella Bounareli, Vasileios ArgyriouSubject-Driven Image Generation Compositional T2I Generation
Department of Networks and Digital Media, Kingston University London, UK

Jul 6, 2026J/K move · Enter open · S save

LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models

Marian Lupascu, Sebastian Ripa, Mihai Trascau, Mariana-Iuliana Georgescu+1

Adobe Research, Romania · International Computer High School of Bucharest, Romania

Subject-Driven Image Generation Compositional Image Generation Diffusion Model Fine-Tuning Low-Rank Adaptation T2I Diffusion Models Diffusion Model Adaptation

Open paper AskPDF arXiv

Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coherent. Methods that fuse independently trained concept adapters in a shared weight space (via federated averaging, gradient fusion, or orthogonality constraints) suffer from identity confusion and style bleeding. In this work, we show that composing concepts as separate image layers, instead of merging their adapters in a shared weight space, avoids parameter-level interference. We introduce LILAC, a framework that composes independently trained low-rank adapters at inference time: each subject is conditioned on the frozen composite of previously placed subjects, with exactly one adapter active at a time, therefore identities never interfere at the parameter level. LILAC composes the adapters without joint training, scales linearly with the number of concepts, and is backbone-agnostic. Under the Orthogonal Adaptation protocol, LILAC applied on Qwen-Image-Edit+Qwen-Image-Layered reaches an ArcFace detection rate of 0.861. Code is available at https://github.com/marianlupascu/LILAC.

Abstract

T2I Diffusion Models

Explore similar work

LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models

LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models

Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting