Low-Rank Adaptation (LoRA) has become a widely adopted technique in text-to-image diffusion models, enabling the personalisation of visual concepts such as characters, styles, and objects. However, existing approaches struggle to effectively compose multiple LoRA adapters, particularly in open-ended settings where the number and nature of required skills are not known in advance. In this work, we present LoRAtorio, a novel train-free framework for multi-LoRA composition that leverages intrinsic model behaviour. Our method is motivated by two key observations: (1) LoRA adapters trained on narrow domains produce denoised outputs that diverge from the base model, and (2) when operating out-of-distribution, LoRA outputs show behaviour closer to the base model than when conditioned in distribution. The balance between these two observations allows for exceptional performance in the single LoRA scenario, which nevertheless deteriorates when multiple LoRAs are loaded. Our method operates in the latent space by dividing it into spatial patches and computing cosine similarity between each patch's predicted noise and that of the base model. These similarities are used to construct a spatially-aware weight matrix, which guides a weighted aggregation of LoRA outputs. To address domain drift, we further propose a modification to classifier-free guidance that incorporates the base model's unconditional score into the composition. We extend this formulation to a dynamic module selection setting, enabling inference-time selection of relevant LoRA adapters from a large pool. LoRAtorio achieves state-of-the-art performance, showing up to a 1.3% improvement in ClipScore and a 72.43% win rate in GPT-4V pairwise evaluations, and generalises effectively to multiple latent diffusion models. Project page: https://nickyfot.github.io/loratorio-skill-composer/
Figures & tables
Figure 1: Observations of noise prediction from the ComposLoRA testbed. (a) t-SNE visualisation of the unconditioned latent space, for the Base Model and LoRA-adapted models. (b) Generated images from each Character LoRA, conditioned with text prompts originally associated with other LoRAs. When a prompt falls outside a LoRA’s training distribution, the predicted latent eθi(z0,0,c) of the ith LoRA tends to align closely with that of the base (Naïve) model, showing minimal deviation due to changes in pθi(x) , also shown by the cosine similarity of the conditioned noise prediction of the Naïve model with that of the ith LoRA.
Figure 2: Overview of LoRAtorio . Skill Composition: At each denoising timestep t , the conditional score from the ith LoRA, eθi(zt,t,c) , is partitioned into P spatial patches (equation 3 ). Each patch is flattened and compared to its corresponding patch in the base model’s predicted noise using cosine similarity. The resulting similarity matrix is passed through a SoftMin function (equation 4 ) to produce a weight matrix Ω , assigning higher weights to patches that diverge more from the base model. These weights are used to compute a spatially-aware weighted average across LoRA outputs (equation 5 and equation 6 ). Re-Centering: To mitigate domain drift of the unconditional score by the multiple LoRAs, we alter classifier-free guidance by incorporating a weighted combination of the base model’s unconditional score and the aggregated LoRA score (equation 7 ).
Figure 3: Visualisation of the effect of re-centering guidance (equation 7 ) on the unconditional noise score. Re-centering ensures the p(x∣c) is not over-emphasising implausible or under-trained regions of the collective data distribution after CFG. When the scores are similar, the transformation is not significant, but when there is a large adjustment, the difference is in the direction towards more probable samples.
Model
N=2
N=3
N=4
N=5
Avg.
Naïve [ 33 ]
35.014
34.927
34.384
33.809
34.534
Merge [ 15 ]
33.726
34.139
33.399
32.364
33.407
Switch [ 46 ]
35.394
35.107
34.478
33.475
34.614
Composite [ 46 ]
35.073
34.082
34.802
32.582
34.135
LoraHub [ 12 ]
3 5.681
35.127
34.970
33.485
34.816
Switch-A [ 48 ]
35.451
35.383
34.877
33.366
34.769
Table 1: CLIPScore of LoRAtorio against previous composition methods on ComposLoRA .
Figure 4: GPT-4V Evaluation on ComposLoRA against previous SoTA works.
Element Integration
Spatial Consistency
Semantic Accuracy
Aesthetic Quality
LoRAtorio
7.64
7.58
7.33
6.83
CMLoRA
5.63
5.58
6.08
5.25
Composite
6.46
6.71
6.71
6.46
Switch
7 .57
7 .50
6 .88
6 .71
Merge
6.83
6.71
6.58
6.08
Table 2: Human Evaluation of our Method against previous SoTA along four qualitative axes.
N=2
N=3
N=4
N=5
Avg.
LoRAtorio
34.593
35.563
36.480
37.028
35.916
Naïve
35.014
34.927
34.384
33.809
34.534
Merge
27.167
27.151
27.023
27.272
27.153
Table 3: CLIPScore of LoRAtorio against previous SoTA on ComposLoRA in a dynamic module selection setting, where N is the number of LoRA experts needed.
Table 4: CLIPScore of LoRAtorio against selected composition methods, using Flux architecture.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
LoRA Model
KL Divergence
Character 1
34.81
Character 2
35.09
Character 3
36.15
Style 1
36.44
Style 2
36.31
Object 1
35.52
Appendix
Table 5: KL Divergence of single LoRA adapters against the base model.
Table 6: ClipScores for ComposLoRA on anime and reality subsets.
Figure 5: Composition and Image Quality scores of LoRAtorio against previous SoTA on ComposeLoRA , by number of LoRA’s included (a-b) and image style (c).
Figure 6: Win/Tie/Loss for LoRAtorio compared to previous SoTA across different number of LoRAs (N) .
Figure 7: Qualitative comparison of LoRAtorio’s re-centering guidance across different values of λ , evaluated against auto-guidance [ 18 ] . The figure illustrates the impact of varying λ on image coherence, identity preservation, and visual quality. 1 1 1 CLIPScore for the images in each row is identical, including autoguidance samples highlighting the necessity of visual inspection and qualitative evaluation. Row 1:32.552, Row 2: 31.810, Row 3: 31.763, Row 4: 32.653
Table 7: Ablation study of our method on ComposLoRA Anime subset, using CLIPScore.
Figure 8: Average values of Ωt over denoising process, for the entire image in the ComposLoRA testbed for N=2 .
Figure 9: Ωt map for character overlayed over the final generated images across timesteps, in Character + Style generation. As style has more global effect, we see clearly the effect of the character LoRA on the predicted noise through the Ωt heatmap (high effect is red, no effect is transparent).
Figure 10: Examples of failure cases of LoRAtorio on ComposLoRA anime (top), Flux (mid) and ComposLoRA reality (bottom) test beds.
Adapter group
Divergence from base
CLIP-fidelity
Clean (5 adapters)
0.008–0.022
25–36
Over-fitted
0.050
35
Domain-mismatched
0.082
16.6
Appendix
Table 8: Divergence from the base model and CLIP-fidelity for clean and deliberately corrupted adapters.
LoRA
Category
Trigger
Source
Yennefer of Vengerberg
Character
Yennefer
Link
The amazing Spiderman
Character
Spider-Man, Peter Parker
Link
B1 Battle Droid
Character
7-B1 droid
Link
Star Wars imperial officer uniform
Clothing
Wearing an imperial officer IMPOFF uniform
Link
Japanese school uniform - sailorfuku
Clothing
wearing a japanese school uniform sailorfuku serafuku sailor suit
Link
Frank Frazetta Style Oil Painting
Style
in the style of Frank Frazetta fantasy oil painting
Link
Appendix
Table 9: Details of LoRA adapters used in Flux experiments.
Table 10: Comparison of Multiply-Accumulate Operations (MACs) and qualitative latency estimates under different N values.
Table 11: Exploratory retrieval results for threshold-based (k-free) dynamic selection.
Figure 13: Images generated with N LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using SD1.5 base model on the anime subset of ComposLoRA .
Figure 14: Images generated with N LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using SD1.5 base model on the reality subset of ComposLoRA .
Figure 15: Images generated with N LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using Flux base model.
Figure 16: Images generated with N LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using Flux base model.
Figure 17: Images generated with N LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using Flux base model.
Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.
Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning of image diffusion models, with over 200K models trained to replicate artistic styles and shared in online communities. One way to analyze styles of LoRA models is to generate image samples and apply image-based features like CLIP. However, this generation requires additional computation and prompt selection. In this work, we propose analyzing styles directly in the LoRA weight space, bypassing these limitations while achieving better performance even on unseen LoRAs. Since in-the-wild LoRAs lack ground truth model details, we design an experimental setup and show that LoRA weights alone serve as effective feature representations of adapted styles, robust to variations in fine-tuning seeds and image selection. We extensively evaluate using SD1.5 (the most popular base model within online communities) across data domains (fine-art paintings, architectural photos) and diverse prompt settings, with a small-scale SDXL experiment further verifying generalization. These evaluations confirm that our approach, under the selected aspects of variability, preserves fine-grained styles and outperforms image-based methods in clustering and retrieval of unseen LoRAs.
Chenxi Liu, Towaki Takikawa, Alec Jacobson
University of Toronto · Outerport · Adobe Research
Users commonly combine multiple Low-Rank Adaptation (LoRA) adapters to personalize images with different subjects, styles, and visual attributes. Yet inspecting adapters individually does not establish the safety of their composition. We identify and characterize a pair-conditioned attack in text-to-image diffusion: individually useful and benign-appearing adapters redirect image generation when co-loaded with a specifically matched partner, whose identity serves as the trigger. We introduce LoRango to realize this attack through complementary Signature and Payload adapters. The Signature writes a pair-specific code into intermediate carrier representations, while the Payload uses code-selective responses and opposing signal/reference branches. These branches approximately cancel for standalone adapters and mismatched pairs; matched code-reader alignment breaks cancellation within native GEGLU blocks and releases the programmed action. Both adapters are exported as ordinary static LoRA files compatible with standard loaders, requiring no prompt trigger or base-pipeline modification. LoRango achieves matched-pair attack success rates of 97.9% on SD v1.5 and 98.7% on SDXL, compared with 2.8--4.6% when implanted adapters are loaded individually. Further experiments evaluate pair selectivity, standalone fidelity, robustness to deployment variations, and applicability across denoiser architectures. These findings show that individual-adapter inspection is insufficient to assess the security of multi-LoRA personalization and motivate auditing adapter compositions.
Jin Wei, Rundong Li, Ruihao Yang +10
School of Computer Science, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · School of Computing, Xi’an Jiaotong-Liverpool University +2