LoRAtorio: An intrinsic approach to LoRA Skill Composition

Niki Foteinopoulou; Ignas Budvytis; Stephan Liwicki

cs.CVAug 15, 2025

LoRAtorio: An intrinsic approach to LoRA Skill Composition

Authors: Niki Foteinopoulou, Ignas Budvytis, Stephan Liwicki

Organizations: Cambridge Research Laboratory, Toshiba Europe · Independent Researcher

Abstract

Low-Rank Adaptation (LoRA) has become a widely adopted technique in text-to-image diffusion models, enabling the personalisation of visual concepts such as characters, styles, and objects. However, existing approaches struggle to effectively compose multiple LoRA adapters, particularly in open-ended settings where the number and nature of required skills are not known in advance. In this work, we present LoRAtorio, a novel train-free framework for multi-LoRA composition that leverages intrinsic model behaviour. Our method is motivated by two key observations: (1) LoRA adapters trained on narrow domains produce denoised outputs that diverge from the base model, and (2) when operating out-of-distribution, LoRA outputs show behaviour closer to the base model than when conditioned in distribution. The balance between these two observations allows for exceptional performance in the single LoRA scenario, which nevertheless deteriorates when multiple LoRAs are loaded. Our method operates in the latent space by dividing it into spatial patches and computing cosine similarity between each patch's predicted noise and that of the base model. These similarities are used to construct a spatially-aware weight matrix, which guides a weighted aggregation of LoRA outputs. To address domain drift, we further propose a modification to classifier-free guidance that incorporates the base model's unconditional score into the composition. We extend this formulation to a dynamic module selection setting, enabling inference-time selection of relevant LoRA adapters from a large pool. LoRAtorio achieves state-of-the-art performance, showing up to a 1.3% improvement in ClipScore and a 72.43% win rate in GPT-4V pairwise evaluations, and generalises effectively to multiple latent diffusion models. Project page: https://nickyfot.github.io/loratorio-skill-composer/

Figures & tables

Panel of Figure 1 — Figure 1: Observations of noise prediction from the ComposLoRA testbed. (a) t-SNE visualisation of the unconditioned latent space, for the Base Model and LoRA-adapted models. (b) Generated images from each Character LoRA, conditioned with text prompts originally associated with other LoRAs. When a prompt falls outside a LoRA’s training distribution, the predicted latent $e_{\theta_{i}}(z_{0},0,c)$ of the $i^{th}$ LoRA tends to align closely with that of the base (Naïve) model, showing minimal deviation due to changes in $p_{\theta_{i}}(x)$ , also shown by the cosine similarity of the conditioned noise prediction of the Naïve model with that of the $i^{th}$ LoRA.

Figure 2: Overview of LoRAtorio . Skill Composition: At each denoising timestep $t$ , the conditional score from the $i^{\text{th}}$ LoRA, $e_{\theta_{i}}(z_{t},t,c)$ , is partitioned into $P$ spatial patches (equation 3 ). Each patch is flattened and compared to its corresponding patch in the base model’s predicted noise using cosine similarity. The resulting similarity matrix is passed through a SoftMin function (equation 4 ) to produce a weight matrix $\Omega$ , assigning higher weights to patches that diverge more from the base model. These weights are used to compute a spatially-aware weighted average across LoRA outputs (equation 5 and equation 6 ). Re-Centering: To mitigate domain drift of the unconditional score by the multiple LoRAs, we alter classifier-free guidance by incorporating a weighted combination of the base model’s unconditional score and the aggregated LoRA score (equation 7 ).

Figure 3: Visualisation of the effect of re-centering guidance (equation 7 ) on the unconditional noise score. Re-centering ensures the $p(x|c)$ is not over-emphasising implausible or under-trained regions of the collective data distribution after CFG. When the scores are similar, the transformation is not significant, but when there is a large adjustment, the difference is in the direction towards more probable samples.

Model	$N=2$	$N=3$	$N=4$	$N=5$	Avg.
Naïve [ 33 ]	35.014	34.927	34.384	33.809	34.534
Merge [ 15 ]	33.726	34.139	33.399	32.364	33.407
Switch [ 46 ]	35.394	35.107	34.478	33.475	34.614
Composite [ 46 ]	35.073	34.082	34.802	32.582	34.135
LoraHub [ 12 ]	3 5.681	35.127	34.970	33.485	34.816
Switch-A [ 48 ]	35.451	35.383	34.877	33.366	34.769

Table 1: CLIPScore of LoRAtorio against previous composition methods on ComposLoRA .

Panel of Figure 4 — Figure 4: GPT-4V Evaluation on ComposLoRA against previous SoTA works.

	Element Integration	Spatial Consistency	Semantic Accuracy	Aesthetic Quality
LoRAtorio	7.64	7.58	7.33	6.83
CMLoRA	5.63	5.58	6.08	5.25
Composite	6.46	6.71	6.71	6.46
Switch	7 .57	7 .50	6 .88	6 .71
Merge	6.83	6.71	6.58	6.08

Table 2: Human Evaluation of our Method against previous SoTA along four qualitative axes.

	$N=2$	$N=3$	$N=4$	$N=5$	Avg.
LoRAtorio	34.593	35.563	36.480	37.028	35.916
Naïve	35.014	34.927	34.384	33.809	34.534
Merge	27.167	27.151	27.023	27.272	27.153

Table 3: CLIPScore of LoRAtorio against previous SoTA on ComposLoRA in a dynamic module selection setting, where

N

is the number of LoRA experts needed.

Table 4: CLIPScore of LoRAtorio against selected composition methods, using Flux architecture.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

LoRA Model	KL Divergence
Character 1	34.81
Character 2	35.09
Character 3	36.15
Style 1	36.44
Style 2	36.31
Object 1	35.52

Appendix

Table 5: KL Divergence of single LoRA adapters against the base model.

Table 6: ClipScores for ComposLoRA on anime and reality subsets.

Panel of Figure 5 — Figure 5: Composition and Image Quality scores of LoRAtorio against previous SoTA on ComposeLoRA , by number of LoRA’s included (a-b) and image style (c).

Panel of Figure 6 — Figure 6: Win/Tie/Loss for LoRAtorio compared to previous SoTA across different number of LoRAs $\left(N\right)$ .

Figure 7: Qualitative comparison of LoRAtorio’s re-centering guidance across different values of $\lambda$ , evaluated against auto-guidance [ 18 ] . The figure illustrates the impact of varying $\lambda$ on image coherence, identity preservation, and visual quality. 1 1 1 CLIPScore for the images in each row is identical, including autoguidance samples highlighting the necessity of visual inspection and qualitative evaluation. Row 1:32.552, Row 2: 31.810, Row 3: 31.763, Row 4: 32.653

Table 7: Ablation study of our method on ComposLoRA Anime subset, using CLIPScore.

Figure 8: Average values of $\Omega^{t}$ over denoising process, for the entire image in the ComposLoRA testbed for $N=2$ .

Figure 9: $\Omega^{t}$ map for character overlayed over the final generated images across timesteps, in Character + Style generation. As style has more global effect, we see clearly the effect of the character LoRA on the predicted noise through the $\Omega^{t}$ heatmap (high effect is red, no effect is transparent).

Figure 10: Examples of failure cases of LoRAtorio on ComposLoRA anime (top), Flux (mid) and ComposLoRA reality (bottom) test beds.

Adapter group	Divergence from base	CLIP-fidelity
Clean (5 adapters)	0.008–0.022	25–36
Over-fitted	0.050	35
Domain-mismatched	0.082	16.6

Appendix

Table 8: Divergence from the base model and CLIP-fidelity for clean and deliberately corrupted adapters.

LoRA	Category	Trigger	Source
Yennefer of Vengerberg	Character	Yennefer	Link
The amazing Spiderman	Character	Spider-Man, Peter Parker	Link
B1 Battle Droid	Character	7-B1 droid	Link
Star Wars imperial officer uniform	Clothing	Wearing an imperial officer IMPOFF uniform	Link
Japanese school uniform - sailorfuku	Clothing	wearing a japanese school uniform sailorfuku serafuku sailor suit	Link
Frank Frazetta Style Oil Painting	Style	in the style of Frank Frazetta fantasy oil painting	Link

Appendix

Table 9: Details of LoRA adapters used in Flux experiments.

Table 10: Comparison of Multiply-Accumulate Operations (MACs) and qualitative latency estimates under different

N

values.

Table 11: Exploratory retrieval results for threshold-based (k-free) dynamic selection.

Figure 13: Images generated with $N$ LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using SD1.5 base model on the anime subset of ComposLoRA .

Figure 14: Images generated with $N$ LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using SD1.5 base model on the reality subset of ComposLoRA .

Figure 15: Images generated with $N$ LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using Flux base model.

Figure 16: Images generated with $N$ LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using Flux base model.

Figure 17: Images generated with $N$ LoRA candidates (L1 Character, L2 Clothing, L3 Style, L4 Background and L5 Object) across our proposed framework and baseline methods using Flux base model.

T2I Diffusion Models

Text-to-image (T2I) diffusion models generate images from prompts by learning to reverse noise under text conditioning.

See all 393 papers

Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting
Jun 2, 2026Georgios Tsoumplekas, Stella Bounareli, Vasileios ArgyriouSubject-Driven Image Generation Compositional T2I Generation
Department of Networks and Digital Media, Kingston University London, UK
A LoRA is Worth a Thousand Pictures
Dec 16, 2024Chenxi Liu, Towaki Takikawa, Alec JacobsonWeight-Space Analysis Low-Rank Adaptation
University of Toronto · Outerport · Adobe Research
LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models
Sep 22, 2026Jin Wei, Rundong Li, Ruihao Yang +10Adversarial Attacks on Diffusion Models Backdoor Attacks
School of Computer Science, Fudan University, Shanghai, China · Institute of Big Data, Fudan University, Shanghai, China · School of Computing, Xi’an Jiaotong-Liverpool University +2

Jun 2, 2026J/K move · Enter open · S save

Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting

Georgios Tsoumplekas, Stella Bounareli, Vasileios Argyriou

Department of Networks and Digital Media, Kingston University London, UK

Subject-Driven Image Generation Compositional T2I Generation T2I Diffusion Models

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) successfully enables personalization in text-to-image generation by adapting pre-trained diffusion models to specific visual concepts and styles. However, extending such models to multi-concept customization remains challenging. Naively combining multiple LoRA weights or their outputs often leads to interference among concepts, resulting in degraded visual quality and reduced fidelity to the reference images of individual concepts. This paper proposes a simple yet effective approach for multi-concept customization by optimally combining the outputs of multiple LoRA modules. We leverage the relative importance of each concept during generation, as inferred from its corresponding prompt tokens and introduce two methods, W-Switch and W-Composite, that employ a prompt-aware importance weighting strategy in which each LoRA is weighted according to the semantic influence of its trigger words in the target prompt. In addition, we extend existing quantitative evaluation metrics by proposing a new image-based similarity evaluation framework that assesses image fidelity and identity preservation through comparisons between real-world reference images and automatically segmented concept regions from generated images. We evaluate our approach on the ComposLoRA testbed and demonstrate consistent improvements over existing state-of-the-art methods in terms of visual quality, identity preservation and compositionality. Qualitative evaluations, including a Large Language Model (LLM) based assessment and a user study, further validate the effectiveness of the proposed methods and align with the newly introduced quantitative image-based metrics. Our code is available at https://github.com/GeorgeTsoumplekas/Prompt-Aware-Multi-LoRA-Composition.

Abstract

T2I Diffusion Models

Explore similar work

Training-Free Multi-Concept LoRA Composition with Prompt-Aware Weighting

A LoRA is Worth a Thousand Pictures

LoRango: It Takes Two LoRAs to Unlock Hidden Behaviors in Diffusion Models