SDO: Subspace Deconflicting Operator for Multi-Adapter Composition

Zhongsheng Wang; Zhedong Lin; Qian Liu; Xinyu Zhang; Jiamou Liu

cs.AIAug 13, 2026

SDO: Subspace Deconflicting Operator for Multi-Adapter Composition

Authors: Zhongsheng Wang, Zhedong Lin, Qian Liu, Xinyu Zhang, Jiamou Liu

Organizations: University of Auckland Auckland, New Zealand

Abstract

Composing independently trained adapters within a shared diffusion backbone provides a modular approach to multi-character generation, but naive joint deployment often causes identity mixing, cross-character attribute leakage, and unstable scene composition. We study this interference from a parameter-space perspective and hypothesize that it arises partly from conflicts between overlapping dominant subspaces in shared layers. To address this issue, we propose SDO, a Subspace Deconflicting Operator for multi-adapter composition. SDO reconstructs layer-wise low-rank updates from the selected adapters, extracts compact subspace signatures, measures pairwise conflict through output-subspace overlap, and applies a permutation-equivariant transformation that suppresses harmful shared directions while retaining identity-specific characteristics. The resulting representations are mapped back to standard adapter updates and can be directly incorporated into existing diffusion inference pipelines. Experiments demonstrate that SDO consistently improves identity fidelity and compositional stability, with particularly clear gains as the number of jointly composed adapters increases.

Figures & tables

Figure 1. Qualitative comparison on Flintstones multi-character generation with 2 to 5 jointly activated adapters. Each row corresponds to an increasing number of characters, and each column compares different composition methods. Red boxes highlight typical failure cases such as identity mixing and attribute leakage. As scene complexity increases, naive merging and prior methods progressively suffer from degraded identity fidelity and cross-character interference. In contrast, SDO consistently preserves character identity, improves separation, and maintains coherent multi-character interactions.

Figure 2. Conceptual illustration of subspace conflict in multi-adapter composition. Before: overlapping dominant directions. After: reduced overlap with preserved identity-specific directions.

Figure 3. Workflow overview for multi-adapter composition. (A) Independently trained adapters exhibit overlapping dominant directions when injected into a shared backbone, leading to subspace conflict. (B) Each adapter is mapped to a low-rank signature via truncated SVD. (C) A conflict-aware, permutation-equivariant set operator rewrites the signatures to reduce pairwise overlap. (D) The updated representations are reconstructed into standard adapter updates and jointly applied for stable multi-character generation.

Setting	Method	2 LoRAs				3 LoRAs				4 LoRAs				5 LoRAs
Setting	Method	ID $\uparrow$	IR $\uparrow$	FCS $\uparrow$	CL $\uparrow$	ID $\uparrow$	IR $\uparrow$	FCS $\uparrow$	CL $\uparrow$	ID $\uparrow$	IR $\uparrow$	FCS $\uparrow$	CL $\uparrow$	ID $\uparrow$	IR $\uparrow$	FCS $\uparrow$	CL $\uparrow$
SM-FLUX	Naive Merge	0.6107	1.00	1.00	0.2669	0.5425	1.00	1.00	0.1779	0.5637	1.00	1.00	0.2382	0.4805	0.60	0.80	0.2250
	CharCom	0.6974	1.00	1.00	0.2966	0.5462	1.00	1.00	0.1929	0.5575	1.00	1.00	0.2565	0.5006	0.80	1.00	0.2924
	FreeFuse	0.7847	1.00	1.00	0.2395	0.5991	1.00	1.00	0.1962	0.5564	0.75	1.00	0.2298	0.5038	0.60	0.80	0.2274
	LoRA-Composer	0.7421	1.00	1.00	0.2487	0.5874	1.00	1.00	0.2056	0.5338	0.75	1.00	0.2215	0.4916	0.60	0.80	0.2307
	SDO	0.7634	1.00	1.00	0.2523	0.6682	1.00	1.00	0.2091	0.6680	1.00	1.00	0.2447	0.6565	1.00	1.00	0.2537

Table 1. Main quantitative results under three evaluation settings: in-domain (Shakoomaku on FLUX-dev), cross-topic (Flintstones on FLUX-dev), and cross-backbone (Shakoomaku on SDXL). ‘SM’ and ‘FL’ denote the Shakoomaku and Flintstones LoRA pools, respectively. All methods are evaluated under identical adapter subsets, prompts, random seeds, and inference configurations. Performance differences become more noticeable in higher-cardinality settings (4–5 LoRAs), where multi-adapter composition is more challenging. Higher is better for all metrics; ties in saturated IR/FCS columns are not additionally bolded.

Variant	4 LoRAs				5 LoRAs
Variant	ID $\uparrow$	IR $\uparrow$	FCS $\uparrow$	CL $\uparrow$	ID $\uparrow$	IR $\uparrow$	FCS $\uparrow$	CL $\uparrow$
Full SDO	0.6680	1.00	1.00	0.2447	0.6565	1.00	1.00	0.2537
w/o CP	0.6512	0.75	1.00	0.2398	0.5676	0.80	0.80	0.2459
w/o GC	0.6404	1.00	1.00	0.2429	0.6098	0.80	1.00	0.2488
w/o SR	0.6241	0.75	0.75	0.2416	0.5913	0.80	0.60	0.2474

Table 2. Ablation study of SDO under 4- and 5-LoRA composition. Removing the conflict prior (CP), global context (GC), or structured reconstruction (SR) consistently degrades performance, especially in the more challenging 5-LoRA setting. The largest drop occurs when CP is removed, highlighting the importance of explicitly modeling subspace conflict.

Figure 4. Qualitative comparison under progressively increasing scene complexity. As more adapters are jointly loaded, baseline methods tend to show stronger identity blending, missing characters, and cross-character attribute leakage. In contrast, SDO better maintains subject identity and separation, yielding more coherent multi-character compositions, with differences becoming more noticeable in the 4- and 5-LoRA settings.

Figure 5. Human evaluation results visualized as radar plots for 4- and 5-LoRA settings. Each axis corresponds to a different criterion: identity fidelity, identity separation, character completeness, prompt/scene alignment, and visual quality. Different colors represent different methods. SDO achieves the strongest overall performance, especially in the 5-LoRA setting.

Figure 6. PCA visualization of adapter signatures before and after SDO under FLUX and SDXL. Each point represents one adapter, formed by concatenating layer-wise behavior signatures across shared layers. Circles denote original adapters and triangles denote the SDO-transformed version.

Compositional Image Generation

Compositional image generation studies how to combine objects, attributes, spatial relations, and visual references into coherent, controllable images.

See all 85 papers

LoRAtorio: An intrinsic approach to LoRA Skill Composition
Aug 15, 2025Niki Foteinopoulou, Ignas Budvytis, Stephan LiwickiT2I Diffusion Models Low-Rank Adaptation
Cambridge Research Laboratory, Toshiba Europe · Independent Researcher
OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models
Apr 6, 2026Ali Aliev, Kamil Garifullin, Nikolay Yudin +5Model Merging Diffusion Model Adaptation
HSE University · FusionBrain Lab · AXXX
LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
Jul 6, 2026Marian Lupascu, Sebastian Ripa, Mihai Trascau +2Subject-Driven Image Generation Compositional Image Generation
Adobe Research, Romania · International Computer High School of Bucharest, Romania

Aug 15, 2025J/K move · Enter open · S save

LoRAtorio: An intrinsic approach to LoRA Skill Composition

Niki Foteinopoulou, Ignas Budvytis, Stephan Liwicki

Cambridge Research Laboratory, Toshiba Europe · Independent Researcher

T2I Diffusion Models Low-Rank Adaptation Diffusion Model Guidance

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become a widely adopted technique in text-to-image diffusion models, enabling the personalisation of visual concepts such as characters, styles, and objects. However, existing approaches struggle to effectively compose multiple LoRA adapters, particularly in open-ended settings where the number and nature of required skills are not known in advance. In this work, we present LoRAtorio, a novel train-free framework for multi-LoRA composition that leverages intrinsic model behaviour. Our method is motivated by two key observations: (1) LoRA adapters trained on narrow domains produce denoised outputs that diverge from the base model, and (2) when operating out-of-distribution, LoRA outputs show behaviour closer to the base model than when conditioned in distribution. The balance between these two observations allows for exceptional performance in the single LoRA scenario, which nevertheless deteriorates when multiple LoRAs are loaded. Our method operates in the latent space by dividing it into spatial patches and computing cosine similarity between each patch's predicted noise and that of the base model. These similarities are used to construct a spatially-aware weight matrix, which guides a weighted aggregation of LoRA outputs. To address domain drift, we further propose a modification to classifier-free guidance that incorporates the base model's unconditional score into the composition. We extend this formulation to a dynamic module selection setting, enabling inference-time selection of relevant LoRA adapters from a large pool. LoRAtorio achieves state-of-the-art performance, showing up to a 1.3% improvement in ClipScore and a 72.43% win rate in GPT-4V pairwise evaluations, and generalises effectively to multiple latent diffusion models. Project page: https://nickyfot.github.io/loratorio-skill-composer/

Abstract

Compositional Image Generation

Explore similar work

LoRAtorio: An intrinsic approach to LoRA Skill Composition

OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models

LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models