In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the manifold formed by the recently proposed Group-and-Shuffle (GS) orthogonal matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a spectra restoration transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two GS orthogonal matrices is capable of uniting concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the link.
Figures & tables
Figure 2: Distribution of the eigenvalues of orthogonal fine-tuning adapters before and after merging. (Left): eigenvalues of LC1 . (Center): eigenvalues of LS1 . (Right): eigenvalues of B(0.5) (orange) and of BOrthoFuse(0.5) (blue). Eigenvalues are calculated for the query GS -orthogonal adapter of StableDiffusion-XL model.
Figure 3: Ablation of the fusion parameter t from ( 7 ). When t=0 , the merged adapter reduces to a pure concept adapter, preserving identity with no stylization. When t=1 , the merged weights correspond to a pure style adapter. Intermediate values produce a continuous fusion curve between concept preservation and style strength, with t=0.6 yielding the most balanced trade-off.
Method
Style Sim
CLIP
DINO
Geo. Mean (Style, DINO)
Geo. Mean (Style, CLIP)
Merging time
Training-based
Joint training
0.48
0.79
0.67
0.57
0.62
1.5 hours
ZipLoRA r=8
0.49
0.74
0.55
0.52
0.6
4 minutes
ZipLoRA r=64
0.49
0.76
0.64
0.56
0.61
Training-free
K-LoRA r=8
0.46
0.76
0.55
0.5
0.59
<1 sec
Table 1: Quantitative comparison of OrthoFuse and baseline methods. Style Sim measures style fidelity using CLIP similarity with the reference style image. CLIP and DINO concept metrics evaluate semantic consistency with the original concept. The geometric mean combines style and concept metrics to summarize overall trade-off between style fidelity and concept preservation.
Figure 4: Qualitative comparisons. We present images generated by OrthoFuse alongside those created using the compared baselines. OrthoFuse strikes an ideal balance between concept and style, preserving both the concept and style.
Question
Our vs K-L.
Our vs Z.
Concept Preserv. (Q1)
48% vs 52%
54% vs 46%
Style Transfer (Q2)
77% vs 23%
83% vs 17%
Overall Preference (Q3)
67% vs 33%
76% vs 24%
Table 2: User study . Comparison of our method with K-LoRA and ZipLoRA on all images used in the quantitative evaluation. 65 participants, 1460 pairwise comparisons.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Velocity of geodesics measured in the Frobenius norm: comparison of the exact SO(N) geodesic, the block‑wise orthogonal approximation, and the block‑wise Cayley space approximation.
Figure 6: Relative error (in Frobenius norm) between the blockwise geodesic (geodesic projection from G ) and the geodesic in M , as well as the computation time of the two methods.
Figure 7: Qualitative results of OrthoFuse merging on the FLUX model. Each row shows generations produced from different prompts after merging a style adapter and a concept adapter. OrthoFuse maintains consistent concept preservation and style fidelity across prompts.
Figure 8: Comparison of block-wise geodesic interpolation and OrthoFuse merging trajectories. At t=0.6 , OrthoFuse achieves near-ideal style transfer while preserving the target concept. All images were generated with the prompt: “A < concept >< superclass > in jungle in < style > style”.
Figure 9: Qualitative results of OrthoFuse merging on SDXL . Rows correspond to different prompts; columns show generations obtained from different style–concept adapter pairs. OrthoFuse yields coherent style–concept merges across both prompts and adapter combinations.
Figure 10: Comparison of OrthoFuse (orthogonal adapters merging) with merging low-rank (LoRA) adapters. Merging low-rank adapters results in noticeably weaker performance compared to OrthoFuse, even when both approaches are tuned to use approximately the same number of trainable parameters. We attribute this gap to the scale mismatch inherent to low-rank adapters, which makes their merging substantially more difficult. All images were generated with the prompt: “a < concept > dog in the jungle in < style > style”.
Figure 11: Direct Merging via Multiplication. The result of directly merging orthogonal adapters, accomplished through the multiplication of two GS orthogonal matrices, exhibits limitations in style preservation, struggles to maintain color consistency, and has a negative impact on concept fidelity.
Figure 12: Ablation of the fusion parameter η0 .
Figure 13: Ablation of the fusion parameter t .
Figure 14: Comparative analysis of OrthoFuse and its accelerated version . Using identical concept and style adapters and the same fusion parameter t , both methods produce visually indistinguishable results. The accelerated version removes the eigendecomposition step while preserving identity and style fidelity. We note, however, that the two methods are not strictly identical; for example, in the first row, the dog’s right paw in the OrthoFuse result is not fully placed on the surfboard, whereas in FastOrthoFuse it is.
With the growing availability of open-sourced adapters trained on the same diffusion backbone for diverse scenes and objects, combining these pretrained weights enables low-cost customized generation. However, most existing model merging methods are designed for classification or text generation, and when applied to image generation, they suffer from content drift due to error accumulation across multiple diffusion steps. For image-oriented methods, training-based approaches are computationally expensive and unsuitable for edge deployment, while training-free ones use uniform fusion strategies that ignore inter-adapter differences, leading to detail degradation. We find that since different adapters are specialized for generating different types of content, the contribution of each diffusion step carries different significance for each adapter. Accordingly, we propose a frequency-domain importance-driven dynamic LoRA switch method. Furthermore, we observe that maintaining semantic consistency across adapters effectively mitigates detail loss; thus, we design an automatic Generation Alignment mechanism to align generation intents at the semantic level. Experiments demonstrate that our FREE-Switch (Frequency-based Efficient and Dynamic LoRA Switch) framework efficiently combines adapters for different objects and styles, substantially reducing the training cost of high-quality customized generation.
Unified fashion generation integrates tasks like virtual try-on and garment reconstruction into a single model to reduce task-specific adaptation costs. However, naive parameter sharing across semantically distinct tasks induces negative transfer through severe inter-task gradient conflict. We propose OrthoTryOn, a unified framework mitigating this interference within a shared Low-Rank Adaptation (LoRA) module. Its Orthogonal Subspace Projection (OSP) applies task-specific orthogonal rotations to bottleneck features, mapping them into decorrelated coordinate frames. To address residual semantic coupling at inference time, we further propose Fisher-guided Negative Guidance (FNG), a parameter-free strategy that utilizes diagonal Fisher information to quantify inter-task sensitivity overlap and explicitly repels generation trajectories from the most confusable task via Classifier-Free Guidance. Extensive experiments demonstrate that OrthoTryOn avoids the severe performance degradation typical of naive unified training and even surpasses independently trained task-specific models, achieving state-of-the-art results across multiple benchmarks while generalizing robustly across diverse diffusion backbones. Code is available at https://github.com/NJU-PCALab/OrthoTryOn.
Zhaotong Yang, Ying Tai, Jiahui Zhan +3
PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology · PCA Lab, School of Intelligence Science and Technology, Nanjing University · Shanghai Jiao Tong University
Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coherent. Methods that fuse independently trained concept adapters in a shared weight space (via federated averaging, gradient fusion, or orthogonality constraints) suffer from identity confusion and style bleeding. In this work, we show that composing concepts as separate image layers, instead of merging their adapters in a shared weight space, avoids parameter-level interference. We introduce LILAC, a framework that composes independently trained low-rank adapters at inference time: each subject is conditioned on the frozen composite of previously placed subjects, with exactly one adapter active at a time, therefore identities never interfere at the parameter level. LILAC composes the adapters without joint training, scales linearly with the number of concepts, and is backbone-agnostic. Under the Orthogonal Adaptation protocol, LILAC applied on Qwen-Image-Edit+Qwen-Image-Layered reaches an ArcFace detection rate of 0.861. Code is available at https://github.com/marianlupascu/LILAC.
Marian Lupascu, Sebastian Ripa, Mihai Trascau +2
Adobe Research, Romania · International Computer High School of Bucharest, Romania