OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models

Ali Aliev; Kamil Garifullin; Nikolay Yudin; Vera Soboleva; Alexander Molozhavenko; Ivan Oseledets; Aibek Alanov; Maxim Rakhuba

cs.CVApr 6, 2026

OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models

Authors: Ali Aliev, Kamil Garifullin, Nikolay Yudin, Vera Soboleva, Alexander Molozhavenko, Ivan Oseledets, Aibek Alanov, Maxim Rakhuba

Organizations: HSE University · FusionBrain Lab · AXXX

Abstract

In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the manifold formed by the recently proposed Group-and-Shuffle ( $\mathcal{GS}$ ) orthogonal matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a $\text{spectra restoration}$ transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two $\mathcal{GS}$ orthogonal matrices is capable of uniting concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the link.

Figures & tables

Figure 2: Distribution of the eigenvalues of orthogonal fine-tuning adapters before and after merging. (Left): eigenvalues of

L_{C}^{1}

. (Center): eigenvalues of

L_{S}^{1}

. (Right): eigenvalues of

B(0.5)

(orange) and of

B_{\text{OrthoFuse}}(0.5)

(blue). Eigenvalues are calculated for the query

\mathcal{GS}

-orthogonal adapter of StableDiffusion-XL model.

Figure 3: Ablation of the fusion parameter $t$ from ( 7 ). When $t=0$ , the merged adapter reduces to a pure concept adapter, preserving identity with no stylization. When $t=1$ , the merged weights correspond to a pure style adapter. Intermediate values produce a continuous fusion curve between concept preservation and style strength, with $t=0.6$ yielding the most balanced trade-off.

Method	Style Sim	CLIP	DINO	Geo. Mean (Style, DINO)	Geo. Mean (Style, CLIP)	Merging time
Training-based
Joint training	0.48	0.79	0.67	0.57	0.62	1.5 hours
ZipLoRA $r=8$	0.49	0.74	0.55	0.52	0.6	4 minutes
ZipLoRA $r=64$	0.49	0.76	0.64	0.56	0.61	4 minutes
Training-free
K-LoRA $r=8$	0.46	0.76	0.55	0.5	0.59	$<1$ sec

Table 1: Quantitative comparison of OrthoFuse and baseline methods. Style Sim measures style fidelity using CLIP similarity with the reference style image. CLIP and DINO concept metrics evaluate semantic consistency with the original concept. The geometric mean combines style and concept metrics to summarize overall trade-off between style fidelity and concept preservation.

Figure 4: Qualitative comparisons. We present images generated by OrthoFuse alongside those created using the compared baselines. OrthoFuse strikes an ideal balance between concept and style, preserving both the concept and style.

Question	Our vs K-L.	Our vs Z.
Concept Preserv. (Q1)	48% vs 52%	54% vs 46%
Style Transfer (Q2)	77% vs 23%	83% vs 17%
Overall Preference (Q3)	67% vs 33%	76% vs 24%

Table 2: User study . Comparison of our method with K-LoRA and ZipLoRA on all images used in the quantitative evaluation. 65 participants, 1460 pairwise comparisons.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Velocity of geodesics measured in the Frobenius norm: comparison of the exact

\mathrm{SO}(N)

geodesic, the block‑wise orthogonal approximation, and the block‑wise Cayley space approximation.

Figure 6: Relative error (in Frobenius norm) between the blockwise geodesic (geodesic projection from

G

) and the geodesic in

M

, as well as the computation time of the two methods.

Figure 7: Qualitative results of OrthoFuse merging on the FLUX model. Each row shows generations produced from different prompts after merging a style adapter and a concept adapter. OrthoFuse maintains consistent concept preservation and style fidelity across prompts.

Figure 8: Comparison of block-wise geodesic interpolation and OrthoFuse merging trajectories. At $t=0.6$ , OrthoFuse achieves near-ideal style transfer while preserving the target concept. All images were generated with the prompt: “A $<$ concept $>$ $<$ superclass $>$ in jungle in $<$ style $>$ style”.

Figure 9: Qualitative results of OrthoFuse merging on SDXL . Rows correspond to different prompts; columns show generations obtained from different style–concept adapter pairs. OrthoFuse yields coherent style–concept merges across both prompts and adapter combinations.

Figure 10: Comparison of OrthoFuse (orthogonal adapters merging) with merging low-rank (LoRA) adapters. Merging low-rank adapters results in noticeably weaker performance compared to OrthoFuse, even when both approaches are tuned to use approximately the same number of trainable parameters. We attribute this gap to the scale mismatch inherent to low-rank adapters, which makes their merging substantially more difficult. All images were generated with the prompt: “a $<$ concept $>$ dog in the jungle in $<$ style $>$ style”.

Figure 11: Direct Merging via Multiplication. The result of directly merging orthogonal adapters, accomplished through the multiplication of two $\mathcal{GS}$ orthogonal matrices, exhibits limitations in style preservation, struggles to maintain color consistency, and has a negative impact on concept fidelity.

Figure 12: Ablation of the fusion parameter $\eta_{0}$ .

Figure 13: Ablation of the fusion parameter $t$ .

Figure 14: Comparative analysis of OrthoFuse and its accelerated version . Using identical concept and style adapters and the same fusion parameter $t$ , both methods produce visually indistinguishable results. The accelerated version removes the eigendecomposition step while preserving identity and style fidelity. We note, however, that the two methods are not strictly identical; for example, in the first row, the dog’s right paw in the OrthoFuse result is not fully placed on the surfboard, whereas in FastOrthoFuse it is.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 187 papers

FREE-Switch: Frequency-based Dynamic LoRA Switch for Style Transfer
Apr 11, 2026Shenghe Zheng, Minyu Zhang, Tianhao Liu +1Diffusion-Based Image Generation Diffusion Models
Harbin Institute of Technology
OrthoTryOn: Geometric Orthogonalization for Conflict-Free Unified Fashion Generation
Jun 26, 2026Zhaotong Yang, Ying Tai, Jiahui Zhan +3Diffusion Model Guidance Virtual Try-On
PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology · PCA Lab, School of Intelligence Science and Technology, Nanjing University · Shanghai Jiao Tong University
LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
Jul 6, 2026Marian Lupascu, Sebastian Ripa, Mihai Trascau +2Subject-Driven Image Generation Compositional Image Generation
Adobe Research, Romania · International Computer High School of Bucharest, Romania

Apr 11, 2026J/K move · Enter open · S save

FREE-Switch: Frequency-based Dynamic LoRA Switch for Style Transfer

Shenghe Zheng, Minyu Zhang, Tianhao Liu, Hongzhi Wang

Harbin Institute of Technology

Diffusion-Based Image Generation Diffusion Models Image Style Transfer Low-Rank Adaptation

Open paper AskPDF arXiv

With the growing availability of open-sourced adapters trained on the same diffusion backbone for diverse scenes and objects, combining these pretrained weights enables low-cost customized generation. However, most existing model merging methods are designed for classification or text generation, and when applied to image generation, they suffer from content drift due to error accumulation across multiple diffusion steps. For image-oriented methods, training-based approaches are computationally expensive and unsuitable for edge deployment, while training-free ones use uniform fusion strategies that ignore inter-adapter differences, leading to detail degradation. We find that since different adapters are specialized for generating different types of content, the contribution of each diffusion step carries different significance for each adapter. Accordingly, we propose a frequency-domain importance-driven dynamic LoRA switch method. Furthermore, we observe that maintaining semantic consistency across adapters effectively mitigates detail loss; thus, we design an automatic Generation Alignment mechanism to align generation intents at the semantic level. Experiments demonstrate that our FREE-Switch (Frequency-based Efficient and Dynamic LoRA Switch) framework efficiently combines adapters for different objects and styles, substantially reducing the training cost of high-quality customized generation.

Abstract

Model Merging

Explore similar work

FREE-Switch: Frequency-based Dynamic LoRA Switch for Style Transfer

OrthoTryOn: Geometric Orthogonalization for Conflict-Free Unified Fashion Generation

LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models