With the growing availability of open-sourced adapters trained on the same diffusion backbone for diverse scenes and objects, combining these pretrained weights enables low-cost customized generation. However, most existing model merging methods are designed for classification or text generation, and when applied to image generation, they suffer from content drift due to error accumulation across multiple diffusion steps. For image-oriented methods, training-based approaches are computationally expensive and unsuitable for edge deployment, while training-free ones use uniform fusion strategies that ignore inter-adapter differences, leading to detail degradation. We find that since different adapters are specialized for generating different types of content, the contribution of each diffusion step carries different significance for each adapter. Accordingly, we propose a frequency-domain importance-driven dynamic LoRA switch method. Furthermore, we observe that maintaining semantic consistency across adapters effectively mitigates detail loss; thus, we design an automatic Generation Alignment mechanism to align generation intents at the semantic level. Experiments demonstrate that our FREE-Switch (Frequency-based Efficient and Dynamic LoRA Switch) framework efficiently combines adapters for different objects and styles, substantially reducing the training cost of high-quality customized generation.
Figures & tables
Figure 1: Different LoRA combinations require different hyper-parameter selections. (a) and (b) show two sets of LoRA combinations, where a LoRA combination method that works well for one set may not necessarily be suitable for the other.
Figure 2: The framework diagram of FREE-Switch. The Dynamic LoRA Switch module dynamically switches LoRA during the denoising process by analyzing the timestep. The Generation Alignment module automatically expands the prompt to ensure that the outputs remain as aligned as possible during LoRA switching, thereby reducing detail loss.
Figure 3: For content and style generation, we examine how the frequency-domain variation rate across denoising steps affects the final output. As shown in (a) and (b), removing the fastest-varying steps of frequency domain as calculated in Eq. 1 causes a more significant degradation, indicating that rapidly changing steps are essential for maintaining generation fidelity.
CLIP Score (Style) ↑
DINO Score (Content) ↑
Gemini Feedback ↑
Speed (s/pair) ↓
Direct Generation (Refine Prompt)
58.62
36.01
6.67%
150
ZipLoRA [ 26 ]
53.93
69.18
3.33%
550
Joint Train
60.56
27.15
6.67%
589
Merge
60.56
41.44
16.67%
150
K-LoRA [ 19 ]
54.17
61.32
13.33%
290
FREE-Switch
61.59
68.57
53.33%
320
Table 1: Comparison of performance of different model combination methods on SDXL v1.0, where speed denotes the time required to generate 10 images for a given pair of content and style.
Figure 4: Qualitative Analysis. We show the results of generating different content–style combinations using various LoRA combination methods on SDXL v1.0 and FLUX 1. Our FREE-Switch method demonstrates the most consistent and stable performance.
Figure 5: (a) Ablation study on the components of FREE-Switch, showing their impact on content and style preservation. (b) Ablation study on the proposed frequency-domain importance-based dynamic LoRA Switch, highlighting its role in adaptive LoRA selection.
Figure 6: Quantitative analysis of FREE-Switch component ablation on content and style preservation.
Figure 7: Effect of the Output Alignment Method. Incorporating output alignment optimization allows the model to better interpret the original requirements during LoRA switching, resulting in higher-quality images that more accurately reflect user intent.
Figure 8: Analysis of Model Merging Methods for This Task. Model merging tends to accumulate errors throughout the diffusion process, leading to degraded generation quality and loss of fine details. Therefore, we adopt the LoRA Switch strategy instead.
Figure 9: Parameter value distribution of LoRAs trained on different sources based on FLUX.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Notations
Descriptions
f
Pre-trained diffusion model.
θc
Fine-tuning parameters for content.
θs
Fine-tuning parameters for style.
ht
The output of t -th diffuison step.
f(ht−1)
The denoising process of t -th step using base model.
Personalized image generation requires effectively balancing content fidelity with stylistic consistency when synthesizing images based on text and reference examples. Low-Rank Adaptation (LoRA) offers an efficient personalization approach, with potential for precise control through combining LoRA weights on different concepts. However, existing combination techniques face persistent challenges: entanglement between content and style representations, insufficient guidance for controlling elements' influence, and unstable weight fusion that often require additional training. We address these limitations through CRAFT-LoRA, with complementary components: (1) rank-constrained backbone fine-tuning that injects low-rank projection residuals to encourage learning decoupled content and style subspaces; (2) a prompt-guided approach featuring an expert encoder with specialized branches that enables semantic extension and precise control through selective adapter aggregation; and (3) a training-free, timestep-dependent classifier-free guidance scheme that enhances generation stability by strategically adjusting noise predictions across diffusion steps. Our method significantly improves content-style disentanglement, enables flexible semantic control over LoRA module combinations, and achieves high-fidelity generation without additional retraining overhead.
Yu Li, Yujun Cai, Chi Zhang
AGI Lab, Westlake University · George Washington University · The University of Queensland
Recent studies have explored the combination of multiple LoRAs to simultaneously generate user-specified subjects and styles. However, most existing approaches fuse LoRA weights using static statistical heuristics that deviate from LoRA's original purpose of learning adaptive feature adjustments and ignore the randomness of sampled inputs. To address this, we propose a dynamic training-free fusion framework that operates throughout the generation process. During the forward pass, at each LoRA-applied layer, we dynamically compute the KL divergence between the base model's original features and those produced by subject and style LoRAs, respectively, and adaptively select the most appropriate weights for fusion. In the reverse denoising stage, we further refine the generation trajectory by dynamically applying gradient-based corrections derived from objective metrics such as CLIP and DINO scores, providing continuous semantic and stylistic guidance. By integrating these two complementary mechanisms-feature-level selection and metric-guided latent adjustment-across the entire diffusion timeline, our method dynamically achieves coherent subject-style synthesis without any retraining. Extensive experiments across diverse subject-style combinations demonstrate that our approach consistently outperforms state-of-the-art LoRA fusion methods both qualitatively and quantitatively.
Qinglong Cao, Yuntian Chen, Chao Ma +1
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo
In a rapidly growing field of model training there is a constant practical interest in parameter-efficient fine-tuning and various techniques that use a small amount of training data to adapt the model to a narrow task. However, there is an open question: how to combine several adapters tuned for different tasks into one which is able to yield adequate results on both tasks? Specifically, merging subject and style adapters for generative models remains unresolved. In this paper we seek to show that in the case of orthogonal fine-tuning (OFT), we can use structured orthogonal parametrization and its geometric properties to get the formulas for training-free adapter merging. In particular, we derive the structure of the manifold formed by the recently proposed Group-and-Shuffle (GS) orthogonal matrices, and obtain efficient formulas for the geodesics approximation between two points. Additionally, we propose a spectra restoration transform that restores spectral properties of the merged adapter for higher-quality fusion. We conduct experiments in subject-driven generation tasks showing that our technique to merge two GS orthogonal matrices is capable of uniting concept and style features of different adapters. To the best of our knowledge, this is the first training-free method for merging multiplicative orthogonal adapters. Code is available via the link.