FREE-Switch: Frequency-based Dynamic LoRA Switch for Style Transfer

Shenghe Zheng; Minyu Zhang; Tianhao Liu; Hongzhi Wang

cs.CVApr 11, 2026

FREE-Switch: Frequency-based Dynamic LoRA Switch for Style Transfer

Authors: Shenghe Zheng, Minyu Zhang, Tianhao Liu, Hongzhi Wang

Organizations: Harbin Institute of Technology

Abstract

With the growing availability of open-sourced adapters trained on the same diffusion backbone for diverse scenes and objects, combining these pretrained weights enables low-cost customized generation. However, most existing model merging methods are designed for classification or text generation, and when applied to image generation, they suffer from content drift due to error accumulation across multiple diffusion steps. For image-oriented methods, training-based approaches are computationally expensive and unsuitable for edge deployment, while training-free ones use uniform fusion strategies that ignore inter-adapter differences, leading to detail degradation. We find that since different adapters are specialized for generating different types of content, the contribution of each diffusion step carries different significance for each adapter. Accordingly, we propose a frequency-domain importance-driven dynamic LoRA switch method. Furthermore, we observe that maintaining semantic consistency across adapters effectively mitigates detail loss; thus, we design an automatic Generation Alignment mechanism to align generation intents at the semantic level. Experiments demonstrate that our FREE-Switch (Frequency-based Efficient and Dynamic LoRA Switch) framework efficiently combines adapters for different objects and styles, substantially reducing the training cost of high-quality customized generation.

Figures & tables

Figure 1: Different LoRA combinations require different hyper-parameter selections. (a) and (b) show two sets of LoRA combinations, where a LoRA combination method that works well for one set may not necessarily be suitable for the other.

Figure 2: The framework diagram of FREE-Switch. The Dynamic LoRA Switch module dynamically switches LoRA during the denoising process by analyzing the timestep. The Generation Alignment module automatically expands the prompt to ensure that the outputs remain as aligned as possible during LoRA switching, thereby reducing detail loss.

Figure 3: For content and style generation, we examine how the frequency-domain variation rate across denoising steps affects the final output. As shown in (a) and (b), removing the fastest-varying steps of frequency domain as calculated in Eq. 1 causes a more significant degradation, indicating that rapidly changing steps are essential for maintaining generation fidelity.

	CLIP Score (Style) $\uparrow$	DINO Score (Content) $\uparrow$	Gemini Feedback $\uparrow$	Speed (s/pair) $\downarrow$
Direct Generation (Refine Prompt)	58.62	36.01	6.67%	150
ZipLoRA [ 26 ]	53.93	69.18	3.33%	550
Joint Train	60.56	27.15	6.67%	589
Merge	60.56	41.44	16.67%	150
K-LoRA [ 19 ]	54.17	61.32	13.33%	290
FREE-Switch	61.59	68.57	53.33%	320

Table 1: Comparison of performance of different model combination methods on SDXL v1.0, where speed denotes the time required to generate 10 images for a given pair of content and style.

Figure 4: Qualitative Analysis. We show the results of generating different content–style combinations using various LoRA combination methods on SDXL v1.0 and FLUX 1. Our FREE-Switch method demonstrates the most consistent and stable performance.

Figure 5: (a) Ablation study on the components of FREE-Switch, showing their impact on content and style preservation. (b) Ablation study on the proposed frequency-domain importance-based dynamic LoRA Switch, highlighting its role in adaptive LoRA selection.

Figure 6: Quantitative analysis of FREE-Switch component ablation on content and style preservation.

Figure 7: Effect of the Output Alignment Method. Incorporating output alignment optimization allows the model to better interpret the original requirements during LoRA switching, resulting in higher-quality images that more accurately reflect user intent.

Figure 8: Analysis of Model Merging Methods for This Task. Model merging tends to accumulate errors throughout the diffusion process, leading to degraded generation quality and loss of fine details. Therefore, we adopt the LoRA Switch strategy instead.

Figure 9: Parameter value distribution of LoRAs trained on different sources based on FLUX.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Notations	Descriptions
$f$	Pre-trained diffusion model.
$\theta_{c}$	Fine-tuning parameters for content.
$\theta_{s}$	Fine-tuning parameters for style.
$h_{t}$	The output of $t$ -th diffuison step.
$f(h_{t-1})$	The denoising process of $t$ -th step using base model.
$f_{\theta}(h_{t-1})$	The denoising process of $t$ -th step with LoRA $\theta$ .

Appendix

Table 2: Notations.

Parameter	Value
rank	64
resolution	1024
train_batch_size	1
learning_rate	5.00E-05
lr_scheduler	constant
lr_warmup_steps	0

Appendix

Table 3: Training Configuations for LoRA.

Figure 10: The Pipeline of Generation Alignment.

Figure 11: Failure case analysis using SDXL v1.0.

Diffusion-Based Image Generation

Diffusion-based image generation creates images by progressively denoising noise, with controls for content, style, and structure.

See all 312 papers

CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion
Feb 21, 2026Yu Li, Yujun Cai, Chi ZhangImage Generation T2I Diffusion Models
AGI Lab, Westlake University · George Washington University · The University of Queensland
Dynamic Training-Free Fusion of Subject and Style LoRAs
Feb 17, 2026Qinglong Cao, Yuntian Chen, Chao Ma +1Diffusion-Based Image Generation Model Merging
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo
OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models
Apr 6, 2026Ali Aliev, Kamil Garifullin, Nikolay Yudin +5Model Merging Diffusion Model Adaptation
HSE University · FusionBrain Lab · AXXX

Feb 21, 2026J/K move · Enter open · S save

CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion

Yu Li, Yujun Cai, Chi Zhang

AGI Lab, Westlake University · George Washington University · The University of Queensland

Image Generation T2I Diffusion Models Low-Rank Adaptation Representation Disentanglement Diffusion Model Guidance

Open paper AskPDF arXiv

Personalized image generation requires effectively balancing content fidelity with stylistic consistency when synthesizing images based on text and reference examples. Low-Rank Adaptation (LoRA) offers an efficient personalization approach, with potential for precise control through combining LoRA weights on different concepts. However, existing combination techniques face persistent challenges: entanglement between content and style representations, insufficient guidance for controlling elements' influence, and unstable weight fusion that often require additional training. We address these limitations through CRAFT-LoRA, with complementary components: (1) rank-constrained backbone fine-tuning that injects low-rank projection residuals to encourage learning decoupled content and style subspaces; (2) a prompt-guided approach featuring an expert encoder with specialized branches that enables semantic extension and precise control through selective adapter aggregation; and (3) a training-free, timestep-dependent classifier-free guidance scheme that enhances generation stability by strategically adjusting noise predictions across diffusion steps. Our method significantly improves content-style disentanglement, enables flexible semantic control over LoRA module combinations, and achieves high-fidelity generation without additional retraining overhead.

Abstract

Diffusion-Based Image Generation

Explore similar work

CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion

Dynamic Training-Free Fusion of Subject and Style LoRAs

OrthoFuse: Training-free Riemannian Fusion of Orthogonal Style-Concept Adapters for Diffusion Models