Recent studies have explored the combination of multiple LoRAs to simultaneously generate user-specified subjects and styles. However, most existing approaches fuse LoRA weights using static statistical heuristics that deviate from LoRA's original purpose of learning adaptive feature adjustments and ignore the randomness of sampled inputs. To address this, we propose a dynamic training-free fusion framework that operates throughout the generation process. During the forward pass, at each LoRA-applied layer, we dynamically compute the KL divergence between the base model's original features and those produced by subject and style LoRAs, respectively, and adaptively select the most appropriate weights for fusion. In the reverse denoising stage, we further refine the generation trajectory by dynamically applying gradient-based corrections derived from objective metrics such as CLIP and DINO scores, providing continuous semantic and stylistic guidance. By integrating these two complementary mechanisms-feature-level selection and metric-guided latent adjustment-across the entire diffusion timeline, our method dynamically achieves coherent subject-style synthesis without any retraining. Extensive experiments across diverse subject-style combinations demonstrate that our approach consistently outperforms state-of-the-art LoRA fusion methods both qualitatively and quantitatively.
Figures & tables
Figure 1: (a) Existing methods directly rely on properties of LoRA weights to achieve fusion. (b) Our method integrates feature-level selection in the forward pass and latent-level refinement in the reverse process to enable dynamic training-free LoRAs fusion.
Figure 2: Overview of our method. By performing dynamic feature selection based on representation perturbation and applying metric-guided refinement throughout the denoising process, our framework enables training-free fusion of subject and style LoRAs.
Method
Style Sim ↑
CLIP Score ↑
DINO Score ↑
Direct
48.9%
66.6%
43.0%
B-LoRA [ 11 ]
58.0%
63.8%
30.6%
ZipLoRA [ 32 ]
60.4%
64.4%
35.7%
K-LoRA [ 28 ]
58.7%
69.4%
46.9%
Ours
63.0%
78.5%
43.3%
Table 1: Comparison of alignment results. Direct denotes direct arithmetic merging.
Method
User Preference
GPT-4o Feedback
Qwen2.5-VL Feedback
ZipLoRA [ 32 ]
13.80%
20.13%
3.40%
B-LoRA [ 11 ]
21.89%
11.67%
9.11%
K-LoRA [ 28 ]
11.11%
12.56%
21.82%
Ours
53.20%
55.64%
65.67%
Table 2: The Performance Comparison of user study results, GPT-4o and Qwen2.5-VL feedback.
Figure 3: Qualitative comparisons. We present images generated by our method and the compared advanced generation methods. Through nput-adaptive, representation-aware decisions throughout the generation process, our method effectively enables training-free fusion of subject and style LoRAs.
Figure 4: Robustness Analysis. We present images generated by our method and K-LoRA with random seeds to analyze the robustness.
Method
Style Sim ↑
CLIP Score ↑
DINO Score ↑
Baseline
60.3%
75.6%
40.1%
Only with FLS
59.5%
75.9%
43.7%
Only with LLR
66.1%
78.9%
40.7%
Ours(FLS+LLR)
64.0%
79.1%
43.4%
Table 3: The ablation study of different components, including Feature-Level Selection (FLS) and Latent-Level Refinement (LLR).
Divergence
Style Sim ↑
CLIP Score ↑
DINO Score ↑
KL
59.5%
75.9%
43.7%
JS
59.3%
75.3%
43.1%
Cosine Similarity
58.9%
75.4%
43.3%
Dot Product
58.4%
74.9%
43.8%
Table 4: The ablation study of selection criteria in feature-level selection, including Kullback-Leibler (KL), Jensen-Shannon (JS) divergence, cosine similarity, and dot product.
Scaling Factor m
Style Sim ↑
CLIP Score ↑
DINO Score ↑
1
62.1%
72.8%
32.4%
5
64.8%
74.1%
32.3%
10
64.0%
79.1%
43.4%
20
66.8%
76.8%
35.1%
Table 5: The ablation study of scaling factor m for Latent-Level Refinement.
Figure 5: LoRA Selection During the Generation Process. The vertical axis represents 50 diffusion steps, while the horizontal axis denotes varying LoRA layers. Dark green indicates the selection of subject LoRA, and light blue indicates the selection of style LoRA.
Method
ViT Content ↑
ViT Style ↑
BLIP-2 Content ↑
BLIP-2 Style ↑
ZipLoRA [ 32 ]
23.2%
18.0%
48.2%
46.0%
B-LoRA [ 11 ]
29.6%
19.9%
42.0%
54.3%
K-LoRA [ 28 ]
32.8%
20.7%
49.4%
51.1%
Ours
33.7%
21.8%
49.5%
51.4%
Table 1: Comparison of alignment results with additional evaluation metrics.
Figure 1: Additional results generated using FLUX. Each image corresponds to the object label indicated above and the style reference on the left. The results demonstrate the effects of applying different LoRA modules through our proposed method.
Figure 2: Additional results generated using FLUX. Each image corresponds to the object label indicated above and the style reference on the left. The results demonstrate the effects of applying different LoRA modules through our proposed method.
Personalized image generation requires effectively balancing content fidelity with stylistic consistency when synthesizing images based on text and reference examples. Low-Rank Adaptation (LoRA) offers an efficient personalization approach, with potential for precise control through combining LoRA weights on different concepts. However, existing combination techniques face persistent challenges: entanglement between content and style representations, insufficient guidance for controlling elements' influence, and unstable weight fusion that often require additional training. We address these limitations through CRAFT-LoRA, with complementary components: (1) rank-constrained backbone fine-tuning that injects low-rank projection residuals to encourage learning decoupled content and style subspaces; (2) a prompt-guided approach featuring an expert encoder with specialized branches that enables semantic extension and precise control through selective adapter aggregation; and (3) a training-free, timestep-dependent classifier-free guidance scheme that enhances generation stability by strategically adjusting noise predictions across diffusion steps. Our method significantly improves content-style disentanglement, enables flexible semantic control over LoRA module combinations, and achieves high-fidelity generation without additional retraining overhead.
Yu Li, Yujun Cai, Chi Zhang
AGI Lab, Westlake University · George Washington University · The University of Queensland
With the growing availability of open-sourced adapters trained on the same diffusion backbone for diverse scenes and objects, combining these pretrained weights enables low-cost customized generation. However, most existing model merging methods are designed for classification or text generation, and when applied to image generation, they suffer from content drift due to error accumulation across multiple diffusion steps. For image-oriented methods, training-based approaches are computationally expensive and unsuitable for edge deployment, while training-free ones use uniform fusion strategies that ignore inter-adapter differences, leading to detail degradation. We find that since different adapters are specialized for generating different types of content, the contribution of each diffusion step carries different significance for each adapter. Accordingly, we propose a frequency-domain importance-driven dynamic LoRA switch method. Furthermore, we observe that maintaining semantic consistency across adapters effectively mitigates detail loss; thus, we design an automatic Generation Alignment mechanism to align generation intents at the semantic level. Experiments demonstrate that our FREE-Switch (Frequency-based Efficient and Dynamic LoRA Switch) framework efficiently combines adapters for different objects and styles, substantially reducing the training cost of high-quality customized generation.
Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning of image diffusion models, with over 200K models trained to replicate artistic styles and shared in online communities. One way to analyze styles of LoRA models is to generate image samples and apply image-based features like CLIP. However, this generation requires additional computation and prompt selection. In this work, we propose analyzing styles directly in the LoRA weight space, bypassing these limitations while achieving better performance even on unseen LoRAs. Since in-the-wild LoRAs lack ground truth model details, we design an experimental setup and show that LoRA weights alone serve as effective feature representations of adapted styles, robust to variations in fine-tuning seeds and image selection. We extensively evaluate using SD1.5 (the most popular base model within online communities) across data domains (fine-art paintings, architectural photos) and diverse prompt settings, with a small-scale SDXL experiment further verifying generalization. These evaluations confirm that our approach, under the selected aspects of variability, preserves fine-grained styles and outperforms image-based methods in clustering and retrieval of unseen LoRAs.
Chenxi Liu, Towaki Takikawa, Alec Jacobson
University of Toronto · Outerport · Adobe Research