CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion

Yu Li; Yujun Cai; Chi Zhang

cs.CVFeb 21, 2026

CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion

Authors: Yu Li, Yujun Cai, Chi Zhang

Organizations: AGI Lab, Westlake University · George Washington University · The University of Queensland

Abstract

Personalized image generation requires effectively balancing content fidelity with stylistic consistency when synthesizing images based on text and reference examples. Low-Rank Adaptation (LoRA) offers an efficient personalization approach, with potential for precise control through combining LoRA weights on different concepts. However, existing combination techniques face persistent challenges: entanglement between content and style representations, insufficient guidance for controlling elements' influence, and unstable weight fusion that often require additional training. We address these limitations through CRAFT-LoRA, with complementary components: (1) rank-constrained backbone fine-tuning that injects low-rank projection residuals to encourage learning decoupled content and style subspaces; (2) a prompt-guided approach featuring an expert encoder with specialized branches that enables semantic extension and precise control through selective adapter aggregation; and (3) a training-free, timestep-dependent classifier-free guidance scheme that enhances generation stability by strategically adjusting noise predictions across diffusion steps. Our method significantly improves content-style disentanglement, enables flexible semantic control over LoRA module combinations, and achieves high-fidelity generation without additional retraining overhead.

Figures & tables

Figure 2: Overview of CRAFT-LoRA , a unified pipeline for personalized image synthesis. In the Training Stage , content ( $\Delta W_{c}$ ) and style ( $\Delta W_{s}$ ) LoRA adapters are decoupled from reference images using a rank-restricted initialization offset. The Prompt Text Guidance module employs an expert system with specialized branches to produce distinct content and style tokens for semantic control. In the Inference Stage , a timestep-aware asymmetric CFG scheme selectively integrates LoRA updates, ensuring stable and high-fidelity image generation.

Figure 3

Figure 4: Visual comparison of content-style combinations. The figure presents a systematic evaluation of different methods in combining specific content elements with distinct artistic styles. Each column showcases the content and style reference followed by the outputs generated. Prompts follow the format “A [content] <c> in [style] <s> ”. Competing methods often fail to simultaneously preserve structure and render style, whereas our method produces consistent and coherent content–style compositions.

Figure 5: Extended visual results. (a) Content–style generations augmented with additional prompt descriptions e.g . “catching a frisbee”, “wearing a hat”, and “driving a car”. Our method flexibly integrates dynamic semantics while preserving the specified content and style. (b) Single-branch generations where only the content LoRA or only the style LoRA is activated. Content-only generations preserve object identity across neutral renderings, while style-only generations transfer artistic features across different subjects.

Figure 6: Quantitative evaluation of our method. (a) Comparison against baselines using CLIP-I similarities and GPT-4o binary judgment scores. (b) Component ablation: Rank-FT contributes most to disentanglement, Router provides content-style routing, and ACFG stabilizes fusion; all three are complementary. The baseline (no components) corresponds to Direct Merging. (c) User study ratings on a 1–5 scale (30 participants, 50 samples).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Property	$I_{c}^{k}$ (content fixed)	$I_{s}^{k}$ (style fixed)
Target image	Generated with fixed content	Generated with fixed style
Base prompt	$P_{c}^{k}$ : “a red car”	$P_{s}^{k}$ : “in the style of Van Gogh”
Varying modifier	$P_{sm}^{k}\in\{\text{watercolor},\ \text{oil painting},\dots\}$	$P_{cm}^{k}\in\{\text{starry night},\ \text{sunflower},\dots\}$
Dominant frequency filter	Low $\mathcal{M}_{\text{low}}$	High $\mathcal{M}_{\text{high}}$
Resulting characteristic	Content preserved with varied textures	Style preserved with varied subjects

Appendix

Table 1: Conceptual illustration of content–style contrast pair generation. For each pair

(I_{c}^{k},I_{s}^{k})

, one varying modifier is chosen for

I_{c}^{k}

and one for

I_{s}^{k}

\mathcal{L}_{\text{task}}(\{W_{l}\},\mathcal{D})=\frac{1}{N}\sum_{k=1}^{N}\mathbb{E}_{t,z_{t}}\Bigg[\big\|G(\{W_{l}\},z_{t},t,\mathrm{TextualEmbeddings}(P_{c}^{k},P_{sm}^{k}))-I_{c}^{k}\big\|_{1}\\ +\big\|G(\{W_{l}\},z_{t},t,\mathrm{TextualEmbeddings}(P_{cm}^{k},P_{s}^{k}))-I_{s}^{k}\big\|_{1}\Bigg]+\alpha\,\mathcal{L}_{\text{perceptual}}(\{W_{l}\},\mathcal{D}).

(15)

Appendix

Table 8

\mathcal{L}_{\text{perceptual}}=\frac{1}{N}\sum_{k=1}^{N}\mathbb{E}_{t,z_{t}}\sum_{j\in\mathrm{VGG\_layers}}\frac{1}{M_{j}}\Bigg[\big\|\phi_{j}\big(G(\{W_{l}\},z_{t},t,\mathrm{TextualEmbeddings}(P_{c}^{k},P_{sm}^{k}))\big)-\phi_{j}(I_{c}^{k})\big\|_{1}\\ +\big\|\phi_{j}\big(G(\{W_{l}\},z_{t},t,\mathrm{TextualEmbeddings}(P_{cm}^{k},P_{s}^{k}))\big)-\phi_{j}(I_{s}^{k})\big\|_{1}\Bigg].

(16)

Appendix

Table 9

Parameter	Value
Optimizer	AdamW
Initial learning rate	$1\text{\times}{10}^{-5}$
LR schedule	Cosine decay with 500 warm-up steps
Minimum learning rate	$1\text{\times}{10}^{-7}$
Batch size	8
Optimization steps	5000

Appendix

Table 2: Summary of backbone fine-tuning hyperparameters.

Parameter	Value
Optimizer	AdamW
Learning rate	$1\text{\times}{10}^{-5}$
Training steps	1000–2000
Batch size	1
Backbone host	$W_{\text{init}}$ (from Eq. 13 )

Appendix

Table 3: LoRA adapter training hyperparameters.

Content layers $(I_{c})$	Style layers $(I_{s})$
down_blocks.2.attentions.0 ; mid_block.attentions.0 ; up_blocks.0.attentions.1 ; up_blocks.1.attentions.0 ; up_blocks.2.attentions.0	down_blocks.0.attentions.0 ; down_blocks.1.resnets.0 ; mid_block.resnets.1 ; down_blocks.0.resnets.1 ; down_blocks.1.attentions.1

Appendix

Table 4: U-Net layer assignments for content

(I_{c})

and style

(I_{s})

adapters.

Metric	Formula	Description
Content preservation $\mathcal{S}_{c}$	$\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}\mathrm{sim}_{c}(\mathcal{G}(p^{\prime}_{c},p_{s}^{j}),p^{\prime}_{c})$	Similarity to target content
Style fidelity $\mathcal{S}_{s}$	$\frac{1}{N_{c}}\sum_{i=1}^{N_{c}}\mathrm{sim}_{s}(\mathcal{G}(p_{c}^{i},p^{\prime}_{s}),p^{\prime}_{s})$	Similarity to target style
Cross-influence $\mathcal{S}_{x}$	$\frac{1}{N_{c}N_{s}}\sum_{i,j}\mathrm{interference}(p_{c}^{i},p_{s}^{j})$	Undesired cross effect

Appendix

Table 5: Disentanglement evaluation metrics.

\mathcal{G}(p_{c},p_{s})

denotes the final image generated from content prompt

p_{c}

and style prompt

p_{s}

. Higher is better for

\mathcal{S}_{c},\mathcal{S}_{s}

; lower is better for

\mathcal{S}_{x}

Method	$\mathcal{S}_{c}\uparrow$	$\mathcal{S}_{s}\uparrow$	$\mathcal{S}_{x}\downarrow$
Standard LoRA	0.75	0.70	0.42
Trunk fine-tune + layer-selective LoRA (Ours)	0.90	0.88	0.18

Appendix

Table 6: Quantitative comparison of content–style disentanglement.

Type	Condition	Description
I	$p^{\prime}_{c}\neq p_{c}^{i},\ p^{\prime}_{s}=p_{s}^{j}$	New content, trained style
II	$p^{\prime}_{c}=p_{c}^{i},\ p^{\prime}_{s}\neq p_{s}^{j}$	Trained content, new style
III	$p^{\prime}_{c}\neq p_{c}^{i},\ p^{\prime}_{s}\neq p_{s}^{j}$	New content, new style

Appendix

Table 7: Types of semantic extension (generalization).

Trained content	Trained style	Inference prompts	Type
“a red car”	“Van Gogh style”	$p^{\prime}_{c}$ : “a blue car”, $p^{\prime}_{s}$ : “Van Gogh style”	I
“a red car”	“Van Gogh style”	$p^{\prime}_{c}$ : “a red car”, $p^{\prime}_{s}$ : “sketch style”	II
“a red car”	“Van Gogh style”	$p^{\prime}_{c}$ : “a bicycle”, $p^{\prime}_{s}$ : “sketch style”	III
“person ID X”	“pixel art”	$p^{\prime}_{c}$ : “person ID Y”, $p^{\prime}_{s}$ : “pixel art”	I
“person ID X”	“pixel art”	$p^{\prime}_{c}$ : “person ID X”, $p^{\prime}_{s}$ : “cyberpunk style”	II

Appendix

Table 8: Semantic extension examples via prompt variation.

Figure 7: Visual results demonstrating content and style adapter performance.

Figure 8: Additional visual results, including ACFG comparisons and semantic extensions.

Content–style pairing	Fixed-weights baseline	ACFG (ours)	Relative improvement
Similar semantics	0.81	0.87	+7.4%
Distant semantics	0.68	0.79	+16.2%
Complex composition	0.65	0.80	+23.1%

Appendix

Table 9: Performance by content–style pairing with ACFG. Metric shown is a composite quality score (higher is better).

Component configuration	Quality score	Relative change
Full ACFG as proposed	0.86	Reference
Constant temporal scaling (no time dependence)	0.78	$-9.3\%$
No low-rank constraint on temporal adjustments	0.82	$-4.7\%$
Standard CFG (uncond. path uses cond. weights)	0.75	$-12.8\%$

Appendix

Table 10: Ablation of ACFG components. Higher indicates better quality.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 444 papers

Dynamic Training-Free Fusion of Subject and Style LoRAs
Feb 17, 2026Qinglong Cao, Yuntian Chen, Chao Ma +1Diffusion-Based Image Generation Model Merging
Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo
A LoRA is Worth a Thousand Pictures
Dec 16, 2024Chenxi Liu, Towaki Takikawa, Alec JacobsonWeight-Space Analysis Low-Rank Adaptation
University of Toronto · Outerport · Adobe Research
Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation
Mar 23, 2026Donald Shenaj, Federico Errica, Antonio CartaLow-Rank Adaptation Adaptive Rank Allocation
University of Pisa · NEC Laboratories Europe

Feb 17, 2026J/K move · Enter open · S save

Dynamic Training-Free Fusion of Subject and Style LoRAs

Qinglong Cao, Yuntian Chen, Chao Ma, Xiaokang Yang

Shanghai Jiao Tong University · Eastern Institute of Technology, Ningbo

Diffusion-Based Image Generation Model Merging Subject-Driven Image Generation Diffusion Model Guidance

Open paper AskPDF arXiv

Recent studies have explored the combination of multiple LoRAs to simultaneously generate user-specified subjects and styles. However, most existing approaches fuse LoRA weights using static statistical heuristics that deviate from LoRA's original purpose of learning adaptive feature adjustments and ignore the randomness of sampled inputs. To address this, we propose a dynamic training-free fusion framework that operates throughout the generation process. During the forward pass, at each LoRA-applied layer, we dynamically compute the KL divergence between the base model's original features and those produced by subject and style LoRAs, respectively, and adaptively select the most appropriate weights for fusion. In the reverse denoising stage, we further refine the generation trajectory by dynamically applying gradient-based corrections derived from objective metrics such as CLIP and DINO scores, providing continuous semantic and stylistic guidance. By integrating these two complementary mechanisms-feature-level selection and metric-guided latent adjustment-across the entire diffusion timeline, our method dynamically achieves coherent subject-style synthesis without any retraining. Extensive experiments across diverse subject-style combinations demonstrate that our approach consistently outperforms state-of-the-art LoRA fusion methods both qualitatively and quantitatively.

Abstract

Image Generation

Explore similar work

Dynamic Training-Free Fusion of Subject and Style LoRAs

A LoRA is Worth a Thousand Pictures

Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation