CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization

Xiaoman Feng; Mingkun Lei; Yang Wang; Dingwen Fu; Chi Zhang

cs.CVFeb 24, 2026

CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization

Authors: Xiaoman Feng, Mingkun Lei, Yang Wang, Dingwen Fu, Chi Zhang

Organizations: AGI Lab, Westlake University

Abstract

Style transfer in diffusion models enables controllable visual generation by injecting the style of a reference image. However, recent encoder-based methods, while efficient and tuning-free, often suffer from content leakage, where semantic elements from the style image undesirably appear in the output, impairing prompt fidelity and stylistic consistency. In this work, we introduce CleanStyle, a plug-and-play framework that filters out content-related noise from the style embedding without retraining. Motivated by empirical analysis, we observe that such leakage predominantly stems from the tail components of the style embedding, which are isolated via Singular Value Decomposition (SVD). To address this, we propose CleanStyleSVD (CS-SVD), which dynamically suppresses tail components using a time-aware exponential schedule, providing clean, style-preserving conditional embeddings throughout the denoising process. Furthermore, we present Style-Specific Classifier-Free Guidance (SS-CFG), which reuses the suppressed tail components to construct style-aware unconditional inputs. Unlike conventional methods that use generic negative embeddings (e.g., zero vectors), SS-CFG introduces targeted negative signals that reflect style-specific but prompt-irrelevant visual elements. This enables the model to effectively suppress these distracting patterns during generation, thereby improving prompt fidelity and enhancing the overall visual quality of stylized outputs. Our approach is lightweight, interpretable, and can be seamlessly integrated into existing encoder-based diffusion models without retraining. Extensive experiments demonstrate that CleanStyle substantially reduces content leakage, improves stylization quality and improves prompt alignment across a wide range of style references and prompts.

Figures & tables

Figure 1: CleanStyle improves text-aligned style transfer by effectively mitigating content leakage. Compared to InstantStyle, our results better preserve prompt semantics while faithfully reflecting the reference style.

Figure 2: Overview of CleanStyle . We decompose cross-attention style embeddings via SVD into main and tail components, apply time-aware suppression to the tail component in CS-SVD, and form conditional embeddings. From the visualization of singular value (the Key $K$ is used as as an example), at the earlier time step $t_{0}$ , suppression is stronger, while suppression is weaker at the later time step to preserve style details. SS-CFG uses the isolated tail component to build style-aware unconditional inputs. The figure shows the decomposition, the time-dependent filtering, and the conditional/unconditional pathways in sampling.

Figure 3: Motivational illustration. The baseline exhibits clear content leakage. Using only the tail component (as defined in Fig. 2 ) further amplifies these artifacts, indicating that the tail region mainly encodes content-related signals rather than stylistic information. Conversely, relying solely on the main component weakens the overall style expression. These observations motivate our design: CS-SVD suppresses tail-induced content leakage, while the time-aware strategy modulates this suppression to avoid over-attenuating stylistic details, achieving a balanced and faithful stylization.

Figure 4: Qualitative comparison with the state-of-the-art encoder-based style transfer methods. Our approach effectively suppresses content leakage (row 1), achieves stronger prompt alignment (rows 2–4), and maintains higher visual fidelity with fewer structural or stylistic distortions (row 5).

Method	Text Align $\uparrow$	Style Sim $\uparrow$	IQ $\uparrow$
InstantStyle †	0.572	0.565	0.541
InstantStyle	0.133	0.128	0.128
CSGO	0.035	0.030	0.040
IP-Adapter	0.016	0.014	0.016
StyleShot	0.053	0.058	0.056
DEADiff	0.037	0.034	0.033

Table 1: User Study. IQ denotes Image Quality and

{\dagger}

: Baseline integrated with our method.

Method	CLIP TA $\uparrow$	CLIP SS $\uparrow$	DINO SS $\uparrow$	CT(s)
InstantStyle †	0.231	0.711	0.386	4.81
InstantStyle	0.228	0.718	0.396	4.13
DEADiff †	0.245	0.671	0.281	6.81
DEADiff	0.239	0.670	0.307	3.92
StyleShot †	0.227	0.701	0.386	5.47
StyleShot	0.221	0.717	0.431	4.23

Table 2: Quantitative Comparisons on CleanStyle . CT(s) means computation time in seconds. As shown, integrating our modules introduces only marginal overhead, resulting in computation times comparable to the original baselines.

{\dagger}

: Baseline integrated with our method.

Figure 5: Integrated with StyleShot and DEADiff. On both the comparisons, ours mitigate the content leakage issue and keep stylistic features.

Figure 6: Comparison of different design choices. “SVD on $f_{i}$ ” denotes filtering applied directly to the image encoder output. We evaluate four strategies: fixed (w/o time-aware), linear ( $\alpha_{t}=\alpha_{0}(1-\frac{t}{T})$ ), exponential ( $\alpha_{t}=\alpha_{0}e^{-\lambda\frac{t}{T}}$ ), and ours. Our sigmoid-based schedule leads to stronger stylization and achieves the richest stylistic detail.

Figure 7: Qualitative ablation study of CleanStyle . Using SS-CFG alone (w/o CS-SVD, third column) produces outputs visually close to the baseline, indicating that the unconditional pathway cannot function effectively without a properly filtered conditional branch; this observation is also supported by the quantitative results in Tab. 3 . Using CS-SVD alone (w/o SS-CFG, fourth column) suppresses major artifacts but still leaves subtle style-related structures ( e.g . , flowers) that are difficult to remove. Only the full model eliminates content-leakage artifacts while maintaining strong prompt alignment, demonstrating that CS-SVD and SS-CFG are both necessary and complementary.

Method	CLIP TA $\uparrow$	CLIP SS $\uparrow$	DINO SS $\uparrow$
Baseline	0.228	0.718	0.396
w/o CS-SVD	0.229	0.717	0.393
w/o SS-CFG	0.244	0.671	0.390
Ours	0.231	0.711	0.386

Table 3: Quantitative ablation study of CleanStyle . Using SS-CFG alone offers limited improvement, as CFG requires a properly filtered conditional branch to take effect. Using CS-SVD alone improves prompt alignment but noticeably degrades style similarity, reflecting the inherent trade-off between content suppression and style preservation. Combining both modules achieves the most balanced performance across all metrics.

Figure 8: Comparison of different top-k value and $\alpha_{0}$ . Larger $\alpha_{0}$ values apply stronger attenuation to tail components, leading to reduced stylistic influence. Conversely, increasing top- $k$ preserves more dominant singular components, which can reintroduce content leakage from the style image. A moderate setting achieves the best balance between style strength and content cleanliness.

Figure 9: Effect of varying guidance weight $w$ in SS-CFG. SS-CFG maintains stable color distribution, coherent structure, and consistent style expression even at high $w$ .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	CLIP TA $\uparrow$	CLIP SS $\uparrow$	DINO SS $\uparrow$
InstantStyle †	0.231	0.640	0.335
InstantStyle	0.227	0.665	0.386
DEADiff †	0.234	0.602	0.306
DEADiff	0.231	0.619	0.330
StyleShot †	0.222	0.661	0.411
StyleShot	0.215	0.677	0.456

Appendix

Table A.1 : Quantitative Comparisons on StyleBench.

{\dagger}

: Baseline integrated with our method.

	CLIP-TA $\uparrow$ [ 27 ]				CLIP-SS $\uparrow$ [ 27 ]				DINO-SS $\uparrow$ [ 4 ]
	$\alpha_{0}$ =0.01	$\alpha_{0}$ =0.02	$\alpha_{0}$ =0.03	$\alpha_{0}$ =0.05	$\alpha_{0}$ =0.01	$\alpha_{0}$ =0.02	$\alpha_{0}$ =0.03	$\alpha_{0}$ =0.05	$\alpha_{0}$ =0.01	$\alpha_{0}$ =0.02	$\alpha_{0}$ =0.03	$\alpha_{0}$ =0.05
k=0	0.241	0.250	0.251	0.251	0.598	0.576	0.566	0.557	0.317	0.265	0.240	0.222
k=1	0.230	0.235	0.237	0.239	0.625	0.612	0.606	0.599	0.382	0.359	0.348	0.333
k=2	0.223	0.224	0.225	0.226	0.641	0.639	0.637	0.635	0.412	0.408	0.406	0.403
k=3	0.222	0.222	0.223	0.224	0.642	0.641	0.641	0.640	0.412	0.412	0.411	0.410

Appendix

Table A.2 : Hyperparameter study of SVD truncation rank

k

and suppression strength

\alpha_{0}

across CLIP-TA [ 27 ] , CLIP-SS [ 27 ] , and DINO-SS [ 4 ] . Larger

k

preserves more singular components but reintroduces content leakage, while larger

\alpha_{0}

applies stronger suppression and may weaken style richness. A balanced trade-off is achieved around

k{=}1

and

\alpha_{0}{\in}[0.01,\,0.02]

Figure A.1 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.

Figure A.2 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.

Figure A.3 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.

Figure A.4 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.

Figure A.5 : The qualitative comparison between ours (Combined with StyleShot) and StyleShot [ 7 ] .

Figure A.6 : The qualitative comparison between ours (Combined with StyleShot) and StyleShot [ 7 ] . In the case of “A hamster eating a carrot”, the mountain in the background is the leakage content. The leakage content of “A lake with calm water and reflecting.” is the wood shelf.

Figure A.7 : The qualitative comparison between ours (Combined with StyleShot) and StyleShot [ 7 ] . The leakage content of “A house covered with ice and snow.” is green tree. The leakage content of “A mountain goat on a cliff.” is the yellow grass.

Figure A.8 : The qualitative comparison between ours (Combined with DEADiff) and DEADiff [ 26 ] . In the case of “A beautiful latus.”, the lotus leaves in the pond have turned into leaves. In the case of “A palm tree.”, the leaves of the palm tree have become ordinary leaves and lost their characteristics.

Figure A.9 : The qualitative comparison between ours (Combined with DEADiff) and DEADiff [ 26 ] . In the case of “A duck swimming in a small pond.”, the floating duckweed in the pond turned into yellow flowers. In the case of “A person cooking noodles in the kitchen.”, the noodles in the pot and bowl turned into red berries.

Figure A.10 : Style similarity comparison between our method and baselines with visible content leakage. We observe that instances with content leakage ( e.g . , extraneous flowers, backgrounds, or accessories copied from the style reference) often yield higher CLIP-SS [ 27 ] and DINO-SS [ 4 ] scores, despite compromising prompt fidelity and visual coherence. This highlights a potential limitation of current style similarity metrics, which may overestimate stylization quality when semantic content from the style image is inadvertently transferred.

Figure A.11 : The questionnaire format for the user study. Each option represents the generation result of a method under a given style and prompt.

Figure A.12 : The styles used for quantitative comparisons in CleanStyle .

Figure A.13 : The prompts used in the quantitative experiments were derived from StyleAdapter [ 37 ] .

Image Style Transfer

Image style transfer changes an image’s visual appearance to match an artistic, photographic, or other reference while preserving its content.

See all 60 papers

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization
Sep 18, 2026Zhangping Yang, Min Li, Song Yan +4Image Style Transfer Diffusion-Based Image Editing
Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China
Scale-Separated Conditioning for Style-Encoder-Free Diffusion Stylization
Aug 20, 2026Jingtao Zhang, Haorui Gao, Youqing Liang +1Image Style Transfer Conditional Diffusion Models
College of Computing, Georgia Institute of Technology, Atlanta, GA, USA · Courant Institute of Mathematical Sciences, New York University, New York, NY, USA · Department of Computer Science, Brown University, Providence, RI, USA
TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer
Oct 4, 2026Duc Khoan Le, Kim Ngoc Tran, Minh Nhat Le +5Image Style Transfer Diffusion Model Guidance
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Sep 18, 2026J/K move · Enter open · S save

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

Zhangping Yang, Min Li, Song Yan, Rong Gao+3

Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China

Image Style Transfer Diffusion-Based Image Editing Diffusion Model Guidance Representation Disentanglement

Open paper AskPDF arXiv

Reference-guided diffusion stylization aims to transfer visual style from a reference image while preserving the semantics specified by a text prompt. However, image conditioning often entangles transferable style cues with reference-specific content, leading to an inherent trade-off: stronger conditioning improves style fidelity but increases content leakage, whereas aggressive suppression reduces leakage at the cost of style expression. This challenge is further complicated by the distinct spatial organization of texture- and geometry-dominant styles. To address these issues, we propose SafeStyle, a training-free framework for calibrated style residual injection in frozen diffusion models. SafeStyle first estimates style-supported and content-associated subspaces from compact calibration sets, preserving their informative overlap while suppressing useless content variations. It then transports the purified style evidence over adaptive spatial granularity and constrains its effective influence through an explicit residual-norm budget. Experiments across texture- and geometry-dominant styles show that SafeStyle achieves a DINO style similarity of 0.432 while maintaining competitive text alignment. On a semantically disjoint leakage-stress benchmark, it further achieves a DINO style similarity of 0.474 with only 0.8% semantic leakage, demonstrating an effective balance between style fidelity and reference-content suppression.

Abstract

Image Style Transfer

Explore similar work

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

Scale-Separated Conditioning for Style-Encoder-Free Diffusion Stylization

TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer