cs.CVFeb 24, 2026

CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization

Authors: Xiaoman Feng, Mingkun Lei, Yang Wang, Dingwen Fu, Chi Zhang

Organizations: AGI Lab, Westlake University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Style transfer in diffusion models enables controllable visual generation by injecting the style of a reference image. However, recent encoder-based methods, while efficient and tuning-free, often suffer from content leakage, where semantic elements from the style image undesirably appear in the output, impairing prompt fidelity and stylistic consistency. In this work, we introduce CleanStyle, a plug-and-play framework that filters out content-related noise from the style embedding without retraining. Motivated by empirical analysis, we observe that such leakage predominantly stems from the tail components of the style embedding, which are isolated via Singular Value Decomposition (SVD). To address this, we propose CleanStyleSVD (CS-SVD), which dynamically suppresses tail components using a time-aware exponential schedule, providing clean, style-preserving conditional embeddings throughout the denoising process. Furthermore, we present Style-Specific Classifier-Free Guidance (SS-CFG), which reuses the suppressed tail components to construct style-aware unconditional inputs. Unlike conventional methods that use generic negative embeddings (e.g., zero vectors), SS-CFG introduces targeted negative signals that reflect style-specific but prompt-irrelevant visual elements. This enables the model to effectively suppress these distracting patterns during generation, thereby improving prompt fidelity and enhancing the overall visual quality of stylized outputs. Our approach is lightweight, interpretable, and can be seamlessly integrated into existing encoder-based diffusion models without retraining. Extensive experiments demonstrate that CleanStyle substantially reduces content leakage, improves stylization quality and improves prompt alignment across a wide range of style references and prompts.

Figures & tables

Figure 1: CleanStyle improves text-aligned style transfer by effectively mitigating content leakage. Compared to InstantStyle, our results better preserve prompt semantics while faithfully reflecting the reference style.
Figure 2: Overview of CleanStyle . We decompose cross-attention style embeddings via SVD into main and tail components, apply time-aware suppression to the tail component in CS-SVD, and form conditional embeddings. From the visualization of singular value (the Key KK is used as as an example), at the earlier time step t0t_{0} , suppression is stronger, while suppression is weaker at the later time step to preserve style details. SS-CFG uses the isolated tail component to build style-aware unconditional inputs. The figure shows the decomposition, the time-dependent filtering, and the conditional/unconditional pathways in sampling.
Figure 3: Motivational illustration. The baseline exhibits clear content leakage. Using only the tail component (as defined in Fig. 2 ) further amplifies these artifacts, indicating that the tail region mainly encodes content-related signals rather than stylistic information. Conversely, relying solely on the main component weakens the overall style expression. These observations motivate our design: CS-SVD suppresses tail-induced content leakage, while the time-aware strategy modulates this suppression to avoid over-attenuating stylistic details, achieving a balanced and faithful stylization.
Figure 4: Qualitative comparison with the state-of-the-art encoder-based style transfer methods. Our approach effectively suppresses content leakage (row 1), achieves stronger prompt alignment (rows 2–4), and maintains higher visual fidelity with fewer structural or stylistic distortions (row 5).
MethodText Align ↑\uparrowStyle Sim ↑\uparrowIQ ↑\uparrow
InstantStyle †0.5720.5650.541
InstantStyle0.1330.1280.128
CSGO0.0350.0300.040
IP-Adapter0.0160.0140.016
StyleShot0.0530.0580.056
DEADiff0.0370.0340.033
Table 1: User Study. IQ denotes Image Quality and †{\dagger} : Baseline integrated with our method.
MethodCLIP TA ↑\uparrowCLIP SS ↑\uparrowDINO SS ↑\uparrowCT(s)
InstantStyle †0.2310.7110.3864.81
InstantStyle0.2280.7180.3964.13
DEADiff †0.2450.6710.2816.81
DEADiff0.2390.6700.3073.92
StyleShot †0.2270.7010.3865.47
StyleShot0.2210.7170.4314.23
Table 2: Quantitative Comparisons on CleanStyle . CT(s) means computation time in seconds. As shown, integrating our modules introduces only marginal overhead, resulting in computation times comparable to the original baselines. †{\dagger} : Baseline integrated with our method.
Figure 5: Integrated with StyleShot and DEADiff. On both the comparisons, ours mitigate the content leakage issue and keep stylistic features.
Figure 6: Comparison of different design choices. “SVD on fif_{i} ” denotes filtering applied directly to the image encoder output. We evaluate four strategies: fixed (w/o time-aware), linear ( αt=α0(1−tT)\alpha_{t}=\alpha_{0}(1-\frac{t}{T}) ), exponential ( αt=α0e−λtT\alpha_{t}=\alpha_{0}e^{-\lambda\frac{t}{T}} ), and ours. Our sigmoid-based schedule leads to stronger stylization and achieves the richest stylistic detail.
Figure 7: Qualitative ablation study of CleanStyle . Using SS-CFG alone (w/o CS-SVD, third column) produces outputs visually close to the baseline, indicating that the unconditional pathway cannot function effectively without a properly filtered conditional branch; this observation is also supported by the quantitative results in Tab. 3 . Using CS-SVD alone (w/o SS-CFG, fourth column) suppresses major artifacts but still leaves subtle style-related structures ( e.g . , flowers) that are difficult to remove. Only the full model eliminates content-leakage artifacts while maintaining strong prompt alignment, demonstrating that CS-SVD and SS-CFG are both necessary and complementary.
MethodCLIP TA ↑\uparrowCLIP SS ↑\uparrowDINO SS ↑\uparrow
Baseline0.2280.7180.396
w/o CS-SVD0.2290.7170.393
w/o SS-CFG0.2440.6710.390
Ours0.2310.7110.386
Table 3: Quantitative ablation study of CleanStyle . Using SS-CFG alone offers limited improvement, as CFG requires a properly filtered conditional branch to take effect. Using CS-SVD alone improves prompt alignment but noticeably degrades style similarity, reflecting the inherent trade-off between content suppression and style preservation. Combining both modules achieves the most balanced performance across all metrics.
Figure 8: Comparison of different top-k value and α0\alpha_{0} . Larger α0\alpha_{0} values apply stronger attenuation to tail components, leading to reduced stylistic influence. Conversely, increasing top- kk preserves more dominant singular components, which can reintroduce content leakage from the style image. A moderate setting achieves the best balance between style strength and content cleanliness.
Figure 9: Effect of varying guidance weight ww in SS-CFG. SS-CFG maintains stable color distribution, coherent structure, and consistent style expression even at high ww .
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

MethodCLIP TA ↑\uparrowCLIP SS ↑\uparrowDINO SS ↑\uparrow
InstantStyle †0.2310.6400.335
InstantStyle0.2270.6650.386
DEADiff †0.2340.6020.306
DEADiff0.2310.6190.330
StyleShot †0.2220.6610.411
StyleShot0.2150.6770.456
Appendix
Table A.1 : Quantitative Comparisons on StyleBench. †{\dagger} : Baseline integrated with our method.
 CLIP-TA ↑\uparrow [ 27 ]CLIP-SS ↑\uparrow [ 27 ]DINO-SS ↑\uparrow [ 4 ]
 α0\alpha_{0} =0.01α0\alpha_{0} =0.02α0\alpha_{0} =0.03α0\alpha_{0} =0.05α0\alpha_{0} =0.01α0\alpha_{0} =0.02α0\alpha_{0} =0.03α0\alpha_{0} =0.05α0\alpha_{0} =0.01α0\alpha_{0} =0.02α0\alpha_{0} =0.03α0\alpha_{0} =0.05
k=00.2410.2500.2510.2510.5980.5760.5660.5570.3170.2650.2400.222
k=10.2300.2350.2370.2390.6250.6120.6060.5990.3820.3590.3480.333
k=20.2230.2240.2250.2260.6410.6390.6370.6350.4120.4080.4060.403
k=30.2220.2220.2230.2240.6420.6410.6410.6400.4120.4120.4110.410
Appendix
Table A.2 : Hyperparameter study of SVD truncation rank kk and suppression strength α0\alpha_{0} across CLIP-TA [ 27 ] , CLIP-SS [ 27 ] , and DINO-SS [ 4 ] . Larger kk preserves more singular components but reintroduces content leakage, while larger α0\alpha_{0} applies stronger suppression and may weaken style richness. A balanced trade-off is achieved around k=1k{=}1 and α0∈[0.01, 0.02]\alpha_{0}{\in}[0.01,\,0.02] .
Figure A.1 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.
Figure A.2 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.
Figure A.3 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.
Figure A.4 : Qualitative comparison with the state-of-the-art encoder-based style transfer methods.
Figure A.5 : The qualitative comparison between ours (Combined with StyleShot) and StyleShot [ 7 ] .
Figure A.6 : The qualitative comparison between ours (Combined with StyleShot) and StyleShot [ 7 ] . In the case of “A hamster eating a carrot”, the mountain in the background is the leakage content. The leakage content of “A lake with calm water and reflecting.” is the wood shelf.
Figure A.7 : The qualitative comparison between ours (Combined with StyleShot) and StyleShot [ 7 ] . The leakage content of “A house covered with ice and snow.” is green tree. The leakage content of “A mountain goat on a cliff.” is the yellow grass.
Figure A.8 : The qualitative comparison between ours (Combined with DEADiff) and DEADiff [ 26 ] . In the case of “A beautiful latus.”, the lotus leaves in the pond have turned into leaves. In the case of “A palm tree.”, the leaves of the palm tree have become ordinary leaves and lost their characteristics.
Figure A.9 : The qualitative comparison between ours (Combined with DEADiff) and DEADiff [ 26 ] . In the case of “A duck swimming in a small pond.”, the floating duckweed in the pond turned into yellow flowers. In the case of “A person cooking noodles in the kitchen.”, the noodles in the pot and bowl turned into red berries.
Figure A.10 : Style similarity comparison between our method and baselines with visible content leakage. We observe that instances with content leakage ( e.g . , extraneous flowers, backgrounds, or accessories copied from the style reference) often yield higher CLIP-SS [ 27 ] and DINO-SS [ 4 ] scores, despite compromising prompt fidelity and visual coherence. This highlights a potential limitation of current style similarity metrics, which may overestimate stylization quality when semantic content from the style image is inadvertently transferred.
Figure A.11 : The questionnaire format for the user study. Each option represents the generation result of a method under a given style and prompt.
Figure A.12 : The styles used for quantitative comparisons in CleanStyle .
Figure A.13 : The prompts used in the quantitative experiments were derived from StyleAdapter [ 37 ] .
More in this topic

Image Style Transfer

Image style transfer changes an image’s visual appearance to match an artistic, photographic, or other reference while preserving its content.

See all 60 papers

Explore similar work

CardsList
Sep 18, 2026

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

Reference-guided diffusion stylization aims to transfer visual style from a reference image while preserving the semantics specified by a text prompt. However, image conditioning often entangles transferable style cues with reference-specific content, leading to an inherent trade-off: stronger conditioning improves style fidelity but increases content leakage, whereas aggressive suppression reduces leakage at the cost of style expression. This challenge is further complicated by the distinct spatial organization of texture- and geometry-dominant styles. To address these issues, we propose SafeStyle, a training-free framework for calibrated style residual injection in frozen diffusion models. SafeStyle first estimates style-supported and content-associated subspaces from compact calibration sets, preserving their informative overlap while suppressing useless content variations. It then transports the purified style evidence over adaptive spatial granularity and constrains its effective influence through an explicit residual-norm budget. Experiments across texture- and geometry-dominant styles show that SafeStyle achieves a DINO style similarity of 0.432 while maintaining competitive text alignment. On a semantically disjoint leakage-stress benchmark, it further achieves a DINO style similarity of 0.474 with only 0.8% semantic leakage, demonstrating an effective balance between style fidelity and reference-content suppression.

Zhangping Yang, Min Li, Song Yan +4

Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China

Image Style TransferDiffusion-Based Image EditingDiffusion Model GuidanceRepresentation Disentanglement
AskPDF
Aug 20, 2026

Scale-Separated Conditioning for Style-Encoder-Free Diffusion Stylization

Reference-based diffusion stylization requires separating target geometry from transferable appearance. Existing tuning-based methods often rely on aligned content-style-target triplets or auxiliary visual encoders, which increases data cost and can transfer unintended scene structure from the style reference. We propose SEFS (Style-Encoder-Free Stylization), a style-encoder-free conditioning framework for diffusion transformers. SEFS forms style tokens from stochastic low-resolution crops of single training images. This crop bottleneck preserves local appearance statistics such as palette, stroke, texture, and material, while reducing access to global layout cues. Target content is encoded by edge and segmentation cues and fused with the noisy latent through parameter-efficient trainable projections. We add style-to-denoising re-normalization for token-statistic alignment and cross-block skip fusion for spatial detail. SEFS trains on unpaired single images; the frozen diffusion VAE is used only to place image conditions in the latent space. On artistic stylization benchmarks, SEFS improves content consistency and leakage diagnostics while retaining reference-style affinity, and ablations support the crop-resolution, re-normalization, and skip-fusion choices. The code of SEFS will be made publicly available.

Jingtao Zhang, Haorui Gao, Youqing Liang +1

College of Computing, Georgia Institute of Technology, Atlanta, GA, USA · Courant Institute of Mathematical Sciences, New York University, New York, NY, USA · Department of Computer Science, Brown University, Providence, RI, USA

Image Style TransferConditional Diffusion ModelsDiffusion-Based Image EditingDiffusion Transformer
AskPDF
Oct 4, 2026

TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer

Reference-guided style transfer aims to preserve the semantic structure of a content image while transferring the visual appearance of a style reference. Recent diffusion-based methods achieve impressive stylization quality by exploiting strong pretrained generative priors. However, training-free approaches still face a difficult trade-off among style fidelity, content preservation, and content leakage. Direct style injection may unintentionally transfer semantic content from the style image, while fixed guidance schedules often ignore the time- and state-dependent nature of diffusion sampling. To address these limitations, we propose TRACE, a training-free diffusion style transfer framework with Time-adaptive Residual Attention Control and Content-Style Decomposition. TRACE first performs offline CLIP-based subspace analysis to separate content and style directions from paired data. During inference, it removes content-related components from the style reference and style-related components from the content reference to reduce leakage. It then injects style information through residual cross-attention and applies uncertainty-aware guidance to adapt the guidance signal at each denoising step. Experiments show that TRACE achieves a favorable trade-off between stylization and preservation. Compared with optimal-control-based baselines, TRACE substantially improves style fidelity (+17.28 CSD and +34.10 SRA). While, compared with stylization methods, it better preserves content structure (+12.80 DINO, +5.52 CLIP-I, and -8.19 LPIPS) and reduces directional semantic leakage by 29.5% in DCL. Our code is publicly available at https://github.com/pixelchemy-research/TRACE.

Duc Khoan Le, Kim Ngoc Tran, Minh Nhat Le +5

Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Image Style TransferDiffusion Model GuidanceAttention Control in Diffusion Models
AskPDF