SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

Zhangping Yang; Min Li; Song Yan; Rong Gao; Xinliang Bi; Guanye Xiong; Yujie He

cs.CVSep 18, 2026

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

Authors: Zhangping Yang, Min Li, Song Yan, Rong Gao, Xinliang Bi, Guanye Xiong, Yujie He

Organizations: Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China

Abstract

Reference-guided diffusion stylization aims to transfer visual style from a reference image while preserving the semantics specified by a text prompt. However, image conditioning often entangles transferable style cues with reference-specific content, leading to an inherent trade-off: stronger conditioning improves style fidelity but increases content leakage, whereas aggressive suppression reduces leakage at the cost of style expression. This challenge is further complicated by the distinct spatial organization of texture- and geometry-dominant styles. To address these issues, we propose SafeStyle, a training-free framework for calibrated style residual injection in frozen diffusion models. SafeStyle first estimates style-supported and content-associated subspaces from compact calibration sets, preserving their informative overlap while suppressing useless content variations. It then transports the purified style evidence over adaptive spatial granularity and constrains its effective influence through an explicit residual-norm budget. Experiments across texture- and geometry-dominant styles show that SafeStyle achieves a DINO style similarity of 0.432 while maintaining competitive text alignment. On a semantically disjoint leakage-stress benchmark, it further achieves a DINO style similarity of 0.474 with only 0.8% semantic leakage, demonstrating an effective balance between style fidelity and reference-content suppression.

Figures & tables

Figure 1: Style groups share non-zero content-style overlap but respond differently to spatial permutation, motivating common purification and adaptive transport granularity.

Figure 2: SafeStyle uses calibration sets to remove unsupported content evidence, transports the purified residual at adaptive spatial granularity, and injects it into a frozen U-Net under an actual residual-norm budget.

Figure 3: Controlled diagnostics. (a) Hard content projection also removes supported style evidence. (b) The effective residual budget governs when purified evidence becomes visible and when reference structure begins to dominate.

Figure 4: Representative texture (top) and geometry (bottom) comparisons. SafeStyle preserves the target subject while retaining stronger local appearance and structured style evidence.

	Overall			Leakage Stress Set
Method	TA $\uparrow$	C-SS $\uparrow$	D-SS $\uparrow$	C-SS $\uparrow$	D-SS $\uparrow$	Leak $\downarrow$
IP-Adapter [ 3 ]	0.085	0.753	0.454	0.754	0.518	1.000
InstantStyle [ 4 ]	0.229	0.726	0.354	0.711	0.361	0.364
StyleID [ 9 ]	0.221	0.709	0.320	0.684	0.276	0.000
CSGO [ 6 ]	0.233	0.701	0.304	0.681	0.357	0.012
DEADiff [ 5 ]	0.243	0.702	0.336	0.684	0.384	0.004

Table 1: Overall and leakage stress set comparisons. For brevity, C-SS, D-SS, and Leak denote CLIP-SS, DINO-SS, and Sem-Leak. Best results are in bold

Configuration	CLIP-TA $\uparrow$	DINO-SS $\uparrow$	Sem-Leak $\downarrow$
Raw evidence	0.190	0.485	0.279
Hard projection	0.224	0.445	0.008
Safe purification	0.198	0.530	0.167
$+$ Transport	0.235	0.498	0.029
$+$ Budget (Ours)	0.238	0.474	0.008

Table 2: Component ablation on the leakage-stress benchmark.

Figure 5: Leakage-stress examples with semantically disjoint reference and target subjects. SafeStyle retains stronger style evidence while suppressing subject transfer.

Figure 6: Unified ablation on oil-painting and paper craft references. The same purification, granularity transport, and residual budget are evaluated across both style groups.

Image Style Transfer

Image style transfer changes an image’s visual appearance to match an artistic, photographic, or other reference while preserving its content.

See all 60 papers

CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization
Feb 24, 2026Xiaoman Feng, Mingkun Lei, Yang Wang +2Image Style Transfer T2I Diffusion Models
AGI Lab, Westlake University
TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer
Oct 4, 2026Duc Khoan Le, Kim Ngoc Tran, Minh Nhat Le +5Image Style Transfer Diffusion Model Guidance
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam
Scale-Separated Conditioning for Style-Encoder-Free Diffusion Stylization
Aug 20, 2026Jingtao Zhang, Haorui Gao, Youqing Liang +1Image Style Transfer Conditional Diffusion Models
College of Computing, Georgia Institute of Technology, Atlanta, GA, USA · Courant Institute of Mathematical Sciences, New York University, New York, NY, USA · Department of Computer Science, Brown University, Providence, RI, USA

Feb 24, 2026J/K move · Enter open · S save

CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization

Xiaoman Feng, Mingkun Lei, Yang Wang, Dingwen Fu+1

AGI Lab, Westlake University

Image Style Transfer T2I Diffusion Models T2I Generation Diffusion Model Guidance

Open paper AskPDF arXiv

Style transfer in diffusion models enables controllable visual generation by injecting the style of a reference image. However, recent encoder-based methods, while efficient and tuning-free, often suffer from content leakage, where semantic elements from the style image undesirably appear in the output, impairing prompt fidelity and stylistic consistency. In this work, we introduce CleanStyle, a plug-and-play framework that filters out content-related noise from the style embedding without retraining. Motivated by empirical analysis, we observe that such leakage predominantly stems from the tail components of the style embedding, which are isolated via Singular Value Decomposition (SVD). To address this, we propose CleanStyleSVD (CS-SVD), which dynamically suppresses tail components using a time-aware exponential schedule, providing clean, style-preserving conditional embeddings throughout the denoising process. Furthermore, we present Style-Specific Classifier-Free Guidance (SS-CFG), which reuses the suppressed tail components to construct style-aware unconditional inputs. Unlike conventional methods that use generic negative embeddings (e.g., zero vectors), SS-CFG introduces targeted negative signals that reflect style-specific but prompt-irrelevant visual elements. This enables the model to effectively suppress these distracting patterns during generation, thereby improving prompt fidelity and enhancing the overall visual quality of stylized outputs. Our approach is lightweight, interpretable, and can be seamlessly integrated into existing encoder-based diffusion models without retraining. Extensive experiments demonstrate that CleanStyle substantially reduces content leakage, improves stylization quality and improves prompt alignment across a wide range of style references and prompts.

Abstract

Image Style Transfer

Explore similar work

CleanStyle: Plug-and-Play Style Conditioning Purification for Text-to-Image Stylization

TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer

Scale-Separated Conditioning for Style-Encoder-Free Diffusion Stylization