Transfer the Style, Not the Semantics: Mitigating Semantic Leakage in Image-Guided Stylization
Authors: Haojun Tang, Qiwei Lin, Tongda Xu, Wenda Zhao, Yian Zhao, Yu Zhang, Lida Huang, Yan Wang
Organizations: Institute for AI Industry Research (AIR), Tsinghua University · School of Information and Communication Engineering, Dalian University of Technology · Beijing Institute of Radio Measurement · School of Information Engineering, Peking University
Image-guided zero-shot style transfer enables training-free stylization of arbitrary content images using a reference style. Despite recent progress, existing methods commonly rely on self-attention manipulation to inject style information, which may transfer semantic content from the style image, leading to semantic leakage. In this work, we revisit this phenomenon from the perspective of key--value coupling in the self-attention of diffusion models. Our analysis shows that the native correspondence between style keys and values plays an important role in transferring semantic information from the style image. Motivated by this observation, we propose SuRe, a timestep-aware key--value regulation framework that mitigates semantic leakage by disrupting the native key--value correspondence while progressively restoring it for faithful style transfer. Specifically, we present value shuffle to decouple style keys and values while preserving the empirical distribution of the values, together with a progressive recoupling mechanism that gradually recouples them throughout the stylization process. Experiments against 18 representative methods demonstrate that SuRe maintains competitive style similarity while consistently improving content similarity across SD1.5, SDXL, SD3, and FLUX.1-dev, with natural generalization to multi-style transfer. The code will be publicly available at https://xinr-tang.github.io/SuRe-homepage/.
Figures & tables
Figure 2: Qualitative comparison of different key–value configurations. Preserving the K–V pairing leads to semantic leakage, while value-only reordering suppresses leaked semantics.
Figure 3: Linear probing of different key–value configurations. Preserving the native K–V pairing retains the semantics of the style image, while value-only reordering reduces it.
Figure 4: Overview of SuRe. The content and style images are first encoded into image latents. V-Shuffle permutes the style value tokens while keeping the style keys fixed, breaking their native correspondence while preserving the empirical value distribution. PRM then progressively combines shuffled and native attention during sampling, suppressing style semantics while refining the target style.
Figure 5: PCA visualization of Vst extracted from the same zs0 at different timesteps. Coarse semantic layouts emerge in the early stage (first 30%) and become clearer in the intermediate stage (middle 40%), while high-frequency stylistic patterns gradually emerge in the late stage (last 30%). We visualize Vst from the 12th self-attention layer of SD1.5, the 45th of SDXL, the 19th of SD3, and the 51st of FLUX.
Category
Method
Paradigm
Style Similarity
Content Similarity
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
Feed-forward methods
AesPA-Net (ICCV’23) [ 11 ]
CNN
31.426
19.763
2.177
0.5135
0.2465
0.5119
EFDM (CVPR’22) [ 31 ]
CNN
30.146
17.344
3.061
0.6434
0.3134
0.4502
CAST (SIGGRAPH’22) [ 33 ]
CNN
34.685
20.395
2.234
0.6212
0.2918
0.5587
StyTR 2 (CVPR’22) [ 6 ]
Transformer
29.887
18.293
1.700
0.5491
0.2852
0.6098
AdaAttN (ICCV’21) [ 17 ]
Transformer
30.933
18.242
2.356
0.6076
0.3155
0.5036
Table 1: Single-style transfer comparison. Top-4 results are highlighted in red , blue , orange , and green , respectively.
Figure 6: Comparison with feed-forward methods. SuRe preserves content structure while transferring reference colors and textures.
Figure 7: Comparison with diffusion attention methods. SuRe suppresses semantic leakage while preserving style.
Figure 8: Comparison with diffusion adaptation methods. SuRe preserves the bathroom layout and transfers the painterly style.
Configuration
Style Similarity
Content Similarity
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
Baseline
24.700
14.763
1.903
0.5670
0.2755
0.4987
+ V-Shuffle
30.815
20.296
3.567
0.4470
0.1996
0.6787
+ PRM
21.714
13.833
1.621
0.4620
0.2049
0.6459
Table 2: Component ablation. Best results are highlighted in red .
m
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
Time ↓
1
24.644
15.797
1.735
0.4672
0.2146
0.6468
42.26s
3
22.415
14.301
1.689
0.4650
0.2146
0.6463
45.91s
5
21.714
13.833
1.621
0.4620
0.2049
0.6459
47.69s
7
21.544
13.832
1.616
0.4605
0.2044
0.6458
57.15s
9
21.543
13.834
1.626
0.4525
0.2045
0.6471
60.32s
Table 3: Effect of the number of permutation matrices m . Top-3 results are highlighted in red , blue , and orange , respectively.
Figure 9: Qualitative analysis of each component. V-Shuffle suppresses semantic leakage while preserving colors, and PRM restores stylistic textures. Red boxes indicate semantic leakage.
Figure 10: Ablation on the shuffle dimension. Shuffling along the l dimension suppresses semantic leakage while preserving style cues. Red dashed boxes indicate leaked semantics.
Figure 11: Ablation on different β and injection strategies.
Figure 12: Visual comparison of different injection strategies.
Figure 13: Qualitative comparison with different numbers of style images. Red dashed boxes highlight reduced semantic leakage.
Statistic
Definition
Max. error
Per-channel mean
∥Meanl(V~st)−Meanl(Vst)∥∞
3.58×10−7
Per-channel variance
∥Varl(V~st)−Varl(Vst)∥∞
0
Gram matrix
∥Vst⊤Vst∥F∥V~st⊤V~st−Vst⊤Vst∥F
1.72×10−7
Table 4: Numerical verification of empirical value-distribution preservation under V-Shuffle, using one representative content–style pair across 200 sampling steps and six selected attention layers (1,200 layer–timestep pairs). “Max. error” is the maximum over all evaluated timesteps, layers, attention heads, and channels.
Setting
Preserve empirical
Break K–V
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
distribution
correspondence
No shuffle
✓
×
24.700
14.763
1.903
0.5670
0.2755
0.4987
Matched Gaussian
×
✓
25.522
15.824
1.628
0.5170
0.2441
0.5664
Block shuffle ( 3×3 )
✓
✓†
23.544
14.858
1.613
0.4847
0.2435
0.5676
SuRe
✓
✓
21.714
13.833
1.621
0.4620
0.2049
0.6459
Table 5: Effect of value-token distribution preservation and K–V decoupling. Best results are highlighted in red bold .
Figure 14: Qualitative comparison of alternative value transformations. SuRe best preserves the content structure while transferring the reference colors and textures.
Figure 15: Qualitative effect of DDIM inversion. For each content–style pair, fixed-latent stylization without DDIM inversion better preserves the original content structure and follows the reference appearance. Dashed boxes highlight representative differences.
Style similarity
Content similarity
Setting
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
w/ DDIM inversion
26.826
15.336
2.048
0.6421
0.3211
0.3837
w/o DDIM inversion
21.714
13.833
1.621
0.4620
0.2049
0.6459
Table 6: Effect of DDIM inversion. Best results are highlighted in red .
Backbone
Resolution
T
β
LR
Layers
SD1.5 [ 23 ]
512×512
200
0.24
0.05
10–15
SDXL [ 21 ]
1024×1024
200
0.30
0.02
50–56
SD3 [ 8 ]
512×512
200
0.22
0.015
17–22
FLUX [ 14 ]
512×512
200
0.25
0.02
50–56
Table 7: Hyperparameters for SuRe across backbones.
Category
Method
Backbone
Memory ↓
Time ↓
Feed-forward methods
AdaIN [ 12 ]
CNN
2.09
0.58
MAST [ 7 ]
CNN
1.11
2.68
EFDM [ 31 ]
CNN
2.09
0.53
CAST [ 33 ]
CNN
33.09
6.04
AesPA-Net [ 11 ]
CNN
3.19
4.01
StyTR 2 [ 6 ]
Transformer
4.34
0.82
Table 8: Efficiency comparison across evaluated methods. Peak memory is reported in GB and inference time in seconds.
Figure 23
Figure 18: Mean user-study ratings across 20 blinded participants for content preservation, style fidelity, semantic leakage control, and overall quality. Ratings use a 1–5 Likert scale; higher scores indicate better performance.
Figure 19: Additional results of our approach on artistic style transfer.
Figure 20: Additional results of our approach on simulation to real.
Figure 21: Additional results of our approach on few-style transfer.
Reference-guided diffusion stylization aims to transfer visual style from a reference image while preserving the semantics specified by a text prompt. However, image conditioning often entangles transferable style cues with reference-specific content, leading to an inherent trade-off: stronger conditioning improves style fidelity but increases content leakage, whereas aggressive suppression reduces leakage at the cost of style expression. This challenge is further complicated by the distinct spatial organization of texture- and geometry-dominant styles. To address these issues, we propose SafeStyle, a training-free framework for calibrated style residual injection in frozen diffusion models. SafeStyle first estimates style-supported and content-associated subspaces from compact calibration sets, preserving their informative overlap while suppressing useless content variations. It then transports the purified style evidence over adaptive spatial granularity and constrains its effective influence through an explicit residual-norm budget. Experiments across texture- and geometry-dominant styles show that SafeStyle achieves a DINO style similarity of 0.432 while maintaining competitive text alignment. On a semantically disjoint leakage-stress benchmark, it further achieves a DINO style similarity of 0.474 with only 0.8% semantic leakage, demonstrating an effective balance between style fidelity and reference-content suppression.
Zhangping Yang, Min Li, Song Yan +4
Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China
Style transfer aims to render target content in the style of a reference image, but existing methods often suffer from content leakage, where objects, layouts, or semantics from the style reference appear in the generated output. Although prior data-driven and training-free methods can reduce leakage, they often face a leakage-degradation dilemma: stronger content suppression may weaken style fidelity, while richer style preservation may reintroduce unwanted reference content. We identify this dilemma across the full style-transfer pipeline, including feature separation, feature-space grounding, and diffusion generation. To address these issues, we propose CLeaR, a training-free framework for content-leakage-resistant style transfer. CLeaR first uses Orthogonal Subspace Projection to define content-reduced style targets in each vision foundation model (VFM) feature space. It then performs Ensemble Inversion, which optimizes a shared pixel-space style anchor satisfying style constraints across multiple VFMs. Finally, Energy-Guided Calibration maintains style alignment during diffusion sampling by steering the denoising trajectory toward the ensemble-defined style manifold. We further provide a theoretical analysis showing that the style-anchor estimation error decreases with the number of VFMs. Experiments on StyleBench demonstrate that CLeaR improves style alignment, reduces content leakage, and achieves better LLM-as-Judge evaluation compared with existing methods. The code is available at https://github.com/0606zt/CLeaR.
Reference-guided style transfer aims to preserve the semantic structure of a content image while transferring the visual appearance of a style reference. Recent diffusion-based methods achieve impressive stylization quality by exploiting strong pretrained generative priors. However, training-free approaches still face a difficult trade-off among style fidelity, content preservation, and content leakage. Direct style injection may unintentionally transfer semantic content from the style image, while fixed guidance schedules often ignore the time- and state-dependent nature of diffusion sampling. To address these limitations, we propose TRACE, a training-free diffusion style transfer framework with Time-adaptive Residual Attention Control and Content-Style Decomposition. TRACE first performs offline CLIP-based subspace analysis to separate content and style directions from paired data. During inference, it removes content-related components from the style reference and style-related components from the content reference to reduce leakage. It then injects style information through residual cross-attention and applies uncertainty-aware guidance to adapt the guidance signal at each denoising step. Experiments show that TRACE achieves a favorable trade-off between stylization and preservation. Compared with optimal-control-based baselines, TRACE substantially improves style fidelity (+17.28 CSD and +34.10 SRA). While, compared with stylization methods, it better preserves content structure (+12.80 DINO, +5.52 CLIP-I, and -8.19 LPIPS) and reduces directional semantic leakage by 29.5% in DCL. Our code is publicly available at https://github.com/pixelchemy-research/TRACE.
Duc Khoan Le, Kim Ngoc Tran, Minh Nhat Le +5
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam