Transfer the Style, Not the Semantics: Mitigating Semantic Leakage in Image-Guided Stylization
Authors: Haojun Tang, Qiwei Lin, Tongda Xu, Wenda Zhao, Yian Zhao, Yu Zhang, Lida Huang, Yan Wang
Organizations: Institute for AI Industry Research (AIR), Tsinghua University · School of Information and Communication Engineering, Dalian University of Technology · Beijing Institute of Radio Measurement · School of Information Engineering, Peking University
Image-guided zero-shot style transfer enables training-free stylization of arbitrary content images using a reference style. Despite recent progress, existing methods commonly rely on self-attention manipulation to inject style information, which may transfer semantic content from the style image, leading to semantic leakage. In this work, we revisit this phenomenon from the perspective of key--value coupling in the self-attention of diffusion models. Our analysis shows that the native correspondence between style keys and values plays an important role in transferring semantic information from the style image. Motivated by this observation, we propose SuRe, a timestep-aware key--value regulation framework that mitigates semantic leakage by disrupting the native key--value correspondence while progressively restoring it for faithful style transfer. Specifically, we present value shuffle to decouple style keys and values while preserving the empirical distribution of the values, together with a progressive recoupling mechanism that gradually recouples them throughout the stylization process. Experiments against 18 representative methods demonstrate that SuRe maintains competitive style similarity while consistently improving content similarity across SD1.5, SDXL, SD3, and FLUX.1-dev, with natural generalization to multi-style transfer. The code will be publicly available at https://xinr-tang.github.io/SuRe-homepage/.
Figures & tables
Figure 2: Qualitative comparison of different key–value configurations. Preserving the K–V pairing leads to semantic leakage, while value-only reordering suppresses leaked semantics.
Figure 3: Linear probing of different key–value configurations. Preserving the native K–V pairing retains the semantics of the style image, while value-only reordering reduces it.
Figure 4: Overview of SuRe. The content and style images are first encoded into image latents. V-Shuffle permutes the style value tokens while keeping the style keys fixed, breaking their native correspondence while preserving the empirical value distribution. PRM then progressively combines shuffled and native attention during sampling, suppressing style semantics while refining the target style.
Figure 5: PCA visualization of Vst extracted from the same zs0 at different timesteps. Coarse semantic layouts emerge in the early stage (first 30%) and become clearer in the intermediate stage (middle 40%), while high-frequency stylistic patterns gradually emerge in the late stage (last 30%). We visualize Vst from the 12th self-attention layer of SD1.5, the 45th of SDXL, the 19th of SD3, and the 51st of FLUX.
Category
Method
Paradigm
Style Similarity
Content Similarity
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
Feed-forward methods
AesPA-Net (ICCV’23) [ 11 ]
CNN
31.426
19.763
2.177
0.5135
0.2465
0.5119
EFDM (CVPR’22) [ 31 ]
CNN
30.146
17.344
3.061
0.6434
0.3134
0.4502
CAST (SIGGRAPH’22) [ 33 ]
CNN
34.685
20.395
2.234
0.6212
0.2918
0.5587
StyTR 2 (CVPR’22) [ 6 ]
Transformer
29.887
18.293
1.700
0.5491
0.2852
0.6098
AdaAttN (ICCV’21) [ 17 ]
Transformer
30.933
18.242
2.356
0.6076
0.3155
0.5036
Table 1: Single-style transfer comparison. Top-4 results are highlighted in red , blue , orange , and green , respectively.
Figure 6: Comparison with feed-forward methods. SuRe preserves content structure while transferring reference colors and textures.
Figure 7: Comparison with diffusion attention methods. SuRe suppresses semantic leakage while preserving style.
Figure 8: Comparison with diffusion adaptation methods. SuRe preserves the bathroom layout and transfers the painterly style.
Configuration
Style Similarity
Content Similarity
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
Baseline
24.700
14.763
1.903
0.5670
0.2755
0.4987
+ V-Shuffle
30.815
20.296
3.567
0.4470
0.1996
0.6787
+ PRM
21.714
13.833
1.621
0.4620
0.2049
0.6459
Table 2: Component ablation. Best results are highlighted in red .
m
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
Time ↓
1
24.644
15.797
1.735
0.4672
0.2146
0.6468
42.26s
3
22.415
14.301
1.689
0.4650
0.2146
0.6463
45.91s
5
21.714
13.833
1.621
0.4620
0.2049
0.6459
47.69s
7
21.544
13.832
1.616
0.4605
0.2044
0.6458
57.15s
9
21.543
13.834
1.626
0.4525
0.2045
0.6471
60.32s
Table 3: Effect of the number of permutation matrices m . Top-3 results are highlighted in red , blue , and orange , respectively.
Figure 9: Qualitative analysis of each component. V-Shuffle suppresses semantic leakage while preserving colors, and PRM restores stylistic textures. Red boxes indicate semantic leakage.
Figure 10: Ablation on the shuffle dimension. Shuffling along the l dimension suppresses semantic leakage while preserving style cues. Red dashed boxes indicate leaked semantics.
Figure 11: Ablation on different β and injection strategies.
Figure 12: Visual comparison of different injection strategies.
Figure 13: Qualitative comparison with different numbers of style images. Red dashed boxes highlight reduced semantic leakage.
Statistic
Definition
Max. error
Per-channel mean
∥Meanl(V~st)−Meanl(Vst)∥∞
3.58×10−7
Per-channel variance
∥Varl(V~st)−Varl(Vst)∥∞
0
Gram matrix
∥Vst⊤Vst∥F∥V~st⊤V~st−Vst⊤Vst∥F
1.72×10−7
Table 4: Numerical verification of empirical value-distribution preservation under V-Shuffle, using one representative content–style pair across 200 sampling steps and six selected attention layers (1,200 layer–timestep pairs). “Max. error” is the maximum over all evaluated timesteps, layers, attention heads, and channels.
Setting
Preserve empirical
Break K–V
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
distribution
correspondence
No shuffle
✓
×
24.700
14.763
1.903
0.5670
0.2755
0.4987
Matched Gaussian
×
✓
25.522
15.824
1.628
0.5170
0.2441
0.5664
Block shuffle ( 3×3 )
✓
✓†
23.544
14.858
1.613
0.4847
0.2435
0.5676
SuRe
✓
✓
21.714
13.833
1.621
0.4620
0.2049
0.6459
Table 5: Effect of value-token distribution preservation and K–V decoupling. Best results are highlighted in red bold .
Figure 14: Qualitative comparison of alternative value transformations. SuRe best preserves the content structure while transferring the reference colors and textures.
Figure 15: Qualitative effect of DDIM inversion. For each content–style pair, fixed-latent stylization without DDIM inversion better preserves the original content structure and follows the reference appearance. Dashed boxes highlight representative differences.
Style similarity
Content similarity
Setting
ArtFID ↓
FID ↓
Gram ↓
LPIPS ↓
CFSD ↓
DINOv2 ↑
w/ DDIM inversion
26.826
15.336
2.048
0.6421
0.3211
0.3837
w/o DDIM inversion
21.714
13.833
1.621
0.4620
0.2049
0.6459
Table 6: Effect of DDIM inversion. Best results are highlighted in red .
Backbone
Resolution
T
β
LR
Layers
SD1.5 [ 23 ]
512×512
200
0.24
0.05
10–15
SDXL [ 21 ]
1024×1024
200
0.30
0.02
50–56
SD3 [ 8 ]
512×512
200
0.22
0.015
17–22
FLUX [ 14 ]
512×512
200
0.25
0.02
50–56
Table 7: Hyperparameters for SuRe across backbones.
Category
Method
Backbone
Memory ↓
Time ↓
Feed-forward methods
AdaIN [ 12 ]
CNN
2.09
0.58
MAST [ 7 ]
CNN
1.11
2.68
EFDM [ 31 ]
CNN
2.09
0.53
CAST [ 33 ]
CNN
33.09
6.04
AesPA-Net [ 11 ]
CNN
3.19
4.01
StyTR 2 [ 6 ]
Transformer
4.34
0.82
Table 8: Efficiency comparison across evaluated methods. Peak memory is reported in GB and inference time in seconds.
Figure 23
Figure 18: Mean user-study ratings across 20 blinded participants for content preservation, style fidelity, semantic leakage control, and overall quality. Ratings use a 1–5 Likert scale; higher scores indicate better performance.
Figure 19: Additional results of our approach on artistic style transfer.
Figure 20: Additional results of our approach on simulation to real.
Figure 21: Additional results of our approach on few-style transfer.
Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China