cs.CVNov 9, 2025

Transfer the Style, Not the Semantics: Mitigating Semantic Leakage in Image-Guided Stylization

Authors: Haojun Tang, Qiwei Lin, Tongda Xu, Wenda Zhao, Yian Zhao, Yu Zhang, Lida Huang, Yan Wang

Organizations: Institute for AI Industry Research (AIR), Tsinghua University · School of Information and Communication Engineering, Dalian University of Technology · Beijing Institute of Radio Measurement · School of Information Engineering, Peking University

Abstract

Image-guided zero-shot style transfer enables training-free stylization of arbitrary content images using a reference style. Despite recent progress, existing methods commonly rely on self-attention manipulation to inject style information, which may transfer semantic content from the style image, leading to semantic leakage. In this work, we revisit this phenomenon from the perspective of key--value coupling in the self-attention of diffusion models. Our analysis shows that the native correspondence between style keys and values plays an important role in transferring semantic information from the style image. Motivated by this observation, we propose SuRe, a timestep-aware key--value regulation framework that mitigates semantic leakage by disrupting the native key--value correspondence while progressively restoring it for faithful style transfer. Specifically, we present value shuffle to decouple style keys and values while preserving the empirical distribution of the values, together with a progressive recoupling mechanism that gradually recouples them throughout the stylization process. Experiments against 18 representative methods demonstrate that SuRe maintains competitive style similarity while consistently improving content similarity across SD1.5, SDXL, SD3, and FLUX.1-dev, with natural generalization to multi-style transfer. The code will be publicly available at https://xinr-tang.github.io/SuRe-homepage/.

Figures & tables

Explore similar work

CardsList
  1. SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

    Sep 18, 2026Zhangping Yang, Min Li, Song Yan +4Image Style TransferDiffusion-Based Image Editing

  2. CLeaR: A Unified Framework for Resolving the Leakage-Degradation Dilemma in Style Transfer

    Sep 29, 2026Teng Zhou, Yunhao ChenImage Style Transfer

  3. TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer

    Oct 4, 2026Duc Khoan Le, Kim Ngoc Tran, Minh Nhat Le +5Image Style TransferDiffusion Model Guidance