CoCoDiff: Correspondence-Consistent Diffusion Model for Fine-grained Style Transfer
Authors: Wenbo Nie, Zixiang Li, Renshuai Tao, Bin Wu, Yunchao Wei, Yao Zhao
Organizations: Institute of Information Science, Beijing Jiaotong University · Visual Intelligence + X International Joint Laboratory of the Ministry of Education
Transferring visual style between images while preserving semantic correspondence between similar objects remains a central challenge in computer vision. While existing methods have made great strides, most of them operate at global level but overlook region-wise and even pixel-wise semantic correspondence. To address this, we propose CoCoDiff, a novel training-free and low-cost style transfer framework that leverages pretrained latent diffusion models to achieve fine-grained, semantically consistent stylization. We identify that correspondence cues within generative diffusion models are under-explored and that content consistency across semantically matched regions is often neglected. CoCoDiff introduces a pixel-wise semantic correspondence module that mines intermediate diffusion features to construct a dense alignment map between content and style images. Furthermore, a cycle-consistency module then enforces structural and perceptual alignment across iterations, yielding object and region level stylization that preserves geometry and detail. Despite requiring no additional training or supervision, CoCoDiff delivers state-of-the-art visual quality and strong quantitative results, outperforming methods that rely on extra training or annotations.
Figures & tables
Figure 1: Comparison of different style transfer methods. We compare our proposed CoCoDiff with three other representative methods with zoomed-in details.
Figure 2: Framework Overview of our proposed Correspondence-Consistent Diffusion (CoCoDiff).
Figure 3: Qualitative comparison. We compare CoCoDiff (Ours) with seven representative methods, selected from diffusion-based, patch-based, CNN-based, transformer-based, and other approaches, to provide a comprehensive evaluation.
Figure 4: (A) Qualitative comparison with additional zoomed-in details . We compare our method with StyleID and StyTR 2 as baseline approaches, highlighting the differences through zoomed-in details. (B) The illustration of cycle-based image style transfer. (a) Direct feature matching between the style image and the content image often results in low matching accuracy and feature correspondence failure. (b) By first transforming the style image to adopt the content image’s style before performing feature matching, the matching accuracy is significantly improved. (c) Direct correspondence result. (d) Indirect correspondence result.
Metric
Ours
StyleID
AesPA
StyTR 2
InST
Instant+
AdaIN
SCSA
FreeStyle
SMS
FID ↓
18.432
21.010
19.645
18.886
21.541
20.982
18.672
20.835
23.654
31.266
LPIPS ↓
0.549
0.565
0.556
0.587
0.785
0.584
0.612
0.562
0.689
0.821
ArtFID ↓
30.100
34.446
32.124
31.559
40.235
34.820
31.711
34.106
41.640
58.756
CFSD ↓
0.609
0.619
0.632
0.687
0.881
0.710
0.642
0.612
0.660
0.704
Table 1: Quantitative evaluation results. We compare every methods across multiple metrics. Columns 2nd - 9th are reference-guided methods while columns 10th - 11th are prompt-based methods.
Table 6
Figure 5: Quantitative comparison of the cycle module. (a) Balance between LPIPS and FID metrics across iterations. (b) CFSD variations across iterations.
Figure 6: Visual results of the iteration process. The group with the best quality is highlighted by a red line.
Table 4: Ablation study analyzing the impact of various design choices on model performance.
Method
Style
Content
Avg
StyleID Chung et al. (2024)
0.212
0.135
0.174
AesPA Hong et al. (2023)
0.033
0.134
0.084
StyTR 2 Deng et al. (2022)
0.132
0.036
0.084
InST Zhang et al. (2023b)
0.004
0.025
0.015
Instant+ Wang et al. (2024a)
0.002
0.174
0.088
AdaIN Huang and Belongie (2017)
0.038
0.013
0.026
Table 5: User study results in consistency and quality.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Additional classic visual outcomes in diverse artistic styles.
Figure 8: Visualization results of our proposed methods.
Figure 9: Comparison of style transfer results. Visual results of our CoCoDiff method compared against four baseline approaches across diverse artistic styles: Ghibli Style, Pixel Art, Chinese Ink, Kid Illustration, and Vintage Style.
Metric
Ours
CNNMRF
Style-Swap
DIA
Avatar-Net
SCSA+StyleID
FID (↓)
18.432
27.872
35.642
31.933
22.356
20.835
LPIPS (↓)
0.549
0.672
0.793
0.661
0.641
0.562
CFSD (↓)
0.609
0.844
0.761
0.649
0.753
0.612
Appendix
Table 6: Comparison of metrics across methods.
Figure 10: Additional comparisons with patch-based methods.
Figure 11: Fine-grained style transfer visual results: Flower.
Figure 12: Fine-grained style transfer visual results: Cow.
Figure 13: Fine-grained style transfer visual results: House.
Precise spatial control in diffusion-based style transfer remains challenging. This challenge arises because diffusion models treat style as a global feature and lack explicit spatial grounding of style representations, making it difficult to restrict style application to specific objects or regions. To our knowledge, existing diffusion models are unable to perform true localized style transfer, typically relying on handcrafted masks or multi-stage post-processing that introduce boundary artifacts and limit generalization. To address this, we propose an attention-supervised diffusion framework that explicitly teaches the model where to apply a given style by aligning the attention scores of style tokens with object masks during training. Two complementary objectives, a Focus loss based on KL divergence and a Cover loss using binary cross-entropy, jointly encourage accurate localization and dense coverage. A modular LoRA-MoE design further enables efficient and scalable multi-style adaptation. To evaluate localized stylization, we introduce the Regional Style Editing Score, which measures Regional Style Matching through CLIP-based similarity within the target region and Identity Preservation via masked LPIPS and pixel-level consistency on unedited areas. Experiments show that our method achieves mask-free, single-object style transfer at inference, producing regionally accurate and visually coherent results that outperform existing diffusion-based editing approaches.
Bowen Chen, Jake Zuena, Alan C. Bovik +1
The University of Texas at Austin · Dolby Laboratories, Inc.
Despite the advancements in diffusion-based image style transfer, existing methods are commonly limited by 1) semantic gap: the style reference could miss proper content semantics, causing uncontrollable stylization; 2) reliance on extra constraints (e.g., semantic masks) restricting applicability; 3) rigid feature associations lacking adaptive global-local alignment, failing to balance fine-grained stylization and global content preservation. These limitations, particularly the inability to flexibly leverage style inputs, fundamentally restrict style transfer in terms of personalization, accuracy, and adaptability. To address these, we propose StyleGallery, a training-free and semantic-aware framework that supports arbitrary reference images as input and enables effective personalized customization. It comprises three core stages: semantic region segmentation (adaptive clustering on latent diffusion features to divide regions without extra inputs); clustered region matching (block filtering on extracted features for precise alignment); and style transfer optimization (energy function-guided diffusion sampling with regional style loss to optimize stylization). Experiments on our introduced benchmark demonstrate that StyleGallery outperforms state-of-the-art methods in content structure preservation, regional stylization, interpretability, and personalized customization, particularly when leveraging multiple style references.
Boyu He, Yunfan Ye, Chang Liu +3
College of Computer Science and Technology, National University of Defense Technology · School of Design, Hunan University
Reference-guided diffusion stylization aims to transfer visual style from a reference image while preserving the semantics specified by a text prompt. However, image conditioning often entangles transferable style cues with reference-specific content, leading to an inherent trade-off: stronger conditioning improves style fidelity but increases content leakage, whereas aggressive suppression reduces leakage at the cost of style expression. This challenge is further complicated by the distinct spatial organization of texture- and geometry-dominant styles. To address these issues, we propose SafeStyle, a training-free framework for calibrated style residual injection in frozen diffusion models. SafeStyle first estimates style-supported and content-associated subspaces from compact calibration sets, preserving their informative overlap while suppressing useless content variations. It then transports the purified style evidence over adaptive spatial granularity and constrains its effective influence through an explicit residual-norm budget. Experiments across texture- and geometry-dominant styles show that SafeStyle achieves a DINO style similarity of 0.432 while maintaining competitive text alignment. On a semantically disjoint leakage-stress benchmark, it further achieves a DINO style similarity of 0.474 with only 0.8% semantic leakage, demonstrating an effective balance between style fidelity and reference-content suppression.
Zhangping Yang, Min Li, Song Yan +4
Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China