Despite the advancements in diffusion-based image style transfer, existing methods are commonly limited by 1) semantic gap: the style reference could miss proper content semantics, causing uncontrollable stylization; 2) reliance on extra constraints (e.g., semantic masks) restricting applicability; 3) rigid feature associations lacking adaptive global-local alignment, failing to balance fine-grained stylization and global content preservation. These limitations, particularly the inability to flexibly leverage style inputs, fundamentally restrict style transfer in terms of personalization, accuracy, and adaptability. To address these, we propose StyleGallery, a training-free and semantic-aware framework that supports arbitrary reference images as input and enables effective personalized customization. It comprises three core stages: semantic region segmentation (adaptive clustering on latent diffusion features to divide regions without extra inputs); clustered region matching (block filtering on extracted features for precise alignment); and style transfer optimization (energy function-guided diffusion sampling with regional style loss to optimize stylization). Experiments on our introduced benchmark demonstrate that StyleGallery outperforms state-of-the-art methods in content structure preservation, regional stylization, interpretability, and personalized customization, particularly when leveraging multiple style references.
Figures & tables
Figure 2: Overall framework. Our pipeline comprises three stages: (a) In stage 1, the content image is diffused for T steps to extract UNet features F0,…,FT , which are weighted into Fmix , then clustered via PCA and K-means to optimize the mask. (b) In stage 2, Fmix and the semantic mask identify cluster features, which are aggregated via self-attention for statistical similarity. Meanwhile, DINOv2 [ 32 ] splits x0 into blocks; tokens are filtered for semantic similarity, while cluster positions yield positional similarity. (c) Stage 3 optimizes the generation through N latent sampling steps. UNet attention maps are extracted, sparsified using the semantic mask, and recombined with style features ( Ks , Vs ). L1 distances are then computed between the actual and combined feature maps, as well as the Q and Qc . These losses guide the final image generation.
Figure 3: Cluster optimization. We compute pairwise semantic distances among clusters, merge those below a threshold (set to 0.85), then split–recombine clusters guided by the input’s depth features, then traverse each pixel, eliminate isolated points.
Figure 4: Sparse attention based on semantic matches. We mask the attention feature map, preserving weights for relevant semantics and zeroing out irrelevant regions. Content-sparse attention matches corresponding style features, yielding optimal stylization. Semantically identical regions are color-coded; connections use matching colors.
Figure 5: Qualitative comparisons with recent state-of-the-art image style transfer methods. Our StyleGallery achieves semantic-level style transfer for fine-grained content (e.g., hair and clothing in the first row; sea and sailboat in the second row) while preserving content consistency, and others either fail to transfer styles accurately or compromise content structure. Regarding multiple references, our method can accurately and adaptively recognize, match, and transfer regional styles (the third row) without introducing incorrect semantic content seen in others (e.g., unexpected semantics in the sky of other methods in the fourth row). Zoom-in is recommended for better comparisons.
Metric
CAST [ 49 ]
CCPL [ 44 ]
AdaAttn [ 29 ]
StyTR-2 [ 6 ]
CSGO [ 45 ]
StyleShot [ 8 ]
StyleID [ 4 ]
AD [ 53 ]
AttnST [ 16 ]
Ours
Style ↑
0.4959
0.5163
0.5094
0.5219
0.5224
0.5198
0.4972
0.5249
0.5032
0.5337
Gram Loss ↓
15.778
17.8606
17.357
16.719
19.937
19.013
14.261
13.862
16.937
13.519
FID [ 15 ] ↓
17.255
18.141
18.498
17.623
19.829
20.638
18.987
17.677
19.233
16.889
LPIPS [ 48 ] ↓
0.4934
0.4549
0.5299
0.3856
0.5005
0.6615
0.4496
0.4032
0.4532
0.3716
ArtFID [ 43 ] ↓
27.262
27.843
29.831
25.804
31.254
35.952
28.973
26.207
29.402
24.536
Table 1: Quantitative comparison with image style transfer methods. The best results are shown in bold , and the second-best results are indicated with an underline .
Figure 6: Zoomed details of our method, AD [ 53 ] and StyleID [ 4 ] . Black boxes highlight enlarged views of the results.AD [ 53 ] under-stylizes the image, with the background remaining unchanged, while StyleID [ 4 ] introduces incorrect patterns. This underscores the need for our semantic-aware regional style transfer.
Figure 8: Ablation study on loss functions. The global content loss preserves semantic consistency while the region style loss enables effective transfer, with their combination yielding faithful stylization; additionally, confining style transfer to semantically matched region pairs further enhances quality and mitigates semantic leakage in the generated images.
Configuration
LPIPS [ 48 ] ↓
FID [ 15 ] ↓
Style ↑
Ours ( λc=0.26 , default)
0.3716
16.89
0.5337
Ours ( λc=0.29 , above)
0.3689
27.26
0.4172
Ours ( λc=0.22 , below)
0.4354
21.84
0.4562
w/o RSL
0.5195
23.56
0.4387
w/o GCL
0.6822
30.83
0.4150
Table 2: Ablation study on our proposed components.
Figure 9: Visual impact of the content loss weight λc . As the global content loss rises from left to right, generated results better preserve the content image’s structure while keeping stylization. In automatic mode, we set 0.26 as the default; in custom mode, this weight can be selected by the user.
Figure 10: Compatibility analysis. We integrated with accelerated models reduces optimization steps from 150 to 28. The reduced inference times (highlighted in green) demonstrate a significant speedup while maintaining high-fidelity stylization quality.
Figure 11: Results of robustness analysis.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Overall composition of the style dataset.
Figure 2: Category distribution in style dataset.
Figure 3: Ablations on cluster merge threshold.
Threshold
0.55
0.65
0.75
0.85
0.95
Accuracy ≈ ( % ) ↑
52
65
77
93
90
Appendix
Table 1: Ablation study: Effect of merge threshold on semantic classification accuracy. The best performance (93%) is achieved at threshold 0.85.
Metric
λ1 =0.05
λ1 =0.25
λ1 =0.85
only λ2
λ2+λ1
λ2+λ1+λ3
LPIPS [ 48 ] ↓
0.2345
0.2317
0.2323
0.2413
0.2321
0.2317
Accuracy ≈ ( % ) ↑
86
89
87
87
89
89
Appendix
Table 2: Ablation study on similarity weight. It shows that values of λ1 (low or high) may yield erroneous semantic correspondences, and indicate that relying solely on semantic features extracted by DINOv2 [ 32 ] can lead to mismatches, whereas incorporating statistical features further improves matching accuracy.
Figure 4: Ablations on optimization steps.
Metric
StyleID
AD
Ours
Visual
3.76
3.77
4.43
Region
3.44
3.75
4.41
Content
4.35
3.73
4.46
Appendix
Table 3: User preference scores across three aspects.
Transferring visual style between images while preserving semantic correspondence between similar objects remains a central challenge in computer vision. While existing methods have made great strides, most of them operate at global level but overlook region-wise and even pixel-wise semantic correspondence. To address this, we propose CoCoDiff, a novel training-free and low-cost style transfer framework that leverages pretrained latent diffusion models to achieve fine-grained, semantically consistent stylization. We identify that correspondence cues within generative diffusion models are under-explored and that content consistency across semantically matched regions is often neglected. CoCoDiff introduces a pixel-wise semantic correspondence module that mines intermediate diffusion features to construct a dense alignment map between content and style images. Furthermore, a cycle-consistency module then enforces structural and perceptual alignment across iterations, yielding object and region level stylization that preserves geometry and detail. Despite requiring no additional training or supervision, CoCoDiff delivers state-of-the-art visual quality and strong quantitative results, outperforming methods that rely on extra training or annotations.
Wenbo Nie, Zixiang Li, Renshuai Tao +3
Institute of Information Science, Beijing Jiaotong University · Visual Intelligence + X International Joint Laboratory of the Ministry of Education
Reference-guided diffusion stylization aims to transfer visual style from a reference image while preserving the semantics specified by a text prompt. However, image conditioning often entangles transferable style cues with reference-specific content, leading to an inherent trade-off: stronger conditioning improves style fidelity but increases content leakage, whereas aggressive suppression reduces leakage at the cost of style expression. This challenge is further complicated by the distinct spatial organization of texture- and geometry-dominant styles. To address these issues, we propose SafeStyle, a training-free framework for calibrated style residual injection in frozen diffusion models. SafeStyle first estimates style-supported and content-associated subspaces from compact calibration sets, preserving their informative overlap while suppressing useless content variations. It then transports the purified style evidence over adaptive spatial granularity and constrains its effective influence through an explicit residual-norm budget. Experiments across texture- and geometry-dominant styles show that SafeStyle achieves a DINO style similarity of 0.432 while maintaining competitive text alignment. On a semantically disjoint leakage-stress benchmark, it further achieves a DINO style similarity of 0.474 with only 0.8% semantic leakage, demonstrating an effective balance between style fidelity and reference-content suppression.
Zhangping Yang, Min Li, Song Yan +4
Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China
Reference-guided style transfer aims to preserve the semantic structure of a content image while transferring the visual appearance of a style reference. Recent diffusion-based methods achieve impressive stylization quality by exploiting strong pretrained generative priors. However, training-free approaches still face a difficult trade-off among style fidelity, content preservation, and content leakage. Direct style injection may unintentionally transfer semantic content from the style image, while fixed guidance schedules often ignore the time- and state-dependent nature of diffusion sampling. To address these limitations, we propose TRACE, a training-free diffusion style transfer framework with Time-adaptive Residual Attention Control and Content-Style Decomposition. TRACE first performs offline CLIP-based subspace analysis to separate content and style directions from paired data. During inference, it removes content-related components from the style reference and style-related components from the content reference to reduce leakage. It then injects style information through residual cross-attention and applies uncertainty-aware guidance to adapt the guidance signal at each denoising step. Experiments show that TRACE achieves a favorable trade-off between stylization and preservation. Compared with optimal-control-based baselines, TRACE substantially improves style fidelity (+17.28 CSD and +34.10 SRA). While, compared with stylization methods, it better preserves content structure (+12.80 DINO, +5.52 CLIP-I, and -8.19 LPIPS) and reduces directional semantic leakage by 29.5% in DCL. Our code is publicly available at https://github.com/pixelchemy-research/TRACE.
Duc Khoan Le, Kim Ngoc Tran, Minh Nhat Le +5
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam