Precise spatial control in diffusion-based style transfer remains challenging. This challenge arises because diffusion models treat style as a global feature and lack explicit spatial grounding of style representations, making it difficult to restrict style application to specific objects or regions. To our knowledge, existing diffusion models are unable to perform true localized style transfer, typically relying on handcrafted masks or multi-stage post-processing that introduce boundary artifacts and limit generalization. To address this, we propose an attention-supervised diffusion framework that explicitly teaches the model where to apply a given style by aligning the attention scores of style tokens with object masks during training. Two complementary objectives, a Focus loss based on KL divergence and a Cover loss using binary cross-entropy, jointly encourage accurate localization and dense coverage. A modular LoRA-MoE design further enables efficient and scalable multi-style adaptation. To evaluate localized stylization, we introduce the Regional Style Editing Score, which measures Regional Style Matching through CLIP-based similarity within the target region and Identity Preservation via masked LPIPS and pixel-level consistency on unedited areas. Experiments show that our method achieves mask-free, single-object style transfer at inference, producing regionally accurate and visually coherent results that outperform existing diffusion-based editing approaches.
Figures & tables
Figure 2: Overview of the proposed framework. The upper figure illustrates the overall pipeline based on the pretrained Flux.1-Kontext [ 29 ] . Given a context image, a noisy input, and a regional style prompt, image and text tokens are processed through Flux.1-Kontext, where LoRA-MoE modules [ 50 ] adapt attention and projection layers for style-specific learning. The model is optimized with flow matching, focus loss, and cover loss to reconstruct the target stylized image. Style-related attention maps are guided by binary masks, where focus loss concentrates attention within the target area and cover loss ensures spatial coverage for precise localized stylization.
Figure 3: Qualitative comparison of state-of-the-art instruction-based image editing methods.
Model
COCO
Pascal VOC
BIG
RSM ↑
LPIPSbg↓
MSEbg↓
RSM ↑
LPIPSbg↓
MSEbg↓
RSM ↑
LPIPSbg↓
MSEbg↓
Flux.1-Kontext [ 29 ]
0.6126 0.0180
0.4546 0.2960
0.1699 0.2254
0.6135 0.0170
0.4326 0.3135
0.1455 0.2258
0.6169 0.0228
0.4815 0.3125
0.1981 0.2408
Qwen-Image-Edit [ 49 ]
0.6235 0.0164
0.7530 0.1605
0.4398 0.3876
0.6333 0.0104
0.8078 0.1550
0.4477 0.3767
0.6329 0.0103
0.7185 0.1687
0.4284 0.3731
ICEdit [ 55 ]
0.6086 0.0152
0.3512 0.1979
0.1568 0.2800
0.6102 0.0147
0.3506 0.1963
0.1284 0.2304
0.6115 0.0164
0.4282 0.2315
0.1570 0.2697
AnyEdit [ 24 ]
0.6085 0.0104
0.6895 0.1564
0.2633 0.2913
0.6081 0.0107
0.6834 0.1715
0.2345 0.2454
0.6078 0.0097
0.7109 0.1679
0.2446 0.2813
Instruct-Pix2Pix [ 4 ]
0.5978 0.0115
0.1867 0.1995
0.0516 0.1587
0.5969 0.0111
0.1554 0.1472
0.0299 0.1078
0.5939 0.0111
0.1466 0.1377
0.0347 0.1403
Table 1: Quantitative comparison with baseline methods across three datasets. We report mean std for RSM (↑), LPIPS bg (↓), and MSE bg (↓). Darker cells indicate better ranks (1–3) within each dataset and metric.
Model
COCO
Pascal VOC
BIG
Q1 ↑
Q2 ↓
Q3 ↓
Q4 ↓
Q1 ↑
Q2 ↓
Q3 ↓
Q4 ↓
Q1 ↑
Q2 ↓
Q3 ↓
Q4 ↓
Flux.1-Kontext [ 29 ]
0.63
0.44
0.08
0.06
0.65
0.43
0.08
0.05
0.63
0.39
0.11
0.10
Qwen-Image-Edit [ 49 ]
0.98
0.86
0.01
0.00
0.97
0.78
0.02
0.01
0.98
0.90
0.05
0.04
ICEdit [ 55 ]
0.72
0.40
0.11
0.05
0.50
0.32
0.06
0.04
0.63
0.36
0.15
0.06
AnyEdit [ 24 ]
0.50
0.41
0.57
0.47
0.46
0.28
0.47
0.28
0.58
0.50
0.58
0.47
Instruct-Pix2Pix [ 4 ]
0.06
0.08
0.03
0.03
0.10
0.05
0.06
0.03
0.10
0.09
0.06
0.07
Table 2: Controllability and semantic reliability evaluation via Vision-Language Model (VLM). Q1: “Is the object in the target style ?”; Q2: “Is the background in the target style ?; Q3: “Is the object in the negative style ?”; Q4: “Is the background in the negative style ?” Each cell shows the probability (%) of a “Yes” response to the corresponding binary question.
Variant
Enabled Components / Blocks
COCO
Pascal VOC
BIG
Lcover
Lfocus
Double
Single
LoRA
RSM ↑
LPIPSbg↓
MSEbg↓
RSM ↑
LPIPSbg↓
MSEbg↓
RSM ↑
LPIPSbg↓
MSEbg↓
Ours
✓
✓
✓
✓
4
0.6128 0.0162
0.2103 0.1933
0.0729 0.1281
0.6147 0.0151
0.1331 0.1429
0.0269 0.0571
0.6159 0.0193
0.1593 0.1681
0.0474 0.0681
w/o Lcover
✗
✓
✓
✓
4
0.6120 0.0163
0.2174 0.1889
0.0730 0.1069
0.6142 0.0155
0.1383 0.1530
0.0359 0.0580
0.6152 0.0193
0.1605 0.1680
0.0488 0.0681
w/o Lfocus
✓
✗
✓
✓
4
0.6127 0.0159
0.2132 0.1852
0.0740 0.1117
0.6147 0.0153
0.1359 0.1393
0.0325 0.0497
0.6158 0.0190
0.1612 0.1620
0.0542 0.0917
w/o Double
✓
✓
✗
✓
4
0.6168 0.0168
0.4225 0.2838
0.1409 0.1831
0.6175 0.0155
0.3887 0.3064
0.0980 0.1435
0.6195 0.0198
0.3843 0.3086
0.1275 0.1458
w/o Single
✓
✓
✓
✗
4
0.6190 0.0163
0.5203 0.3060
0.2284 0.2216
0.6185 0.0158
0.4252 0.3303
0.1390 0.1884
0.6192 0.0205
0.3933 0.3110
0.1402 0.1595
Table 3: Ablation of key loss components, network streams, and LoRA ranks on three datasets. We report mean std for RSM (↑), LPIPS bg (↓), and MSE bg (↓). ✓ denotes the component is enabled.
Figure 4: Visualization of attention maps under different loss configurations. Using only Lcover or Lfocus causes attention spillover to nearby area, whereas our full objective focuses on the motorcycle without leakage to surrounding regions, demonstrating its ability to maintain precise and consistent attention.
Figure 5: Illustration of the pseudo ground-truth (pseudo-GT) generation process. A diffusion-based style transfer model generates a fully stylized version of the image according to a given style prompt. The stylized region corresponding to the mask is then blended back into the original image using seamless cloning, producing an aligned input–target pair for localized style learning.
Figure 6: Examples from our pseudo ground truth dataset for localized style transfer. Each example consists of four elements: the original context image, the corresponding binary mask that specifies the target object, the pseudo ground truth image generated by applying a global style transformation to the entire image and compositing the stylized region back onto the original image, and the training prompt used for fine-tuning.
Figure 7: Additional examples of localized style transfer on objects that also appear in the training data. The model produces clean, structurally consistent stylization within the target regions, showing stable behavior on in-distribution categories.
Figure 8: The first four columns show results on object categories that never appear in the training set. The last two show categories with only synonym-level presence (motorbike → motorcycle, aeroplane → airplane). The model successfully performs localized style transfer even without direct supervision for these object types, indicating strong semantic generalization.
Figure 9: We evaluate the impact of removing single-stream LoRA blocks, double-stream LoRA blocks, the cover loss Lcover , and the focus loss Lfocus . The full RegionRoute model achieves the clearest and most accurate localized style transfer, while ablated variants exhibit style leakage, incomplete stylization, or degraded unedited object consistency.
Figure 10: Comparison of the style-token attention maps for RegionRoute (ours) and the Flux.1-Kontext baseline. For each task, we show the edited output, the averaged style-token attention map, and the step-wise evolution of style-token attention at diffusion steps 5, 10, 15, 20, and 25.
Figure 11: Examples illustrating several types of failure: (1) incomplete stylization when multiple target objects are present, where only the most salient instances are stylized; (2) failure to stylize very small or heavily occluded objects; (3) unintended stylization of regions adjacent to the target object, especially when objects are physically connected; (4) stylization leakage onto objects that are contained within the target object, such as a person inside a car or pillows on a sofa; and (5) partial stylization of a single object in cases where some object parts are difficult to recognize.
Style transfer applies the appearance of a reference image to a content image while preserving its spatial structure. Recent diffusion-based methods achieve strong stylization but typically assume a single global style. We instead consider regional-multi style transfer, which assigns multiple references to user-specified regions of a content image. Extending them to this setting reveals two coupled shared-attention issues: ambiguous mass allocation among content and style partitions and degraded selectivity as more styles are jointly normalized, while style aggregation at the attention output further suppresses fine details. We propose MAST (Mask-Guided Attention Control for Training-Free Regional-Multi Style Transfer), a unified attention-control framework for frozen diffusion models. Logit-level Attention Mass Allocation enforces mask-derived partition masses, Sharpness-aware Temperature Scaling adaptively restores selectivity, and Discrepancy-aware Detail Injection recovers high-frequency content. MAST jointly processes all style--mask pairs in a single denoising pass without training, optimization, or post-hoc composition. Across two to five styles, MAST achieves the best average ArtFID, FID, and R-FID among all baselines, demonstrating regional style fidelity, content preservation, and scalability.
Dongkyung Kang, Jaeyeon Hwang, Junseo Park +6
Department of Computer Science & Artificial Intelligence, Dongguk University
Transferring visual style between images while preserving semantic correspondence between similar objects remains a central challenge in computer vision. While existing methods have made great strides, most of them operate at global level but overlook region-wise and even pixel-wise semantic correspondence. To address this, we propose CoCoDiff, a novel training-free and low-cost style transfer framework that leverages pretrained latent diffusion models to achieve fine-grained, semantically consistent stylization. We identify that correspondence cues within generative diffusion models are under-explored and that content consistency across semantically matched regions is often neglected. CoCoDiff introduces a pixel-wise semantic correspondence module that mines intermediate diffusion features to construct a dense alignment map between content and style images. Furthermore, a cycle-consistency module then enforces structural and perceptual alignment across iterations, yielding object and region level stylization that preserves geometry and detail. Despite requiring no additional training or supervision, CoCoDiff delivers state-of-the-art visual quality and strong quantitative results, outperforming methods that rely on extra training or annotations.
Wenbo Nie, Zixiang Li, Renshuai Tao +3
Institute of Information Science, Beijing Jiaotong University · Visual Intelligence + X International Joint Laboratory of the Ministry of Education
Image stylization with latent-diffusion models entangles two independently refined axes: what a region depicts and how it is depicted. Such pipelines expose only global controls, yet professional retouching demands deliberate, region-specific control. We lift two conditioning weights already present in a ControlNet + IP-Adapter stylization pipeline from global scalars to per-location spatial maps, yielding local, per-axis control of content and style in a single generative pass. Because the two weights act on disjoint pathways, adjusting them independently spans a 2x2 retouching vocabulary, from free regeneration to identity preservation. We validate that edits stay confined to the retouched region and that each weight predominantly steers its own axis. Our approach requires no retraining and drops unchanged into any such pipeline.