cs.CVFeb 22, 2026

RegionRoute: Regional Style Transfer with Diffusion Model

Authors: Bowen Chen, Jake Zuena, Alan C. Bovik, Divya Kothandaraman

Organizations: The University of Texas at Austin · Dolby Laboratories, Inc.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Precise spatial control in diffusion-based style transfer remains challenging. This challenge arises because diffusion models treat style as a global feature and lack explicit spatial grounding of style representations, making it difficult to restrict style application to specific objects or regions. To our knowledge, existing diffusion models are unable to perform true localized style transfer, typically relying on handcrafted masks or multi-stage post-processing that introduce boundary artifacts and limit generalization. To address this, we propose an attention-supervised diffusion framework that explicitly teaches the model where to apply a given style by aligning the attention scores of style tokens with object masks during training. Two complementary objectives, a Focus loss based on KL divergence and a Cover loss using binary cross-entropy, jointly encourage accurate localization and dense coverage. A modular LoRA-MoE design further enables efficient and scalable multi-style adaptation. To evaluate localized stylization, we introduce the Regional Style Editing Score, which measures Regional Style Matching through CLIP-based similarity within the target region and Identity Preservation via masked LPIPS and pixel-level consistency on unedited areas. Experiments show that our method achieves mask-free, single-object style transfer at inference, producing regionally accurate and visually coherent results that outperform existing diffusion-based editing approaches.

Figures & tables

Figure 2: Overview of the proposed framework. The upper figure illustrates the overall pipeline based on the pretrained Flux.1-Kontext [ 29 ] . Given a context image, a noisy input, and a regional style prompt, image and text tokens are processed through Flux.1-Kontext, where LoRA-MoE modules [ 50 ] adapt attention and projection layers for style-specific learning. The model is optimized with flow matching, focus loss, and cover loss to reconstruct the target stylized image. Style-related attention maps are guided by binary masks, where focus loss concentrates attention within the target area and cover loss ensures spatial coverage for precise localized stylization.
Figure 3: Qualitative comparison of state-of-the-art instruction-based image editing methods.
ModelCOCOPascal VOCBIG
RSM ↑\uparrowLPIPSbg\text{LPIPS}_{\text{bg}} ↓\downarrowMSEbg\text{MSE}_{\text{bg}} ↓\downarrowRSM ↑\uparrowLPIPSbg\text{LPIPS}_{\text{bg}} ↓\downarrowMSEbg\text{MSE}_{\text{bg}} ↓\downarrowRSM ↑\uparrowLPIPSbg\text{LPIPS}_{\text{bg}} ↓\downarrowMSEbg\text{MSE}_{\text{bg}} ↓\downarrow
Flux.1-Kontext [ 29 ]0.6126 0.01800.4546 0.29600.1699 0.22540.6135 0.01700.4326 0.31350.1455 0.22580.6169 0.02280.4815 0.31250.1981 0.2408
Qwen-Image-Edit [ 49 ]0.6235 0.01640.7530 0.16050.4398 0.38760.6333 0.01040.8078 0.15500.4477 0.37670.6329 0.01030.7185 0.16870.4284 0.3731
ICEdit [ 55 ]0.6086 0.01520.3512 0.19790.1568 0.28000.6102 0.01470.3506 0.19630.1284 0.23040.6115 0.01640.4282 0.23150.1570 0.2697
AnyEdit [ 24 ]0.6085 0.01040.6895 0.15640.2633 0.29130.6081 0.01070.6834 0.17150.2345 0.24540.6078 0.00970.7109 0.16790.2446 0.2813
Instruct-Pix2Pix [ 4 ]0.5978 0.01150.1867 0.19950.0516 0.15870.5969 0.01110.1554 0.14720.0299 0.10780.5939 0.01110.1466 0.13770.0347 0.1403
Table 1: Quantitative comparison with baseline methods across three datasets. We report mean std for RSM (↑), LPIPS bg (↓), and MSE bg (↓). Darker cells indicate better ranks (1–3) within each dataset and metric.
ModelCOCOPascal VOCBIG
Q1 ↑\uparrowQ2 ↓\downarrowQ3 ↓\downarrowQ4 ↓\downarrowQ1 ↑\uparrowQ2 ↓\downarrowQ3 ↓\downarrowQ4 ↓\downarrowQ1 ↑\uparrowQ2 ↓\downarrowQ3 ↓\downarrowQ4 ↓\downarrow
Flux.1-Kontext [ 29 ]0.630.440.080.060.650.430.080.050.630.390.110.10
Qwen-Image-Edit [ 49 ]0.980.860.010.000.970.780.020.010.980.900.050.04
ICEdit [ 55 ]0.720.400.110.050.500.320.060.040.630.360.150.06
AnyEdit [ 24 ]0.500.410.570.470.460.280.470.280.580.500.580.47
Instruct-Pix2Pix [ 4 ]0.060.080.030.030.100.050.060.030.100.090.060.07
Table 2: Controllability and semantic reliability evaluation via Vision-Language Model (VLM). Q1: “Is the object in the target style ?”; Q2: “Is the background in the target style ?; Q3: “Is the object in the negative style ?”; Q4: “Is the background in the negative style ?” Each cell shows the probability (%) of a “Yes” response to the corresponding binary question.
VariantEnabled Components / BlocksCOCOPascal VOCBIG
 Lcover\mathcal{L}_{\text{cover}}Lfocus\mathcal{L}_{\text{focus}}DoubleSingleLoRARSM ↑\uparrowLPIPSbg\text{LPIPS}_{\text{bg}} ↓\downarrowMSEbg\text{MSE}_{\text{bg}} ↓\downarrowRSM ↑\uparrowLPIPSbg\text{LPIPS}_{\text{bg}} ↓\downarrowMSEbg\text{MSE}_{\text{bg}} ↓\downarrowRSM ↑\uparrowLPIPSbg\text{LPIPS}_{\text{bg}} ↓\downarrowMSEbg\text{MSE}_{\text{bg}} ↓\downarrow
Ours✓✓✓✓40.6128 0.01620.2103 0.19330.0729 0.12810.6147 0.01510.1331 0.14290.0269 0.05710.6159 0.01930.1593 0.16810.0474 0.0681
w/o Lcover\mathcal{L}_{\text{cover}}✗✓✓✓40.6120 0.01630.2174 0.18890.0730 0.10690.6142 0.01550.1383 0.15300.0359 0.05800.6152 0.01930.1605 0.16800.0488 0.0681
w/o Lfocus\mathcal{L}_{\text{focus}}✓✗✓✓40.6127 0.01590.2132 0.18520.0740 0.11170.6147 0.01530.1359 0.13930.0325 0.04970.6158 0.01900.1612 0.16200.0542 0.0917
w/o Double✓✓✗✓40.6168 0.01680.4225 0.28380.1409 0.18310.6175 0.01550.3887 0.30640.0980 0.14350.6195 0.01980.3843 0.30860.1275 0.1458
w/o Single✓✓✓✗40.6190 0.01630.5203 0.30600.2284 0.22160.6185 0.01580.4252 0.33030.1390 0.18840.6192 0.02050.3933 0.31100.1402 0.1595
Table 3: Ablation of key loss components, network streams, and LoRA ranks on three datasets. We report mean std for RSM (↑), LPIPS bg (↓), and MSE bg (↓). ✓ denotes the component is enabled.
Figure 4: Visualization of attention maps under different loss configurations. Using only Lcover\mathcal{L}_{\text{cover}} or Lfocus\mathcal{L}_{\text{focus}} causes attention spillover to nearby area, whereas our full objective focuses on the motorcycle without leakage to surrounding regions, demonstrating its ability to maintain precise and consistent attention.
Figure 5: Illustration of the pseudo ground-truth (pseudo-GT) generation process. A diffusion-based style transfer model generates a fully stylized version of the image according to a given style prompt. The stylized region corresponding to the mask is then blended back into the original image using seamless cloning, producing an aligned input–target pair for localized style learning.
Figure 6: Examples from our pseudo ground truth dataset for localized style transfer. Each example consists of four elements: the original context image, the corresponding binary mask that specifies the target object, the pseudo ground truth image generated by applying a global style transformation to the entire image and compositing the stylized region back onto the original image, and the training prompt used for fine-tuning.
Figure 7: Additional examples of localized style transfer on objects that also appear in the training data. The model produces clean, structurally consistent stylization within the target regions, showing stable behavior on in-distribution categories.
Figure 8: The first four columns show results on object categories that never appear in the training set. The last two show categories with only synonym-level presence (motorbike → motorcycle, aeroplane → airplane). The model successfully performs localized style transfer even without direct supervision for these object types, indicating strong semantic generalization.
Figure 9: We evaluate the impact of removing single-stream LoRA blocks, double-stream LoRA blocks, the cover loss Lcover\mathcal{L}_{\text{cover}} , and the focus loss Lfocus\mathcal{L}_{\text{focus}} . The full RegionRoute model achieves the clearest and most accurate localized style transfer, while ablated variants exhibit style leakage, incomplete stylization, or degraded unedited object consistency.
Figure 10: Comparison of the style-token attention maps for RegionRoute (ours) and the Flux.1-Kontext baseline. For each task, we show the edited output, the averaged style-token attention map, and the step-wise evolution of style-token attention at diffusion steps 5, 10, 15, 20, and 25.
Figure 11: Examples illustrating several types of failure: (1) incomplete stylization when multiple target objects are present, where only the most salient instances are stylized; (2) failure to stylize very small or heavily occluded objects; (3) unintended stylization of regions adjacent to the target object, especially when objects are physically connected; (4) stylization leakage onto objects that are contained within the target object, such as a person inside a car or pillows on a sofa; and (5) partial stylization of a single object in cases where some object parts are difficult to recognize.
More in this topic

Image Style Transfer

Image style transfer changes an image’s visual appearance to match an artistic, photographic, or other reference while preserving its content.

See all 60 papers

Explore similar work

CardsList

  1. MAST: Mask-Guided Attention Control for Training-Free Regional-Multi Style Transfer

    Apr 14, 2026Dongkyung Kang, Jaeyeon Hwang, Junseo Park +6Image Style TransferAttention Control in Diffusion Models

    Department of Computer Science & Artificial Intelligence, Dongguk University

  2. CoCoDiff: Correspondence-Consistent Diffusion Model for Fine-grained Style Transfer

    Feb 16, 2026Wenbo Nie, Zixiang Li, Renshuai Tao +3Cycle ConsistencyImage Style Transfer

    Institute of Information Science, Beijing Jiaotong University · Visual Intelligence + X International Joint Laboratory of the Ministry of Education

  3. Local Content-Style Control for Diffusion-based Image Stylization

    Oct 6, 2026Amir SemmoLatent Diffusion ModelsImage Style Transfer

    Digital Masterpieces GmbH Potsdam, Germany