Local Content-Style Control for Diffusion-based Image Stylization

Amir Semmo

cs.GROct 6, 2026

Local Content-Style Control for Diffusion-based Image Stylization

Authors: Amir Semmo

Organizations: Digital Masterpieces GmbH Potsdam, Germany

Abstract

Image stylization with latent-diffusion models entangles two independently refined axes: what a region depicts and how it is depicted. Such pipelines expose only global controls, yet professional retouching demands deliberate, region-specific control. We lift two conditioning weights already present in a ControlNet + IP-Adapter stylization pipeline from global scalars to per-location spatial maps, yielding local, per-axis control of content and style in a single generative pass. Because the two weights act on disjoint pathways, adjusting them independently spans a 2x2 retouching vocabulary, from free regeneration to identity preservation. We validate that edits stay confined to the retouched region and that each weight predominantly steers its own axis. Our approach requires no retraining and drops unchanged into any such pipeline.

Figures & tables

Figure 1. Locking content and/or relaxing style spans four local retouching operations. Photo by D. Perevoshchikov on Unsplash. A content photo of a woman holding yellow flowers and a cubist style painting, each with a drawn region map, followed by four stylized results: a freely regenerated cubist rendering; locked content keeping her structure; relaxed style making her appear photographic; and both together preserving her identity amid a cubist background.

Figure 2. Weight spatialization overview. (a) A content map $s(\mathbf{p})$ scales the ControlNet control embedding (Eq. 1 ). (b) A style map $\lambda_{b}(\mathbf{p})$ scales the decoupled image-attention term in each IP-Adapter block (Eq. 2 ). Photo by D. Perevoshchikov on Unsplash. Pipeline diagram of an SDXL U-Net with a ControlNet branch and an IP-Adapter path. A user-drawn content map scales the ControlNet control embedding before its residuals enter the U-Net, and a user-drawn style map scales the image-attention term in each IP-Adapter cross-attention block; two insets show the per-location products.

Panel of Figure 3 — Figure 3. Locality and specificity on one evaluation triple (Table 1 ). Content-lock ( $C$ ) recovers the wing markings and style-relax ( $S$ ) restores their contrast, while the change ( $|\Delta|$ heatmaps, LPIPS vs. baseline $B$ ) stays confined to the masked butterfly. A butterfly photo with its mask, a style image, and three results (default stylization, content-lock, style-relax), with heatmaps of change relative to the default that are bright inside the butterfly and dark in the background.

Control	inside	styliz.	roll	edges	texture	palette
	H1: $\ell_{\infty}$ relative to ( $\downarrow$ )			H2: signed probe response
lock content ( $s{\to}0.8$ )	0.33	0.18	0.24	$-0.026$	$-0.046$	$-0.003$
relax style ( $\lambda_{b}{\to}0.7$ )	0.26	0.10	0.13	$-0.004$	$-0.024$	$+0.003$
relax style ( $\lambda_{b}{\to}0.4$ )	0.27	0.16	0.21	$-0.006$	$-0.063$	$-0.005$

Table 1. Locality (H1) and specificity (H2) over

N{=}300

triples (

10

contents

\times

10

styles

\times

3

seeds). H1 :

\ell_{\infty}

relative to the inside response, stylization magnitude, and a seed re-roll (lower

=

more local). H2 : signed responses (negative

=

toward the source; compare within columns—probe scales differ); content loads on structure (edges), style on texture.

Panel of Figure 4 — Figure 4. Control maps from perceptual priors: source, style, and mask (top) and uniform vs. map-driven result (bottom). Depth sets the style weight (left), relaxing style frees the prompt’s “purple” (middle), and an object mask sharpens the glasses (right). Three demos, each with a source photo, a style image, and an automatically derived control mask above a uniform and a map-driven result: depth controlling watercolor strength over a tulip field, a subject mask freeing a water lily's purple color, and an object mask sharpening two glasses in a suprematist scene.

Explore similar work

Staying True to the Origin: Continuous Image Stylization with Smooth Transitions
Aug 8, 2026Rui Xu, Hanmo Zhang, Songhua LiuImage Style Transfer Diffusion-Based Image Editing
Shanghai Jiao Tong University
Stylistic Attribute Control in Latent Diffusion Models
May 4, 2026Max Reimann, Benito Buchheim, Jürgen DöllnerDiffusion Model Guidance Latent Diffusion Models
Hasso-Plattner-Institute, University of Potsdam, Germany
ResetEdit: Precise Text-guided Editing of Generated Image via Resettable Starting Latent
Apr 28, 2026Hanyi Wang, Han Fang, Zheng Wang +2Diffusion Model Inversion Text-Guided Image Editing
Shanghai Jiao Tong University China · University of Science and Technology of China China · Wuhan University China +1

Aug 8, 2026, cs.CVJ/K move · Enter open · S save

Staying True to the Origin: Continuous Image Stylization with Smooth Transitions

Rui Xu, Hanmo Zhang, Songhua Liu

Shanghai Jiao Tong University

Image Style Transfer Diffusion-Based Image Editing Controllable Image Generation

Open paper AskPDF arXiv

Recent advances in generative models have achieved remarkable performance in text- and image-conditioned editing. However, preserving the content of a given image while referencing style patterns from another remains challenging, often leading to uncontrollable stylization results. In this paper, we approach image stylization from the perspective of continuous control, aiming to enable modern Diffusion Transformer (DiT)-based multi-reference editing models to (1) faithfully preserve the semantic structure of the content image, (2) render strong stylization effects, and (3) smoothly transition between the two. To this end, we propose a simple yet effective two-stage training strategy along with a style-strength-aware spline formulation. Specifically, in the first stage, the model is trained to produce strongly stylized outputs while preserving the content semantics as much as possible. In the second stage, with the base model frozen, we learn a set of anchor projectors that map various stylization strengths into the model parameter space. During inference, by performing style-strength-aware spline interpolation in a low-rank space, our method enables continuous control over stylization strength, even though the model is trained with only a few discrete strength levels. Extensive experiments demonstrate that our method supports precise and continuous manipulation of stylization strength while generating high-fidelity results with modern DiT models. Project page: https://reychiaro.github.io/StyleController.

Abstract

Explore similar work

Staying True to the Origin: Continuous Image Stylization with Smooth Transitions

Stylistic Attribute Control in Latent Diffusion Models

ResetEdit: Precise Text-guided Editing of Generated Image via Resettable Starting Latent