TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer

Duc Khoan Le; Kim Ngoc Tran; Minh Nhat Le; Thanh An Tran; Viet Toan Nguyen; Khanh An Lay; Tran Thai Son; Hoang Pham Minh

cs.CVOct 4, 2026

TRACE: Time-Adaptive Residual Attention Control with Content-Style Decomposition for Training-Free Diffusion Style Transfer

Authors: Duc Khoan Le, Kim Ngoc Tran, Minh Nhat Le, Thanh An Tran, Viet Toan Nguyen, Khanh An Lay, Tran Thai Son, Hoang Pham Minh

Organizations: Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Abstract

Reference-guided style transfer aims to preserve the semantic structure of a content image while transferring the visual appearance of a style reference. Recent diffusion-based methods achieve impressive stylization quality by exploiting strong pretrained generative priors. However, training-free approaches still face a difficult trade-off among style fidelity, content preservation, and content leakage. Direct style injection may unintentionally transfer semantic content from the style image, while fixed guidance schedules often ignore the time- and state-dependent nature of diffusion sampling. To address these limitations, we propose TRACE, a training-free diffusion style transfer framework with Time-adaptive Residual Attention Control and Content-Style Decomposition. TRACE first performs offline CLIP-based subspace analysis to separate content and style directions from paired data. During inference, it removes content-related components from the style reference and style-related components from the content reference to reduce leakage. It then injects style information through residual cross-attention and applies uncertainty-aware guidance to adapt the guidance signal at each denoising step. Experiments show that TRACE achieves a favorable trade-off between stylization and preservation. Compared with optimal-control-based baselines, TRACE substantially improves style fidelity (+17.28 CSD and +34.10 SRA). While, compared with stylization methods, it better preserves content structure (+12.80 DINO, +5.52 CLIP-I, and -8.19 LPIPS) and reduces directional semantic leakage by 29.5% in DCL. Our code is publicly available at https://github.com/pixelchemy-research/TRACE.

Figures & tables

Figure 1: Overview of our approach ( TRACE ). TRACE first learns content and style subspaces from paired CLIP embeddings in an offline stage. During inference, it cleans the content and style representations through subspace projection, injects the cleaned style signal via residual style injection, and applies uncertainty-aware time-dependent guidance to control the reverse diffusion trajectory. A detailed TRACE sampling algorithm is provided in Appendix 0.B.2 .

Figure 2: Qualitative comparison with training-free style transfer methods. Red, purple, and cyan boxes denote TRACE, OC-based methods, and stylization methods, respectively. More visualizations are provided in Appendix 0.E.4 .

Model	Style Fidelity		Content Preservation			Leakage
Model	CSD $\uparrow$	SRA $\uparrow$	DINO $\uparrow$	CLIP-I $\uparrow$	LPIPS $\downarrow$	DCL $\downarrow$
Stylization Methods
TRACE	34.920	69.744	65.543	77.271	39.532	21.568
StyleID	44.998	93.333	60.798	74.418	42.633	26.415
AttenST	42.876	88.718	62.115	74.757	38.538	25.699
DiffuseIT	26.468	62.051	42.908	70.109	51.083	38.193

Table 1: Quantitative comparison with stylization and OC-based methods.

\uparrow

indicates higher is better, while

\downarrow

indicates lower is better. Best results are shown in bold, and second-best results are underlined within each group.

Figure 3: Style fidelity and content leakage trade-off. The $y$ -axis shows CSD, and the $x$ -axis shows DCL. TRACE achieves effective stylization while maintaining lower leakage than most stylization baselines. Additional Pareto trade-off analyses are provided in Appendix 0.E.7 .

Exp	$D_{s}$	$R$	$G$	$\rho$	Style Fidelity		Content Preservation			Leakage
Exp	$D_{s}$	$R$	$G$	$\rho$	CSD $\uparrow$	SRA $\uparrow$	DINO $\uparrow$	CLIP-I $\uparrow$	LPIPS $\downarrow$	DCL $\downarrow$
E0	0.6	$\checkmark$	Ada.	0.7	34.920	69.744	65.543	77.271	39.532	21.568
E1	0	$\checkmark$	Ada.	0.7	53.020	90.769	51.586	69.538	44.768	34.098
E2	1	$\checkmark$	Ada.	0.7	26.848	56.410	69.295	78.048	38.002	17.634
E3	0.6	$\times$	Ada.	0.7	17.874	32.821	75.491	84.742	34.489	14.441
E4	0.6	$\checkmark$	No	0.7	20.111	34.359	65.539	79.208	37.444	19.585

Table 2: Unified ablation study of TRACE. E0 denotes the full TRACE model, while E1-E8 denote different ablation settings.

D_{s}

denotes the content-style decomposition strength, where

0

0.6

, and

1

indicate no decomposition, soft decomposition, and full decomposition, respectively.

R

denotes residual attention control, where

\checkmark

indicates residual style injection and

\times

indicates the original cross-attention injection without residual control.

G

denotes the guidance strategy, where Ada., No, C1, and C2 indicate uncertainty-time adaptive guidance, no adaptive guidance, constant guidance with

\lambda=1

, and constant guidance with

\lambda=2

, respectively.

\rho

denotes the noise level, where

0.5

0.7

, and

1.0

correspond to low-noise, ours, and high-noise settings.

\uparrow

indicates higher is better, while

\downarrow

indicates lower is better.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

\hat{x}_{0|t}=\frac{x_{t}-\sqrt{1-\bar{\alpha}_{t}}\epsilon_{\theta}(x_{t},t)}{\sqrt{\bar{\alpha}_{t}}}.

Appendix

Algorithm 1 Uncertainty Estimation

z_{\mathrm{content}}=E_{\phi}(I_{c}),\qquad z_{\mathrm{style}}=E_{\phi}(I_{s}).

Appendix

Algorithm 2 TRACE Sampling

z_{n}\leftarrow\frac{E_{\phi}(I_{n})}{\|E_{\phi}(I_{n})\|_{2}},\quad n=1,\ldots,N.

Appendix

Algorithm 3 Offline Content-Style Decomposition

Figure 4: Representative paired content-style samples used for decomposition analysis. Rows show content identities and columns show style references.

Category	Representative examples
Objects	A chair; A teapot; A laptop; A vintage camera; A pocket watch
Animals	A dog; A lion; A fox; A penguin; A panda
People and actions	A child running in the park; A person dancing; A woman reading in the park; A man skiing down the hill
Vehicles	A train; A racing car; A bicycle; A helicopter; A sailboat
Buildings and scenes	A house; A temple; Golden Gate Bridge; A village; A snowy cabin
Nature and food	Flowers; A sunflower; A watermelon; A bowl with apples; A mushroom

Appendix

Table 3: Representative content and style prompts used for constructing the paired content-style dataset.

$r_{c}$	$r_{s}$	Raw Style	Clean Style	SG $\uparrow$	Clean Content	CR $\uparrow$
32	8/16/32	8.86	15.12	6.26	32.37	21.98
64	8/16/32	8.86	18.55	9.69	21.84	47.37
96	8/16/32	8.86	20.53	11.68	14.13	65.95
128	8/16/32	8.86	21.13	12.27	9.25	77.70

Appendix

Table 4: Top-10 retrieval analysis before and after content-style decomposition. Raw denotes retrieval before decomposition, while Clean denotes retrieval after decomposition. SG indicates style gain and CR indicates content reduction. All values are reported in percentage.

Component	Setting
Diffusion backbone	Stable Cascade [ 23 ]
Content rank	$r_{c}=128$
Style rank	$r_{s}=8$
Decomposition strength	$\alpha=\beta=0.6$
Residual style injection	$\checkmark$
Adaptive guidance	$\checkmark$

Appendix

Table 5: Configuration of TRACE used in the main experiments.

Aspect	Metric	Interpretation
Style fidelity	CSD $\uparrow$	Style similarity to the target style reference
Style fidelity	SRA $\uparrow$	Top-1 retrieval accuracy of the target style
Content preservation	DINO $\uparrow$	Semantic similarity to the content image
Content preservation	CLIP-I $\uparrow$	Image-level similarity to the content image
Content preservation	LPIPS $\downarrow$	Perceptual distance from the content image
Content leakage	DCL $\downarrow$	Semantic leakage from the style reference

Appendix

Table 6: Evaluation metrics used in our experiments.

Panel of Figure 5 — Figure 5: Comparison between residual style injection and original attention.

Panel of Figure 6 — Figure 6: Timestep-wise uncertainty visualization during reverse diffusion. For each example, the top row shows the estimated uncertainty map at different denoising steps, while the bottom row shows the corresponding predicted clean-latent preview $\widetilde{x}_{0|t}$ .

Method	$M$	CSD $\uparrow$	DINO $\uparrow$	DCL $\downarrow$	Time (s) $\downarrow$	VRAM (GB) $\downarrow$	NFE $\downarrow$
		Generation Quality			Efficiency
Const	–	32.386	66.689	20.913	15.448 $\pm$ 0.180	18.499	14
TRACE	2	32.292	70.713	18.838	31.019 $\pm$ 0.441	18.486	28
TRACE	4	37.394	67.582	17.975	33.923 $\pm$ 0.386	18.707	28
TRACE	8	34.883	70.241	18.890	41.682 $\pm$ 0.145	20.061	28

Appendix

Table 7: Effect of uncertainty samples

M

on generation quality and computational cost over 8 styles and 13 content images. Runtime is reported as mean

\pm

standard deviation over 30 runs. The experiments are conducted on an A100 GPU 40GB.

Figure 8: Cross-style ranking of fixed and handcrafted guidance configurations.

Figure 9: Trade-offs of fixed, linear, cosine, and uncertainty-adaptive guidance under different guidance strengths.

Figure 10: Qualitative comparison of uncertainty-adaptive, constant, and handcrafted time-dependent guidance.

Held-out Style-Family Generalization
	Style		Content			Leakage
Setting / Method	CSD $\uparrow$	SRA $\uparrow$	DINO $\uparrow$	CLIP-I $\uparrow$	LPIPS $\downarrow$	DCL $\downarrow$
Full-data	43.213	96.875	68.909	81.269	36.040	19.333
Held-out Style Family	43.292	96.875	69.552	80.684	36.128	19.162
Style Disentanglement
No Decomposition	62.819	66.333	55.170	75.449	38.674	36.307

Appendix

Table 8: Generalization and style-disentanglement analysis. Held-out evaluation uses 6 styles

\times

13 contents, while style disentanglement uses 4 styles with 3 references each

\times

13 contents.

Figure 11: Style disentanglement under semantically different references sharing the same visual style.

Figure 12: Qualitative result on an unseen origami-style reference.

Figure 13: Pairwise Pareto trade-offs among representative style-transfer methods.

Figure 14: Additional qualitative comparison with baseline methods using a texture-dominant style reference. TRACE transfers local texture patterns while better preserving the main content structure.

Figure 15: Additional qualitative comparison with baseline methods using a color-dominant style reference. TRACE captures the target color distribution while maintaining the semantic structure of the content image.

Figure 16: Representative stylization results of TRACE on different content and style pairs. Each example shows how the target style is transferred while preserving the main content structure.

Image Style Transfer

Image style transfer changes an image’s visual appearance to match an artistic, photographic, or other reference while preserving its content.

See all 60 papers

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization
Sep 18, 2026Zhangping Yang, Min Li, Song Yan +4Image Style Transfer Diffusion-Based Image Editing
Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China
Scheduled Style Injection: Expanding the Style-Content Pareto Frontier in Training-Free Diffusion-based Style Transfer
May 26, 2026Amey Sunil KulkarniDiffusion Model Guidance Image Style Transfer
Independent Researcher
AnyStyle: A Single LoRA is Sufficient for Image-Guided Style Transfer
Jul 6, 2026Yongwen Lai, Chaoqun WangDiffusion Model Guidance Image Style Transfer
School of Artificial Intelligence, South China Normal University, Guangzhou, China

Sep 18, 2026J/K move · Enter open · S save

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

Zhangping Yang, Min Li, Song Yan, Rong Gao+3

Xi’an High-tech Research Institute, Shaanxi, China · University of Science and Technology of China, Anhui, China · Xi’an University of Architecture and Technology, Shaanxi, China

Image Style Transfer Diffusion-Based Image Editing Diffusion Model Guidance Representation Disentanglement

Open paper AskPDF arXiv

Reference-guided diffusion stylization aims to transfer visual style from a reference image while preserving the semantics specified by a text prompt. However, image conditioning often entangles transferable style cues with reference-specific content, leading to an inherent trade-off: stronger conditioning improves style fidelity but increases content leakage, whereas aggressive suppression reduces leakage at the cost of style expression. This challenge is further complicated by the distinct spatial organization of texture- and geometry-dominant styles. To address these issues, we propose SafeStyle, a training-free framework for calibrated style residual injection in frozen diffusion models. SafeStyle first estimates style-supported and content-associated subspaces from compact calibration sets, preserving their informative overlap while suppressing useless content variations. It then transports the purified style evidence over adaptive spatial granularity and constrains its effective influence through an explicit residual-norm budget. Experiments across texture- and geometry-dominant styles show that SafeStyle achieves a DINO style similarity of 0.432 while maintaining competitive text alignment. On a semantically disjoint leakage-stress benchmark, it further achieves a DINO style similarity of 0.474 with only 0.8% semantic leakage, demonstrating an effective balance between style fidelity and reference-content suppression.

Abstract

Image Style Transfer

Explore similar work

SafeStyle: Calibrated Style Residual Injection for Controllable Style-Leakage Trade-off in Diffusion Stylization

Scheduled Style Injection: Expanding the Style-Content Pareto Frontier in Training-Free Diffusion-based Style Transfer

AnyStyle: A Single LoRA is Sufficient for Image-Guided Style Transfer