Facial stylization asks for a delicate transformation: the image should look unmistakably artistic, yet the person should remain unmistakably the same. Existing training-free and instruction-driven methods often fail to expose such control, either drifting away from identity or producing only weak stylization. We introduce Stylus, a training-free framework that treats facial style transfer as navigation on the curved latent manifold of a pretrained diffusion model. Rather than interpolating in flat Euclidean space, \method refines the interpolated content--style point by lowering a ratio-weighted discrete geodesic energy under the pullback metric, made tractable by a rank-one power-iteration approximation of the Jacobian. This principle drives two complementary controls: content--style latent interpolation for smooth semantic fusion, and self-attention query optimization for spatially adaptive style injection. The resulting dual-control mechanism provides continuous, fine-grained control over stylization strength while preserving identity-critical facial structure, all without additional training. Experiments show that Stylus achieves the strongest identity and content preservation at competitive stylization strength.
Figures & tables
Figure 1: Geometric interpretation of the gradient computation: the hidden-state map g maps points from coordinate space X to the manifold M , while the Jacobian Jg projects the manifold gradient onto the tangent space of X .
Figure 2: Visual comparison of style transfer results. Our Stylus better preserves facial identity while effectively transferring the artistic style compared to baseline methods.
WikiArt
MetFace
Method
ArtFID ( ↓ )
LPIPS ( ↓ )
ID ( ↓ )
CS ( ↑ )
ArtFID ( ↓ )
LPIPS ( ↓ )
ID ( ↓ )
CS ( ↑ )
AesPA-Net Hong et al. (2023)
28.3402
0.4076
0.2696
0.4254
29.1623
0.4853
0.2968
0.3948
ArtFlow An et al. (2021)
22.9778
0.4933
0.2555
0.3215
30.2701
0.4717
0.2484
0.3932
CAST Zhang et al. (2022)
33.0048
0.3856
0.2832
0.4064
28.6567
0.4109
0.2877
0.4293
StyTR 2 Deng et al. (2022)
24.2880
0.4068
0.1760
0.5181
26.6994
0.4104
0.1599
0.5503
InST Zhang et al. (2023b)
29.8916
0.5975
0.8075
0.1862
33.4454
0.6084
0.8432
0.2447
Table 1: The style transfer results with using WikiArt and MetFace datasets as style reference.
Figure 3: Style interpolation results showing smooth transitions between different artistic styles.
Figure 5
Figure 5: Discrete path-curvature statistics for linear, energy-refined, and randomly perturbed interpolation trajectories.
Figure 7Figure 8
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
k
ArtFID ( ↓ )
LPIPS ( ↓ )
ID ( ↓ )
Time (s/image)
1 (default)
26.6203
0.3118
0.1424
9.71
2
26.51
0.3103
0.1418
15.4
4
26.49
0.3099
0.1416
25.8
Appendix
Table 3: Rank-one vs. top- k singular directions (MetFace style references, SD v1.4). k=1 is the default setting of Stylus, identical to Table 1 .
Figure 8: More Visualization Samples of the proposed Stylus ∗ .
Figure 9: More Visualization Samples of the proposed Stylus ∗ .
Method
Style Faithfulness ↑
Identity Preservation ↑
Visual Quality ↑
Overall Preference ↑
StyTR 2 Deng et al. (2022)
3.62 ± 0.41
3.71 ± 0.38
3.68 ± 0.36
3.67 ± 0.39
StyleID Chung et al. (2024)
3.89 ± 0.36
4.02 ± 0.31
3.94 ± 0.34
3.96 ± 0.33
StyleSSP Xu et al. (2025)
3.74 ± 0.38
3.85 ± 0.33
3.79 ± 0.35
3.80 ± 0.34
Stylus (Ours)
4.31 ± 0.29
4.47 ± 0.24
4.28 ± 0.27
4.35 ± 0.26
Appendix
Table 4: User study results for the four evaluated methods (mean ± standard deviation, scale 1–5, higher is better).
Table 6: Style transfer results of StyleID and Stylus with different Stable Diffusion base models, using WikiArt and MetFace as style references.
Figure 10: The style transfer results of object stylization, scene stylization and art-to-art transfer.
Figure 16
Figure 13: Effect of the number of sampling steps (20 vs. 50) on style transfer performance, using MetFace as style references.
Figure 14: Linear baseline interpolation.
Figure 15: More visualization results showing style transfer examples with different content and style image pairs.
Figure 16: Qualitative comparison of style transfer results against DiT-based editing models.
Figure 17: Qualitative comparison of style transfer results against diffusion-based and conventional baselines.
Figure 18: Zoom-in visualization comparing original and stylized image regions. The detailed views highlight our method’s ability to preserve facial features while incorporating artistic elements.
Figure 19: Additional artistic-to-artistic style transfer results. Each row shows a different example with content image, style reference, and our stylization result.
Figure 20: Additional style transfer results using non-facial artistic references. Each row shows a different example with content image, non-facial style reference, and our stylization result.
Figure 21: Style transfer results using different versions of Stable Diffusion models.
Figure 22: Style transfer results using different versions of Stable Diffusion models.
Image stylization with latent-diffusion models entangles two independently refined axes: what a region depicts and how it is depicted. Such pipelines expose only global controls, yet professional retouching demands deliberate, region-specific control. We lift two conditioning weights already present in a ControlNet + IP-Adapter stylization pipeline from global scalars to per-location spatial maps, yielding local, per-axis control of content and style in a single generative pass. Because the two weights act on disjoint pathways, adjusting them independently spans a 2x2 retouching vocabulary, from free regeneration to identity preservation. We validate that edits stay confined to the retouched region and that each weight predominantly steers its own axis. Our approach requires no retraining and drops unchanged into any such pipeline.
Recent advances in generative models have achieved remarkable performance in text- and image-conditioned editing. However, preserving the content of a given image while referencing style patterns from another remains challenging, often leading to uncontrollable stylization results. In this paper, we approach image stylization from the perspective of continuous control, aiming to enable modern Diffusion Transformer (DiT)-based multi-reference editing models to (1) faithfully preserve the semantic structure of the content image, (2) render strong stylization effects, and (3) smoothly transition between the two. To this end, we propose a simple yet effective two-stage training strategy along with a style-strength-aware spline formulation. Specifically, in the first stage, the model is trained to produce strongly stylized outputs while preserving the content semantics as much as possible. In the second stage, with the base model frozen, we learn a set of anchor projectors that map various stylization strengths into the model parameter space. During inference, by performing style-strength-aware spline interpolation in a low-rank space, our method enables continuous control over stylization strength, even though the model is trained with only a few discrete strength levels. Extensive experiments demonstrate that our method supports precise and continuous manipulation of stylization strength while generating high-fidelity results with modern DiT models. Project page: https://reychiaro.github.io/StyleController.
Style transfer with pre-trained diffusion models has advanced rapidly, but a core question remains underexplored: where in the model should style injection be strongest? StyleID, the leading training-free method, uses a single global parameter (gamma) uniformly across all layers and timesteps, which forces a fixed tradeoff between style quality and content preservation. We show this tradeoff is unnecessarily rigid. We systematically explore four dimensions of control: varying style injection strength across decoder layers, across denoising timesteps, and scheduling ControlNet geometric conditioning along both axes. The pattern is consistent everywhere: decreasing schedules, with stronger structural signal injection in shallower layers and earlier timesteps, reliably outperform the reverse. Beyond direction, schedule shape matters: cosine and square-root timestep schedules outperform linear. Most importantly, we find that gamma scheduling and ControlNet conditioning are nearly independent. The resulting combined configurations expand the Pareto frontier, offering superior tradeoffs between style fidelity and content preservation compared to any single baseline setting. Our best balanced configuration achieves ArtFID of 27.036 versus StyleID's 28.801 - a 6.1% relative improvement, with consistent gains across the full style-content tradeoff frontier. Results are validated across 35 configurations totaling over 28,000 stylized images using four complementary metrics. These findings generalize across SD backbones with identical rank ordering. All modifications are training-free, parameter-free, and require only a few lines of scheduling code; code is available at https://github.com/ameyskulkarni/scheduled_style_injection.