Diffusion models represent one of the most advanced paradigms in generative modeling. Leveraging their development, a growing number of style transfer methods based on diffusion models have been proposed. However, among these methods, multi-image style transfer approaches that require at least five to ten style examples tend to achieve more satisfactory results. Single-image methods, by contrast, often struggle with either insufficient content preservation or inadequate style fidelity. This greatly limits style extraction from scarce artworks and undermines their artistic value. To address this, we propose Abstract-LoRA, a method that pushes the boundaries of single-image style transfer through lightweight LoRA training on specific U-Net blocks in diffusion models. Specifically, our work is inspired by B-LoRA, a style transfer method that achieves basic style-content disentanglement by training specific U-Net blocks. However, it suffers from a critical limitation: the inability to capture complex backgrounds. Building upon B-LoRA, our method conducts a more refined analysis of U-Net blocks, employing additional U-Net blocks and clustering-based abstraction of style images to better disentangle and balance style and content. Extensive experiments demonstrate that our proposed method not only generates visually more harmonious and satisfying artistic images but also quantitatively improves the preservation of both style and content in the final outputs.
Figures & tables
Figure 1: Architecture evolution from B-LoRA to Abstract-LoRA. (a) B-LoRA injects LoRA weights only at B2 and B3, leaving background content unconstrained. (b) Attempt (v1) adds an additional injection at B1 to capture background information, improving content preservation at the cost of style fidelity. (c) The final Abstract-LoRA further injects an abstract color representation (scaled by 0.5) into B3, recovering style fidelity with minimal cost to content.
Figure 2: Prompt injection effect on the generated image. In the subject-injection test (left), injecting p^ into B2 changes the subject while the background remains unaffected. In the background-injection test (right), injecting p^ into B1 changes the background while the subject remains unaffected.
Figure 3: Detailed content–style–abstract disentanglement mechanism. Top: the style, content, and abstract-representation images are each decomposed into per-block LoRA weights; the content-derived weights ( WB1content , WB2content ) and the style-derived weight ( WB3style ) are combined to form Attempt (v1), and further combined with the abstract representation’s weight ( WB3abstract , scaled by 0.5) to form the final Abstract-LoRA. Bottom: pipeline for extracting the abstract representation from the style image via K-means clustering, filtering, and top- n color selection and mapping.
Figure 4: Comparison between Attempt (v1) and naively scaling up the style image’s LoRA weights by a factor of 2, across two representative content–style pairs. Increasing the weight intensifies certain stylistic characteristics but also introduces noticeable content distortion and disrupts style-content disentanglement, confirming that this issue cannot be resolved through weight scaling alone.
Figure 5: Rows 1-4 correspond to the results of a representative content-style pair from scenarios 1-4, respectively, across different methods.
Gatys et al.
B-LoRA
Attempt (v1)
Abstract-LoRA
scenario1
VGG
0.901
0.919
0.872
0.905
DINOv2
0.311
0.389
0.543
0.539
Total
1.212
1.308
1.415
1.444
scenario2
VGG
0.912
0.904
0.881
0.902
DINOv2
0.505
0.408
0.562
0.553
Total
1.417
1.312
1.443
1.455
Table 1: Quantitative results of different methods in four scenarios
Image-guided style transfer aims to apply the artistic characteristics of a style image to a content image while preserving its semantic structure and layout. Despite advances in diffusion-based methods, existing approaches often face challenges in disentangling content and style, particularly when independently optimized adapters are naively combined, causing conflicts between adapters and limiting controllability over the content-style balance in inference. We further demonstrate that training-free structural guidance directly derived from the content image through the internal attention of pre-trained model outperforms a dedicated content LoRA adapter in terms of structural fidelity and computational efficiency. Building on these observations, we propose AnyStyle, a streamlined framework for image-guided style transfer. The framework adopts a unified single-adapter paradigm for coherent style capture from the style image and incorporates training-free structural guidance from the content image, thus avoiding complex entanglement between multiple adapters and improving controllability and stability. Extensive experiments show that our method delivers competitive quantitative performance and significantly improved perceptual quality. Code is available at https://github.com/Yvan1001/AnyStyle.
Yongwen Lai, Chaoqun Wang
School of Artificial Intelligence, South China Normal University, Guangzhou, China
Diffusion-based style transfer must balance inference efficiency with stylization fidelity. Adapter-based methods are efficient, but they inject style as an external condition and can either weaken reference-specific appearance or copy reference semantics into the generated image. Optimization-based personalization methods such as LoRA internalize style more effectively, but require a separate training process for every new style. We introduce i2L (image-to-LoRA), a framework that amortizes style LoRA training into a single forward pass. Given one or more reference images, i2L predicts LoRA weights for a text-to-image model, enabling immediate style instantiation without per-style optimization. The architecture combines an image encoder, learnable LoRA queries, and compressed decoding heads that generate adapted matrices. Training on semantically diverse style pairs encourages the predictor to preserve appearance cues while suppressing reference-content copying. Experiments on Z-Image, FLUX.2, and Hidream-O1 show that i2L improves style fidelity, prompt alignment, and perceptual quality over existing baselines. Because i2L produces explicit LoRA weights, it also supports asymmetric classifier-free guidance, multi-reference style fusion, and composition with controllable-generation modules.
Diffusion models have demonstrated remarkable performance in image generation, particularly within the domain of style transfer. Prevailing style transfer approaches typically leverage pre-trained diffusion models' robust feature extraction capabilities alongside external modular control pathways to explicitly impose style guidance signals. However, these methods often fail to capture complex style reference or retain the identity of user-provided content images, thus falling into the trap of style-content balance. Thus, we propose a training-free style transfer approach via heterogeneous attention modulation (HAM) to protect identity information during image/text-guided style reference transfer, thereby addressing the style-content trade-off challenge. Specifically, we first introduces style noise initialization to initialize latent noise for diffusion. Then, during the diffusion process, it innovatively employs HAM for different attention mechanisms, including Global Attention Regulation (GAR) and Local Attention Transplantation (LAT), which better preserving the details of the content image while capturing complex style references. Our approach is validated through a series of qualitative and quantitative experiments, achieving state-of-the-art performance on multiple quantitative metrics.
Yeqi He, Liang Li, Zhiwen Yang +3
Hangzhou Dianzi University · Institute of Computing Technology, Chinese Academy of Sciences · Macao Polytechnic University