Personalized image generation aims to synthesize text-driven images conditioned on reference images, while mainly casting the generation as image customization for foreground and style transfer for background. Previous arts of diffusion models suffers from the text misalignment with background for image customization and foreground for style transfer during the denoising process. Such facts, as we observed, rooted from the entanglement among hybrid frequency bands during the denoising process. To address such salient limitation, in this paper, we study personalized generation based on dual references - customization and color and style reference - and propose a paradigm to disentangle these Dual image references within Frequency-aware Diffusion Models, dubbed Dual-FDM, to simultaneously tackle two crucial personalized image generation tasks: customization style transfer and color style transfer, by disentangling different frequency bands via mask strategy within frequency domain. For customization style transfer, we replace the mid-frequency band of the background in the style reference with that from the foreground of the customized reference. For color style transfer, we substitute the low-frequency band of the background in the style reference with that from both the foreground and background of the color reference. Both the substituted frequency bands are used as the key and value to reconstruct the query foreground and background of the denoised personalized image.Extensive experiments validate the superiority of Dual-FDM over the state-of-the-art diffusion models for personalized image generation. Our code can be accessed from https://github.com/htyjers/Dual-FDM.
Figures & tables
Figure 1 : Personalized images generated by our Dual-FDM . (a) The generated examples of Dual-FDM cust for customized references (left) and style references (above); (b) The generated examples of Dual-FDM cost for color references (left) and style references (above).
Figure 2 : (a) Comparison of the generated results, as well as the attention score map of text prompts and image references, between IP-Adapter Ye et al. (2023) and MS-Diffusion Wang et al. (2025) for image customization. (b) Comparison of the generated results between InstantStyle Wang et al. (2024a) and StyleStudio Lei et al. (2025) for style transfer. (c) Comparison of generated results guided by different frequency band of the image reference.
Figure 3 : Illustration of our proposed Dual-FDM pipeline (a); For customization style transfer (b), we replace the mid-frequency band of the background in the style reference with that from the foreground of the customized reference, named Dual-FDM cust (Sec. 2.3 ). For the color style transfer (c), we substitute the low-frequency band of the background in the style reference with that from both the foreground and background of the color reference, named Dual-FDM cost (Sec. 2.4 ). Each of the two personalized image generation tasks are conducted in the early and late stage of the whole denoising process guided by text prompts as illustrated in Eq. ( 1 ).
Figure 4
Figure 6 : Illustration of (a) mid-frequency replacing layer and (b) low-frequency replacing layer.
Figure 7 : Comparison of (a) customization style transfer and (b) color style transfer with state-of-the-arts. Dual-FDM delivers superior results compared with existing approaches.
Figure 7
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Both foreground and background
Foreground
Background
1
A snow leopard is leaping across a fractured frozen expanse.
a snow leopard
the fractured frozen expanse
2
A giant squid is drifting in a radiant deep chasm.
a giant squid
the radiant deep chasm
3
A phoenix is glowing over an age-worn conflict field.
the phoenix
the age-worn conflict field
4
A red-crowned crane is gliding through a crimson-lit horizon.
a red-crowned crane
the crimson horizon
5
A polar bear is sitting beside a fractured icy dome-like form.
a polar bear
the fractured icy dome
6
A chameleon is blending into a glare-soaked luminous corridor.
a chameleon
the luminous corridor
Appendix
Table 6 : The examples of text prompts for both foreground and background, foreground, and background to discussion intuitions for the rationale between Wkim and Wkim .
Method
Foreground
Background
CLIP-T ↑
CLIP-I ↑
DINO-I ↑
CLIP-T ↑
CLIP-I ↑
DINO-I ↑
Wkim
0.3115
0.8228
0.5774
0.2303
0.6157
0.1353
Wkim
0.2629
0.7090
0.2591
0.2486
0.6421
0.1966
Appendix
Table 7 : Quantitative comparisons of foreground–background disentanglement across different images via Wkim and Wkim .
Method
CLIP-T-Mean ↑
CLIP-T-Std ↓
CLIP-I-Mean ↑
CLIP-I-Std ↓
σtop-10%=0
0.2469 (-0.68%)
0.0352 (+13.5%)
0.6450 (+0.45%)
0.0867 (+8.36%)
σtop-20%=0
0.2496 (+0.40%)
0.0344 (+11.0%)
0.6433 (+0.19%)
0.0806 (+0.62%)
σtop-30%=0
0.2500 (+0.56%)
0.0332 (+7.10%)
0.6401 (-0.31%)
0.0842 (+5.11%)
σtop-40%=0
0.2495 (+0.36%)
0.0334 (+7.74%)
0.6382 (-0.30%)
0.0826 (+3.11%)
σtop-50%=0
0.2499 (+0.52%)
0.0322 (+3.87%)
0.6416 (-0.08%)
0.0868 (+8.36%)
σe−σ
0.2498 (+0.49%)
0.0330 (+6.45%)
0.6401 (-0.31%)
0.0883 (+10.2%)
Appendix
Table 8 : Ablation studies are conducted to evaluate three strategies for manipulating singular values: σtop-n%=0 , which removes the largest n% of singular values by setting them to zero; σe−σ , a reweighting scheme that suppresses larger singular values while retaining smaller ones; and ours ( σ1 ), which inversely scales singular values. These strategies are examined to assess their effectiveness in suppressing foreground components and enhancing background components.
Figure 10 : Generation results of different T2I models.
Figure 11 : Ablation studies about the influence of different denoising timesteps between the early and late stages on generating the personalized images.
Figure 12 : Ablation studies about the influence of the different masks to filter mid-frequency band on generating the personalized images.
Figure 13 : Ablation studies about the influence of the different masks to filter low-frequency band on generating the personalized images.
Figure 14 : Failure examples for customization style transfer
Image customization learns target subjects from reference concept images and generates conditioned images per text prompts, mainly modifying styles or backgrounds. Prevailing methods adopt fine-tuning to pack diverse concept attributes into a unified latent embedding, yet entangled attributes hinder elimination of irrelevant disturbances from style and background. To address this issue, we propose Equilibrated Diffusion, a frequency-driven approach that disentangles tangled concept features for balanced customization and consistent text-visual matching. Unlike conventional methods learning full concepts with shared embeddings and unified tuning, our work utilizes the inherent link between image frequency components and semantics: low frequencies represent subject content and high frequencies correspond to styles. We decompose concepts in frequency space and optimize each embedding independently. This separate optimization enables the denoiser to capture style detached from subject identity and generalize better to unseen stylistic prompts. Merging multi-frequency embeddings preserves the model's original spatial customization ability. We further deploy mask-guided diffusion to restrict irrelevant background changes and boost text alignment. Residual Reference Attention (RRA) is inserted into spatial attention to retain subject structure and identity consistency. Experiments prove Equilibrated Diffusion exceeds mainstream baselines on subject fidelity and text adherence, verifying our method's superiority.
Liyuan Ma, Xueji Fang, Guo-Jun Qi
Westlake University · Hangzhou, China · Zhejiang University
We consider the problem of generating images whose internal structure -- defined by the distribution of patches across multiple scales -- matches that of a single reference image. Recent approaches address this problem by training a diffusion model on a single image. But even in this setting, training is computationally expensive and requires hours of optimization. Instead, we model the image using a dataset of its patches at different scales. As this dataset is finite and the dimensionality of its patches is small, the score function for a noisy patch can be computed tractably using an optimal, closed-form denoiser, eliminating the need for neural network training. We integrate this patch-based denoiser into an efficient, training-free image diffusion model, and we describe how our method connects to classical patch-based image restoration techniques. Our approach achieves state-of-the-art generation quality and diversity compared to trained single-image diffusion models, and we demonstrate applications, including unconditional image generation, text-guided stylization, image symmetrization, and retargeting. Further, we show that our approach is compatible with latent space diffusion, and we show multiple additional acceleration techniques to achieve megapixel single-image generation in one second, and gigapixel generation in minutes.
Haojun Qiu, Kiriakos N. Kutulakos, David B. Lindell
Department of Computer Science, University of Toronto · Vector Institute
Text-to-image diffusion models generate images by gradually converting white Gaussian noise into a natural image. White Gaussian noise is well suited for producing diverse outputs from a single text prompt due to its absence of structure. However, this very property limits control over, and predictability of, specific visual attributes, as the noise is not human-interpretable. In this work, we investigate the characteristics of the input noise in diffusion models. We show that, although all frequencies in white Gaussian noise have comparable statistical energy, low-frequency components primarily determine the images global structure and color composition, while high-frequency components control finer details. Building on this observation, we demonstrate that simple manipulations of the low-frequency noise using low-frequency image priors can effectively condition the generation process to reconstruct these low-frequency visual cues. This allows us to define a simple, training-free method with minimal overhead that steers overall image structure and color, while letting high-frequency components freely emerge as fine details, enabling variability across generated outputs.