Cross-attention is the primary interface through which text conditions latent diffusion models, yet its step-wise multi-resolution dynamics remain under-characterized, limiting principled training-free control. We cast diffusion cross-attention as a spatiotemporal signal on the latent grid by summarizing token-softmax weights into token-agnostic concentration maps and tracking their radially binned Fourier power over denoising. Across prompts and seeds, encoder cross-attention exhibits a consistent coarse-to-fine spectral progression, yielding a stable time-frequency fingerprint of token competition. Building on this structure, we introduce Attention Frequency Modulation (AFM), a plug-and-play inference-time intervention that edits token-wise pre-softmax cross-attention logits in the Fourier domain: low- and high-frequency bands are reweighted with a progress-aligned schedule and can be adaptively gated by token-allocation entropy, before the token softmax. AFM provides a continuous handle to bias the spatial scale of token-competition patterns without retraining, prompt editing, or parameter updates. Experiments on Stable Diffusion show that AFM reliably redistributes attention spectra and produces substantial visual edits while largely preserving semantic alignment. Finally, we find that entropy mainly acts as an adaptive gain on the same frequency-based edit rather than an independent control axis.
Figures & tables
Figure 3: Quantitative summary of coarse-to-fine and AFM effects (encoder, top- K ). (a) uses denoising progress u(s) (early → late); tick labels show decreasing DDIM timesteps τs . (b) shows radial energy profiles over normalized radius r .
Stage
Neg. pairs (%)
Mean Δρlate (min–max)
Encoder
98–99
[ −0.063,−0.040 ]
Middle
72–78
[ −0.0346,−0.0317 ]
Decoder
56–64
[ −0.0161,−0.0092 ]
Table 1: Robustness of late-stage HF-ratio shift Δρlate across rc∈{0.20,0.25,0.30} and Nsub∈{50,100} (top- K=8 ). Δρlate is averaged over the last 20% denoising steps ( u≥0.8 ). AFM is applied only to the encoder cross-attention; middle/decoder reflect downstream changes. We report the fraction of pairs with Δρlate<0 and the min–max range of the mean Δρlate across sweeps.
SD v1.5
SD v1.4
Comparison
COCO
LAION
COCO
LAION
Baseline vs. AFM-curve
0.237 ± 0.138
0.249 ± 0.142
0.232 ± 0.132
0.258 ± 0.144
Baseline vs. AFM-curve + entropy
0.409 ± 0.131
0.419 ± 0.149
0.405 ± 0.134
0.417 ± 0.143
Baseline vs. Baseline (entropy, λ=0 )
0.000 ± 0.000
0.000 ± 0.000
0.000 ± 0.000
0.000 ± 0.000
Baseline vs. FreeU
0.300 ± 0.131
0.302 ± 0.158
0.296 ± 0.117
0.294 ± 0.112
Baseline vs. SAG
0.111 ± 0.084
0.125 ± 0.108
0.106 ± 0.055
0.121 ± 0.071
Table 2: Paired LPIPS ↑ (mean ± std) under matched prompt/seed sampling. Higher indicates larger perceptual deviation from Baseline. Results are reported for SD v1.5 (main) and SD v1.4 (checkpoint robustness).
Setting
LPIPS low
LPIPS high
High/Low
P(high>low)
AFM-curve
0.171 ± 0.115
0.188 ± 0.112
1.21 ± 0.28
78.0%
AFM-curve + entropy
0.312 ± 0.116
0.329 ± 0.110
1.08 ± 0.15
72.5%
Entropy only ( λ=0 )
0.000 ± 0.000
0.000 ± 0.000
–
–
Table 3: Band-wise LPIPS decomposition (COCO; N=200 ). We decompose each output image into Ilow=Gσ(I) (Gaussian blur; σ=4 at 512×512 ) and Ihigh=clip[0,1](I−Ilow+0.5) , and compute LPIPS on each band after resizing to 256×256 . LPIPS low proxies structure/layout change and LPIPS high proxies detail/texture change. High/Low is the per-pair ratio LPIPS high /LPIPS low averaged over pairs (undefined for the exact no-op). P(high>low) is the fraction of pairs where LPIPS high exceeds LPIPS low .
SD v1.5
SD v1.4
Setting
COCO
LAION
COCO
LAION
Baseline
0.318 ± 0.031
0.306 ± 0.038
0.320 ± 0.027
0.306 ± 0.039
AFM-curve
0.319 ± 0.030
0.306 ± 0.039
0.318 ± 0.029
0.304 ± 0.040
AFM-curve + entropy
0.316 ± 0.030
0.303 ± 0.040
0.317 ± 0.030
0.303 ± 0.038
FreeU
0.307 ± 0.031
0.310 ± 0.037
0.307 ± 0.029
0.310 ± 0.034
SAG
0.305 ± 0.032
0.307 ± 0.036
0.304 ± 0.029
0.308 ± 0.033
Table 4: CLIP cosine similarity ↑ (ViT-B/32; mean ± std) under matched prompt/seed sampling. Higher indicates better text–image alignment. Results are reported for SD v1.5 (main) and SD v1.4 (checkpoint robustness).
Text-to-image diffusion models have achieved remarkable progress in image synthesis, yet can exhibit memorization by closely reproducing individual training examples. Effective mitigation must preserve useful prompt information to guide alternative depictions. We introduce a training-free method that redistributes cross-attention with Gaussian smoothing before reinforcing content-token contributions and attenuating padding contributions, without additional denoiser evaluations. With this intervention, stronger content conditioning can improve prompt alignment at comparable training-image similarity. A local analysis identifies when reinforcement preserves shared value information while redistribution reduces localized attention mass. On Stable Diffusion v1.4 and v2.0, all evaluated smoothing widths lie on the empirical Pareto frontiers for training-image similarity versus both prompt alignment and image preference. A configuration selected on Stable Diffusion reduces template reproduction in DeepFloyd IF without further tuning. These findings support jointly controlling conditioning allocation and strength to generate prompt-consistent alternatives.
Diffusion models have achieved remarkable success in generative modeling, yet how to effectively adapt large pretrained models to new tasks remains challenging. We revisit the reconstruction behavior of diffusion models during denoising to unveil the underlying frequency energy mechanism governing this process. Building upon this observation, we propose FeRA, a frequency driven fine tuning framework that aligns parameter updates with the intrinsic frequency energy progression of diffusion. FeRA establishes a comprehensive frequency energy framework for effective diffusion adaptation fine tuning, comprising three synergistic components: (i) a compact frequency energy indicator that characterizes the latent bandwise energy distribution, (ii) a soft frequency router that adaptively fuses multiple frequency specific adapter experts, and (iii) a frequency energy consistency regularization that stabilizes diffusion optimization and ensures coherent adaptation across bands. Routing operates in both training and inference, with inference time routing dynamically determined by the latent frequency energy. It integrates seamlessly with adapter based tuning schemes and generalizes well across diffusion backbones and resolutions. By aligning adaptation with the frequency energy mechanism, FeRA provides a simple, stable, and compatible paradigm for effective and robust diffusion model adaptation.
Bo Yin, Xiaobin Hu, Xingyu Zhou +7
National University of Singapore · University of Electronic Science and Technology of China · Nanyang Technological University +4
Generating images beyond the training resolution of Diffusion Transformers (DiTs) remains challenging, as the number of tokens grows rapidly with resolution, full attention becomes expensive, and outputs often suffer from structural distortions and degraded detail. We identify uniform full attention as the common source of both problems. Attending to all tokens at every step causes quadratic cost and exposes the model to more tokens and relative distances beyond the training range, pushing the model out of distribution. We observe that attention's effective receptive field narrows during denoising, from global in early steps that form structure to local in later steps that refine details. We propose AReA, a training-free attention mechanism whose receptive field evolves from coarse to fine, following the denoising progression. In early steps, AReA applies cross-scale coarse attention over downsampled keys and values, capturing global context, while in later steps it applies local attention within progressively shrinking windows. Without additional parameters or training, experiments show that AReA consistently outperforms state-of-the-art training-free baselines across target resolutions and different models, while also being faster and more efficient.