Generating images beyond the training resolution of Diffusion Transformers (DiTs) remains challenging, as the number of tokens grows rapidly with resolution, full attention becomes expensive, and outputs often suffer from structural distortions and degraded detail. We identify uniform full attention as the common source of both problems. Attending to all tokens at every step causes quadratic cost and exposes the model to more tokens and relative distances beyond the training range, pushing the model out of distribution. We observe that attention's effective receptive field narrows during denoising, from global in early steps that form structure to local in later steps that refine details. We propose AReA, a training-free attention mechanism whose receptive field evolves from coarse to fine, following the denoising progression. In early steps, AReA applies cross-scale coarse attention over downsampled keys and values, capturing global context, while in later steps it applies local attention within progressively shrinking windows. Without additional parameters or training, experiments show that AReA consistently outperforms state-of-the-art training-free baselines across target resolutions and different models, while also being faster and more efficient.
Figures & tables
Figure 1: High-Resolution Generation. AReA enables training-free and efficient high resolution generation across FLUX.1, Qwen-Image, FLUX.2, and Ideogram 4. Best viewed zoomed-in.
Figure 2: Failure Modes at High Resolution . FLUX.1 at 4096 × 4096. Full attention keeps the model’s intended subject but collapses the layout, while RoPE-based methods keep a layout but fragment, lose, or distort the subject. AReA preserves both and renders fine details.
Figure 3: Attention Locality as Denoising Progresses. Vanilla FLUX.1 ( 1024×1024 ), averaged across heads, blocks, and 24 prompts. Left : attention-weighted query–key distance. Over all keys it shrinks only 1.4× , but over dominant keys ( ≥1/4 max attention) it shrinks 5.5× : late in denoising, the most important keys lie close to the query. Right : attention maps of a center query at early, middle, and late steps (top 60% of mass).
Figure 4: Pipeline of AReA. AReA follows the coarse-to-fine progression of denoising, switching from Phase I to Phase II at noise level σ⋆ . In Phase I, full-resolution queries attend to keys and values pooled to one token per f×f block, with f ramped over steps, and positioned on a grid matching the training resolution. In Phase II, attention is restricted to progressively shrinking local windows, and those over smooth regions of the DWT map are further split to reduce cost.
Figure 5: Qualitative comparisons across resolutions. AReA eliminates structural distortions and subject duplication seen in baseline methods, achieving superior fidelity and prompt alignment.
Figure 6: Generation latency vs. resolution. AReA is significantly faster than full-attention baselines across all resolutions, reaching 3.2× and 2.9× acceleration at 61442 on FLUX.1 and Qwen-Image, respectively.
FLUX.1
Qwen-Image
Res.
Method
Latency(s) ↓
ImageReward ↑
Aesthetic ↑
CLIP ↑
PickScore ↑
FID ↓
FID p↓
Latency(s) ↓
ImageReward ↑
Aesthetic ↑
CLIP ↑
PickScore ↑
FID ↓
FID p↓
2048 × 4096
Base
97
-0.8696
5.91
26.49
20.41
171.4
67.4
274
-0.5186
5.96
27.25
20.83
167.6
57.8
UltraImage
89
-0.1474
6.25
28.11
21.28
173.3
79.0
301
0.4448
5.79
28.19
21.07
161.7
62.6
DyPE
97
0.5927
6.30
28.43
22.15
156.5
79.9
274
0.5769
6.12
28.71
21.64
159.0
74.7
SigMa
103
0.6727
6.14
28.07
22.01
152.5
63.9
275
0.6155
6.22
28.87
22.03
152.5
63.0
SEGA
98
0.7863
6.28
28.95
22.47
153.9
76.1
276
1.2467
6.42
29.89
23.10
146.1
69.6
Table 1: Comparison of AReA against state-of-the-art baselines on FLUX.1 and Qwen-Image across different resolutions and aspect ratios on Aesthetic-4K. Bold : best; underline : second-best.
Method
IR ↑
Aes. ↑
CLIP ↑
Pick ↑
HPS ↑
IS ↑
Base
-1.4649
5.12
23.36
19.04
0.232
7.20
UltraImage
-0.0277
5.77
28.26
20.46
0.269
10.17
DyPE
0.3280
5.83
29.35
20.94
0.266
10.93
SigMa
0.0627
5.63
27.91
20.54
0.258
10.70
SEGA
0.4422
5.82
29.63
21.18
0.270
11.99
AReA
0.6001
5.87
29.89
21.44
0.275
13.90
Table 2: Comparison on FLUX.1 at 4096×4096 on HRGen-1K. Bold : best; underline : second-best.
Figure 7: Qualitative ablation on FLUX.1 at 4096×4096 . Without Phase I, windows diverge and disrupt global layout. Without Phase II, the image lacks fine texture and remains blurry. Mean pooling degrades overall image quality (compared in Table 4 ), while fixed windows without shrinking cause seams and content fragmentation compared to default AReA.
Method
Latency(s) ↓
ImageReward ↑
Aesthetic ↑
MUSIQ ↑
PickScore ↑
HPS ↑
CLIP ↑
FID ↓
Phase I
No Phase I (full attention)
201
-0.41
5.85
38.06
20.73
0.264
26.63
171.42
Flat f=2
165
1.23
6.27
46.67
23.03
0.294
29.13
152.08
Flat f=4 (no ramp)
142
1.27
6.38
44.72
23.17
0.292
29.63
154.21
Phase II
No Phase II (full attention)
248
1.10
6.35
40.67
22.71
0.286
28.78
155.44
Fixed windows ( w=64 )
123
1.26
6.43
45.26
23.12
0.296
29.78
152.15
w/o Global-KV bridge
133
1.32
6.25
45.38
23.25
0.294
29.11
161.25
Table 3: Ablation study on FLUX.1 at 4096×4096 on Aesthetic-4K. Within each phase, the first row replaces the whole phase with full attention; the rows below change one component.
Pooling
IR ↑
Aes. ↑
MUSIQ ↑
Pick ↑
HPS ↑
CLIP ↑
FID ↓
Mean
1.23
6.30
44.93
22.84
0.289
29.28
152.78
Random
1.25
6.35
45.59
23.22
0.293
29.37
152.53
Medoid
1.26
6.31
44.04
23.19
0.293
29.19
151.82
Center †
1.32
6.37
45.22
23.31
0.297
29.71
148.50
Table 4: Phase I pooling strategy on FLUX.1 ( 4096×4096 , Aesthetic-4K). † Default.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Effect of the Phase I pooling schedule on FLUX.1 at 4096×4096 . A fixed f=4 forms the layout on a 64×64 pooled grid, which matches the training resolution but is too coarse to separate objects, so the sheep merge into a single blurred mass. A fixed f=2 forms the layout on a 128×128 grid, twice the training resolution, producing scattered and repeated subjects. Decreasing f from 4 to 2 first sets the layout at the training resolution and then refines it on the finer grid, giving separated subjects and a coherent scene.
Figure 9: Effect of the Phase I pooling strategy on FLUX.1 at 4096×4096 . Mean pooling averages away the components that distinguish keys within a block, blurring fine detail such as armor ornamentation and facial features. Medoid selection keeps a real token but favors the most typical one in each block, partially inheriting this loss. Random selection preserves more detail but depends on the seed. Center pooling (default) uses a fixed, content-free token per block and gives the sharpest and most coherent results.
Figure 10: Effect of switching phases too early or too late. An early switch starts Phase II before the structure has settled, degrading the final result. A late switch leaves too few Phase II steps to add fine details, so the final image lacks detail.
Too early
Stable
Too late
Switch step s ( σ )
2 (0.98)
4 (0.95)
6 (0.92)
8 (0.89)
10 (0.85)
12 (0.81)
14 (0.76)
16 (0.70)
18 (0.64)
20 (0.56)
22 (0.46)
Quality
ImageReward ↑
0.77
1.22
1.18
1.25
1.24
1.25
1.20
1.22
1.21
1.19
1.18
Aesthetic ↑
6.03
6.20
6.22
6.23
6.15
6.16
6.21
6.20
6.18
6.16
6.08
CLIP-IQA ↑
0.685
0.712
0.705
0.705
0.696
0.686
0.671
0.658
0.628
0.579
0.528
Alignment
Appendix
Table 5: Ablation of the phase switch step s (Phase I → Phase II) on FLUX.1 at 4096 × 4096 resolution on 20 prompts from Aesthetic-4K. The noise level σ at the switch is given in parentheses.
Figure 11: Window size follows the locality of the model’s attention. AReA’s window size at 40962 and the model’s attention range at its native resolution of 10242 , computed over keys receiving at least a quarter of the maximum attention (blue) and over all keys (orange). All curves are shown relative to their values at the start of Phase II. In Phase I, every query attends to pooled keys from the whole image, so no window is shown. In Phase II, the window size follows a similar trend to the strongest attention, while attention over all keys remains broad.
Figure 12: AReA Across Denoising on FLUX.1 at 4096 × 4096 . Top row : Phase I steps with pooled keys and values (insets show the pixelation effect). Other rows : Phase II steps, each showing the current image (upper half) and its DWT detail map (lower half), with base windows in red, overlap regions hatched, and DWT splits in orange.
Figure 13: Effect of the Global-KV Bridge and Halo . Left : without the global-KV bridge, each window is blind to content elsewhere on the canvas and paints extra copies of the subject. With the bridge, windows see distinctive tokens from the rest of the grid and the duplicates disappear. Right : without the Halo, tokens at window edges lack context from across the boundary, leaving seams in smooth regions such as the sky.
Figure 14: Key budget per denoising step. Number of image keys each query attends to on FLUX.1 at 4096×4096 , averaged over 20 prompts (shading shows the range). Top : In Phase I, the pooled keys match the token count of the native 10242 resolution for f=4 and four times that for f=2 . In Phase II, the windows shrink from about 32K keys to the native 4K, and full AReA stays within about 30% of plain windowing. No query attends to more than about 40K of the 64K image keys at any step. Bottom : Change in keys per query relative to plain windows. The global-KV bridge adds a roughly constant share of about 25%, the halo takes a growing share as windows shrink, and DWT splits remove up to 35% late in Phase II.
Figure 15: Comparison with Multi-Stage Methods on FLUX.1 at 4096×4096 . Multi-stage methods and AReA-Bootstrap inherit the composition of the same base-resolution image and differ mainly in local detail. AReA with Phase I (default) sets its own layout at the target resolution and renders finer detail (green insets). ScaleDiff introduces repetitive content in low-detail regions (red insets).
Method
Latency (s) ↓
ImageReward ↑
Aesthetic ↑
MUSIQ ↑
CLIP-IQA ↑
ARNIQA ↑
CLIP ↑
HPS ↑
FID ↓
FID p↓
HiFlow
182
1.1710
6.3147
38.674
0.6561
0.4811
29.3
0.2951
153.0
59.0
I-Max
440
1.1716
6.3043
41.086
0.6838
0.5013
29.2
0.2956
154.8
60.1
ScaleDiff
135
1.2339
6.2785
44.496
0.7066
0.5663
29.2
0.2935
152.5
73.7
AReA-Bootstrap
126
1.2658
6.3402
43.224
0.7027
0.5479
29.4
0.2955
152.0
61.2
AReA
151
1.3250
6.3717
45.217
0.7132
0.5942
29.7
0.2969
148.5
58.2
Appendix
Table 6: Comparison with multi-stage methods on FLUX.1 at 4096×4096 on Aesthetic-4K. Bold : best; underline : second-best.
Figure 16: Out-of-Distribution Exposure and Its Consequence on FLUX.1. Left : distance and token-count exposure at 40962 , averaged over denoising and scaled so that 0 is the model at its native 10242 and 1 is the vanilla model at 40962 . RoPE-modifying methods (like SEGA) remove only the distance cause, while AReA reduces both. Right : FID of each method’s outputs to the model’s own native-resolution outputs; lower is closer to the model’s intended and trained distribution. AReA stays lowest and flat as resolution grows.
Figure 17: Statistics of HRGen-1K . left : Distribution of prompt lengths, grouped into short ( ≤15 words), medium (16-40 words), and long ( >40 words) prompts. Right : Share of prompts containing common attributes, estimated via keyword matching. Attributes are not mutually exclusive.
Figure 18: Additional Comparison on FLUX.1 at 4096×4096 on HRGen-1K . Existing methods warp scenes into symmetric, fisheye-like layouts, reduce object-centric prompts to zoomed-in texture, miscount objects, and garble rendered text. AReA preserves scene composition, object counts (highlighted in red), and legible text.
Figure 19: Comparison on Qwen-Image at Non-Square Resolutions . Top : 2048×4096 . Bottom : 6144×2048 . Full attention misplaces the subject on the tall canvas and duplicates it on the wide one. DyPE stretches the subject and introduces artifacts, and SEGA duplicates the subject in the wide setting. AReA keeps a single, well-proportioned subject in both aspect ratios.
Figure 20: Comparison on Qwen-Image at 5120×5120 . Full attention blurs the scene and repeats subjects. SEGA keeps the layout but corrupts the details with high-frequency artifacts. AReA produces clean faces and the prompted number of subjects.
Figure 21: Comparison on Ideogram 4 at 6144×6144 . Full attention completely collapses, while SEGA fails to keep the layout and add fine detail. AReA excels in both structure and fine detail, showing that it transfers to a different backbone without modification.
Diffusion Transformers use attention-based networks to generate and transform images, video, audio, and other structured data through iterative denoising.
Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range. Existing training-free approaches mitigate this by modifying inference-time attention behavior, often through Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling. However, these strategies apply a uniform and content-agnostic scaling across RoPE components with distinct frequency characteristics, inducing a trade-off between preserving global structure and recovering fine detail. We introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent's spatial-frequency structure at each denoising step. This adaptive scaling improves both structural coherence and fine-detail fidelity. Experiments show that SEGA consistently improves high-resolution synthesis across multiple target resolutions, outperforming state-of-the-art training-free baselines.
Diffusion Transformers (DiTs) are a dominant backbone for high-fidelity text-to-image generation due to strong scalability and alignment at high resolutions. However, quadratic self-attention over dense spatial tokens leads to high inference latency and limits deployment. We observe that denoising is spatially non-uniform with respect to aesthetic descriptors in the prompt. Regions associated with aesthetic tokens receive concentrated cross-attention and show larger temporal variation, while low-affinity regions evolve smoothly with redundant computation. Based on this insight, we propose AccelAes, a training-free framework that accelerates DiTs through aesthetics-aware spatio-temporal reduction while improving perceptual aesthetics. AccelAes builds AesMask, a one-shot aesthetic focus mask derived from prompt semantics and cross-attention signals. When localized computation is feasible, SkipSparse reallocates computation and guidance to masked regions. We further reduce temporal redundancy using a lightweight step-level prediction cache that periodically replaces full Transformer evaluations. Experiments on representative DiT families show consistent acceleration and improved aesthetics-oriented quality. On Lumina-Next, AccelAes achieves a 2.11× speedup and improves ImageReward by +11.9% over the dense baseline. Code is available at https://github.com/xuanhuayin/AccelAes.
Xuanhua Yin, Chuanzhi Xu, Haoxian Zhou +2
School of Computer Science, The University of Sydney, NSW 2006, Australia
Denoising diffusion models are the dominant architecture for image generation, whereas most natural language generation and modeling are primarily handled by well-known transformer architectures employing attention mechanism. Here, we show that diffusion models also inherently use an attention mechanism very similar to that of transformers. Therefore, attention emerges as a universal machine learning principle, based on a general training objective. We also show similarities in basic functional principle of auto-encoders and attention-based models. These equivalences allows us to interchange these designs based on practical requirements. As an example, we can reformulate the diffusion framework to reduce the lengthy training process and computation-intensive image generation. Using this approach, a simplified algorithm is proposed for image generation which is based on attention mechanism. Results show that the attention-based implementation achieves comparable performance with significantly less effort and computational resources.
Farzan Haddadi, Leila Monfared, Ebrahim Rezaii +3
School of Electrical Engineering, Iran University of Science & Technology, Tehran, IRAN · Independent researcher