Autoregressive (AR) models have recently emerged as a promising paradigm for image generation, as they can model visual tokens through the same next-token prediction principle that underlies large language models. However, most competitive AR image generators still rely on Transformer attention, whose quadratic token interaction cost and growing key-value cache become increasingly expensive for high-resolution visual sequences. Linear attention provides an attractive route to efficient AR generation, but our study shows that directly importing linear or decay-based attention from language modeling leads to clear quality degradation. We identify a key cause: after a 2D token map is flattened by raster scan, 1D sequence locality no longer faithfully reflects spatial locality, and conventional decay mechanisms may introduce a false-locality bias at row transitions. To address this problem, we propose Linear Attention with Spatial-Aware Decay (LASAD), a simple recurrent attention mechanism that modulates decay according to the original 2D token geometry while preserving linear complexity. Built upon this layer, LASADGen replaces standard attention in an AR image generator with spatial-aware linear attention. Experiments on ImageNet demonstrate that LASADGen achieves a favorable quality-efficiency tradeoff, improving generation quality over linear-attention baselines while reducing inference cost and memory usage compared with Transformer-based AR models.
Figures & tables
Figure 1: Visualization of the Spatial-Aware Decay mechanism . Each color block represents the decay factor of the token at the corresponding position. Our method masks row-boundary positions when raster-scan flattening creates artificial sequence neighbors, preventing the standard 1D decay rule from treating non-adjacent image tokens as ordinary local neighbors.
Model
Layers L
Hidden size d
Heads
#Para.
LASAD-B
12
768
12
111M
LASAD-L
24
1024
16
346M
LASAD-XL
36
1280
20
774M
LASAD-XXL
48
1536
24
1.4B
Table 1: Details of LASAD models. We follow the configurations of LlamaGen [ 51 ] and get different variants.
Size
Model
FID ↓
IS ↑
Precision ↑
Recall ↑
B
LinearAttn [ 17 ]
50.86
19.27
0.45
0.40
SoftmaxAttn
5.46
193.61
0.83
0.45
Hybrid-Linear
6.36
163.65
0.81
0.46
TNL [ 41 ]
6.75
161.92
0.80
0.46
GLA [ 58 ]
6.45
163.21
0.80
0.47
LASADGen
4.86
194.91
0.82
0.48
Table 2: Quantitative comparisons of different architectures for autoregressive image generation. “LinearAttn” denotes the original linear attention augmented with RoPE [ 50 ] . “Hybrid-Linear” refers to the hybrid architecture that integrates one softmax attention layer within the LinearAttn framework. “SoftmaxAttn” represents the conventional softmax-based attention mechanism. TNL and GLA represent two decay-based methods respectively.
Model
#Para.
FID ↓
IS ↑
Precision ↑
Recall ↑
VQGAN [ 9 ]
227M
18.65
80.4
0.78
0.26
VQGAN [ 9 ]
1.4B
15.78
74.3
−
−
VQGAN-re [ 9 ]
1.4B
5.20
280.3
−
−
ViT-VQGAN [ 61 ]
1.7B
4.17
175.1
−
−
ViT-VQGAN-re [ 61 ]
1.7B
3.04
227.4
−
−
RQTran. [ 20 ]
3.8B
7.55
134.0
−
−
Table 3: Class-conditional generation on 256×256 ImageNet. ∗ specifies the generated images are 384×384 and are resized to 256×256 for evaluation. “-re” denotes rejection sampling is used. † indicates the performance after only 50 epochs of training. The evaluation protocol and implementation are the same with ADM.
Figure 2: Autoregressive Image Generation with LASADGen. We show the samples from our class-conditional LASADGen-XL model trained on ImageNet at 256×256 resolution.
Figure 3: Training loss comparison between the model with or without the proposed Spatial-Aware Decay . We visualize the training process of the two groups of models in Table 6 . The blue curve is used to represent the model without using SAD, and the red curve is used to represent the model using SAD.
Model
B
L
XL
XXL
LlamaGen (vllm)
0.610
1.150
1.860
2.760
LASADGen
0.363
0.691
1.116
1.659
Table 4: Inference time (seconds) comparison across model scales.
Figure 4: Scaling properties of LASADGen. Left: FID scores and Right: Training loss vs. model size.
Model
Size
Tokens
FID ↓
IS ↑
Precision ↑
Recall ↑
LlamaGen-L
384 2
576
3.07
256.06
0.83
0.52
LASADGen-L
384 2
576
2.78
265.78
0.83
0.55
LlamaGen-L
256 2
1024
2.95
262.31
0.83
0.53
LASADGen-L
256 2
1024
2.71
271.45
0.83
0.56
Table 5: Long sequence scalability with different resolutions and sequence lengths.
Size
Model
SAD
FID ↓
IS ↑
Precision ↑
Recall ↑
B
GLA [ 58 ]
✗
6.45
163.21
0.80
0.47
GLA [ 58 ]
✓
5.84
173.00
0.80
0.48
LASADGen
✗
7.42
151.02
0.78
0.47
LASADGen
✓
4.86
194.91
0.82
0.48
L
LASADGen
✗
3.73
231.88
0.81
0.51
LASADGen
✓
2.90
255.61
0.82
0.55
Table 6: Quantitative comparisons of the model with or without the SAD.
Autoregressive image generation has emerged as a paradigm for multimodal AI systems due to its compatibility with transformer-based LLM serving infrastructures. However, generating thousands of visual tokens per request makes decoding increasingly bottlenecked by KV cache accesses during attention computation. Sparse attention is particularly attractive for this workload because many visual generation applications tolerate moderate quality degradation in exchange for improved performance and efficiency. While sparse attention has been extensively explored for text-based LLM inference, it remains unclear whether its sparsity assumptions generalize effectively to autoregressive image generation. We present the first systematic characterization of attention sparsity in autoregressive image generation across diverse workloads and representative open-source models. Our analysis reveals several distinguishing properties, including a pronounced prefill-decode asymmetry, strong attention concentration on prompt and local tokens, and a unique diagonal attention sparsity pattern arising from the spatial locality of visual tokens. Motivated by these observations, we propose a diagonal-aware sparse attention mechanism that selectively skips KV entries along the diagonal attention direction within a recent window. Implemented on top of a GPU-based serving system using FlexGen, FlashAttention-2, and custom kernels, our approach achieves up to 3.1x throughput and 1.19x latency improvements with less than 2% quality degradation compared to dense inference.
Daeun Kim, Junwha Hong, Changhun Oh +3
KAIST · Agency for Defense Development · Seoul National University
Large-scale autoregressive models have demonstrated remarkable capabilities in image generation. However, their sequential raster-scan decoding relies on strictly next-token prediction, making inference prohibitively expensive. Existing acceleration methods typically either introduce entirely new generation paradigms that necessitate costly pre-training from scratch, or enable parallel generation at the expense of a training-inference gap or altered prediction objectives. In this paper, we introduce FlashAR, a lightweight post-training adaptation framework that efficiently adapts a pre-trained raster-scan autoregressive model into a highly parallel generator based on two-way next-token prediction. Our key insight is that effective adaptation should minimize modifications to the pre-trained model's original training objective to preserve its learned prior. Accordingly, we retain the original AR head as a horizontal head for row-wise prediction and introduce a complementary, lightweight vertical head for column-wise prediction. To facilitate efficient adaptation, we branch the vertical head from an intermediate layer rather than the final layer, bypassing the inherent horizontal head bias. Moreover, since horizontal and vertical predictions capture complementary dependencies whose relative importance varies across target positions, we employ a learnable fusion gate to dynamically combine the two predictions at each position. To further reduce adaptation cost, we propose a two-stage adaptation pipeline: the vertical head is first initialized through adaptation from the pre-trained autoregressive model before jointly fine-tuned with backbone to adapt to the new decoding paradigm. Extensive experiments on LlamaGen and Emu3.5 show that FlashAR achieves up to a 22.9x speedup for 512x512 image generation through a lightweight post-training with merely 0.05% of the original training data.
Junkang Zhou, Yefei He, Feng Chen +2
Zhejiang University, China · University of Adelaide, Australia
Autoregressive models excel in visual generation by treating images as 1D sequences of discrete tokens, mirroring language modeling. However, this flattening discards the intrinsic 2D spatial locality of visual signals, creating severe computational bottlenecks during inference. We introduce Spatially Speculative Decoding (SSD), a framework that aligns the predictive objective with the natural geometry of images. Rather than predicting only the immediate next token in a 1D sequence, our model simultaneously predicts the adjacent horizontal token and the token directly below it. By capitalizing on this 2D spatial correlation, spatially speculative decoding overcomes the memory wall in visual inference. Our approach accelerates autoregressive image generation by up to 13.3x while maintaining high fidelity on DPG-Bench and GenEval. Our results suggest that respecting the underlying geometry of vision unlocks massive computational efficiencies, paving the way for real-time, high-resolution autoregressive generative models.