Autoregressive (AR) models have recently emerged as a promising paradigm for image generation, as they can model visual tokens through the same next-token prediction principle that underlies large language models. However, most competitive AR image generators still rely on Transformer attention, whose quadratic token interaction cost and growing key-value cache become increasingly expensive for high-resolution visual sequences. Linear attention provides an attractive route to efficient AR generation, but our study shows that directly importing linear or decay-based attention from language modeling leads to clear quality degradation. We identify a key cause: after a 2D token map is flattened by raster scan, 1D sequence locality no longer faithfully reflects spatial locality, and conventional decay mechanisms may introduce a false-locality bias at row transitions. To address this problem, we propose Linear Attention with Spatial-Aware Decay (LASAD), a simple recurrent attention mechanism that modulates decay according to the original 2D token geometry while preserving linear complexity. Built upon this layer, LASADGen replaces standard attention in an AR image generator with spatial-aware linear attention. Experiments on ImageNet demonstrate that LASADGen achieves a favorable quality-efficiency tradeoff, improving generation quality over linear-attention baselines while reducing inference cost and memory usage compared with Transformer-based AR models.
Figures & tables
Figure 1: Visualization of the Spatial-Aware Decay mechanism . Each color block represents the decay factor of the token at the corresponding position. Our method masks row-boundary positions when raster-scan flattening creates artificial sequence neighbors, preventing the standard 1D decay rule from treating non-adjacent image tokens as ordinary local neighbors.
Model
Layers L
Hidden size d
Heads
#Para.
LASAD-B
12
768
12
111M
LASAD-L
24
1024
16
346M
LASAD-XL
36
1280
20
774M
LASAD-XXL
48
1536
24
1.4B
Table 1: Details of LASAD models. We follow the configurations of LlamaGen [ 51 ] and get different variants.
Size
Model
FID ↓
IS ↑
Precision ↑
Recall ↑
B
LinearAttn [ 17 ]
50.86
19.27
0.45
0.40
SoftmaxAttn
5.46
193.61
0.83
0.45
Hybrid-Linear
6.36
163.65
0.81
0.46
TNL [ 41 ]
6.75
161.92
0.80
0.46
GLA [ 58 ]
6.45
163.21
0.80
0.47
LASADGen
4.86
194.91
0.82
0.48
Table 2: Quantitative comparisons of different architectures for autoregressive image generation. “LinearAttn” denotes the original linear attention augmented with RoPE [ 50 ] . “Hybrid-Linear” refers to the hybrid architecture that integrates one softmax attention layer within the LinearAttn framework. “SoftmaxAttn” represents the conventional softmax-based attention mechanism. TNL and GLA represent two decay-based methods respectively.
Model
#Para.
FID ↓
IS ↑
Precision ↑
Recall ↑
VQGAN [ 9 ]
227M
18.65
80.4
0.78
0.26
VQGAN [ 9 ]
1.4B
15.78
74.3
−
−
VQGAN-re [ 9 ]
1.4B
5.20
280.3
−
−
ViT-VQGAN [ 61 ]
1.7B
4.17
175.1
−
−
ViT-VQGAN-re [ 61 ]
1.7B
3.04
227.4
−
−
RQTran. [ 20 ]
3.8B
7.55
134.0
−
−
Table 3: Class-conditional generation on 256×256 ImageNet. ∗ specifies the generated images are 384×384 and are resized to 256×256 for evaluation. “-re” denotes rejection sampling is used. † indicates the performance after only 50 epochs of training. The evaluation protocol and implementation are the same with ADM.
Figure 2: Autoregressive Image Generation with LASADGen. We show the samples from our class-conditional LASADGen-XL model trained on ImageNet at 256×256 resolution.
Figure 3: Training loss comparison between the model with or without the proposed Spatial-Aware Decay . We visualize the training process of the two groups of models in Table 6 . The blue curve is used to represent the model without using SAD, and the red curve is used to represent the model using SAD.
Model
B
L
XL
XXL
LlamaGen (vllm)
0.610
1.150
1.860
2.760
LASADGen
0.363
0.691
1.116
1.659
Table 4: Inference time (seconds) comparison across model scales.
Figure 4: Scaling properties of LASADGen. Left: FID scores and Right: Training loss vs. model size.
Model
Size
Tokens
FID ↓
IS ↑
Precision ↑
Recall ↑
LlamaGen-L
384 2
576
3.07
256.06
0.83
0.52
LASADGen-L
384 2
576
2.78
265.78
0.83
0.55
LlamaGen-L
256 2
1024
2.95
262.31
0.83
0.53
LASADGen-L
256 2
1024
2.71
271.45
0.83
0.56
Table 5: Long sequence scalability with different resolutions and sequence lengths.
Size
Model
SAD
FID ↓
IS ↑
Precision ↑
Recall ↑
B
GLA [ 58 ]
✗
6.45
163.21
0.80
0.47
GLA [ 58 ]
✓
5.84
173.00
0.80
0.48
LASADGen
✗
7.42
151.02
0.78
0.47
LASADGen
✓
4.86
194.91
0.82
0.48
L
LASADGen
✗
3.73
231.88
0.81
0.51
LASADGen
✓
2.90
255.61
0.82
0.55
Table 6: Quantitative comparisons of the model with or without the SAD.