cs.CVJul 2, 2025

Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective

Authors: Yuxin Mao, Zhen Qin, Jinxing Zhou, Hui Deng, Xuyang Shen, Bin Fan, Jing Zhang, Yiran Zhong, 

Organizations: Northwestern Polytechnic University · TapTap · Hefei University of Technology · OpenNLPLab · 

Abstract

Autoregressive (AR) models have recently emerged as a promising paradigm for image generation, as they can model visual tokens through the same next-token prediction principle that underlies large language models. However, most competitive AR image generators still rely on Transformer attention, whose quadratic token interaction cost and growing key-value cache become increasingly expensive for high-resolution visual sequences. Linear attention provides an attractive route to efficient AR generation, but our study shows that directly importing linear or decay-based attention from language modeling leads to clear quality degradation. We identify a key cause: after a 2D token map is flattened by raster scan, 1D sequence locality no longer faithfully reflects spatial locality, and conventional decay mechanisms may introduce a false-locality bias at row transitions. To address this problem, we propose Linear Attention with Spatial-Aware Decay (LASAD), a simple recurrent attention mechanism that modulates decay according to the original 2D token geometry while preserving linear complexity. Built upon this layer, LASADGen replaces standard attention in an AR image generator with spatial-aware linear attention. Experiments on ImageNet demonstrate that LASADGen achieves a favorable quality-efficiency tradeoff, improving generation quality over linear-attention baselines while reducing inference cost and memory usage compared with Transformer-based AR models.

Figures & tables

Explore similar work

CardsList
  1. Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generation

    Sep 17, 2026Daeun Kim, Junwha Hong, Changhun Oh +3Autoregressive Image GenerationSparse Attention

  2. FlashAR: Efficient Post-Training Acceleration for Autoregressive Image Generation

    May 10, 2026Junkang Zhou, Yefei He, Feng Chen +2Parallel DecodingAutoregressive Image Generation