cs.CLNov 5, 2026

SPARE: Fast Linear Attention Decoding with Compact Recurrent States

Authors: Ke Wan, Zaiyi Zheng, Chen Chen

Organizations: University of Virginia

Abstract

Linear attention reduces the quadratic sequence-length complexity of softmax attention by maintaining fixed-size recurrent states. However, these states scale quadratically with the representation dimension and require repeated data transfers during decoding, creating a substantial GPU memory bottleneck. Recent studies have revealed significant low-rank redundancy in recurrent states, suggesting opportunities for more efficient representations. We introduce SPARE (State Projection for Accelerated Recurrent Execution), a training-free method that exploits this redundancy through fixed low-dimensional subspaces, heterogeneous rank allocation, and lightweight write compensation. By performing recurrent reads and updates directly on compact states, SPARE reduces memory traffic without online decomposition or dense-state reconstruction. SPARE achieves 2.55x full-model decoding speedup, 7.49x lower recurrent-kernel DRAM traffic, and 3.31x lower peak GPU memory, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets.

Figures & tables

Explore similar work

CardsList
  1. DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

    Aug 16, 2026Junqing Lin, Jingwei Sun, Guangzhong SunLLM Inference AccelerationLinear Attention

  2. SpecLA: Efficient Speculative Decoding for Linear-Attention Models

    Jul 18, 2026Zhibin Wang, Xuying Han, Zhaohua Yang +5Speculative DecodingLLM Inference Acceleration

  3. SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving

    Nov 10, 2026Haolin Jiang, Weihang Li, Zhongzhu Zhou +3LLM Inference AccelerationMemory-Efficient Inference