SPARE: Fast Linear Attention Decoding with Compact Recurrent States
Organizations: University of Virginia
Abstract
Linear attention reduces the quadratic sequence-length complexity of softmax attention by maintaining fixed-size recurrent states. However, these states scale quadratically with the representation dimension and require repeated data transfers during decoding, creating a substantial GPU memory bottleneck. Recent studies have revealed significant low-rank redundancy in recurrent states, suggesting opportunities for more efficient representations. We introduce SPARE (State Projection for Accelerated Recurrent Execution), a training-free method that exploits this redundancy through fixed low-dimensional subspaces, heterogeneous rank allocation, and lightweight write compensation. By performing recurrent reads and updates directly on compact states, SPARE reduces memory traffic without online decomposition or dense-state reconstruction. SPARE achieves 2.55x full-model decoding speedup, 7.49x lower recurrent-kernel DRAM traffic, and 3.31x lower peak GPU memory, while preserving better associative-retrieval accuracy than structured pruning at matched memory budgets.
Figures & tables
| Language Modeling | Associative Retrieval | Downstream Tasks | |||||
|---|---|---|---|---|---|---|---|
| Method | State (MiB) | NLL | Acc. (%) | SQuAD | TriviaQA | DROP | Avg. |
| Dense | 48 | 2.290 | 70.51 | 36.92 | 64.28 | 21.85 | 41.02 |
| DRRQR-K32 | 6 | 3.068 | 23.44 | 9.91 | 32.29 | 8.15 | 16.78 |
| SPARE-H32 | 6 | 2.817 | 56.84 | 28.62 | 37.68 | 12.75 | 26.35 |
| DRRQR-K64 | 12 | 2.653 | 33.20 | 21.90 | 50.12 | 15.09 | 29.04 |
| SPARE-H64 | 12 | 2.534 | 60.35 | 30.74 | 48.40 | 18.11 | 32.42 |
| Method | State (MiB) | NLL | Acc. (%) |
|---|---|---|---|
| Uniform32 | 6 | 2.798 | 33.59 |
| SPARE-H32 | 6 | 2.817 | 56.84 |
| Uniform48 | 9 | 2.658 | 40.63 |
| SPARE-H48 | 9 | 2.680 | 58.40 |
| Uniform64 | 12 | 2.574 | 46.09 |
| SPARE-H64 | 12 | 2.534 | 60.35 |
Memory-Efficient Inference
Memory-Efficient Inference reduces the memory needed to run neural networks while balancing accuracy, latency, throughput, and energy use.
See all 181 papers