Linear Attention

Follow this topic Ask about this topic

Linear attention studies how to process long sequences without forming the full quadratic attention matrix, reducing computation or memory growth as context length increases. Main approaches include kernelized feature maps, recurrent gated and delta-rule memories, chunkwise parallel scans, and compact latent or addressable states; hybrids often retain softmax attention for selected tokens or layers. These methods are used in long-context language modeling and retrieval, as well as high-resolution vision, video, and time-series tasks. The central tension is whether compressed memory can preserve precise recall and expressive token interactions while remaining efficient, and current work explores better state updates, selective retrieval, and hardware-aware hybrids to close that gap.

Latest papers

131 papers20 papers in the last four weeks, up 43% on the four weeks before. 0.1% of all new papers.

STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4LLM Inference Acceleration Linear Attention
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Dec 14, 2025Jingdi Lei, Di Zhang, Soujanya PoriaDynamical Systems Linear Attention
Nanyang Technological University · Fudan University
Selective Rotary Position Embedding
Nov 21, 2025Sajad Movahedi, Timur Carstensen, Arshia Afzal +3Softmax Attention Rotary Positional Embeddings
ELLIS Institute Tübingen1, LIONS, EPFL2, University of Freiburg3, Max-Planck-Institute for Intelligent Systems4, Prior Labs5
Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Calibration
Nov 5, 2025Mingtao Zhang, Guoli Yang, Zhanxing Zhu +2Linear Attention Multivariate Time Series Forecasting
Advanced Institute of Big Data, Beijing · University of Southampton
LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling
Oct 21, 2025Yang Liu, Yunjiang Jiang, Ayush Agarwal +3Recommender Systems Efficient Inference
Meta AI
MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching
Oct 16, 2025Tingman Yan, Tao Liu, Chenghao Li +4Image Matching Stereo Matching
School of Control Science and Engineering, Dalian University of Technology, Dalian, 116024, China · Pika Labs · Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China +1
Neural Architecture Discovery via Autonomous Evolution
Jul 24, 2025Weixian Xu, Yixiu Liu, Yang Nan +4Autonomous Research Agents AI Agents for Scientific Discovery
Shanghai Jiao Tong University · SII · GAIR +1
A Systematic Analysis of Hybrid Linear Attention
Jul 8, 2025Dustin Wang, Rui-Jie Zhu, Steven Abreu +9Linear Attention Hybrid Attention
1UC Santa Cruz · 2ByteDance Seed · University of Groningen +3
SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging
Jun 10, 2025Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4Self-Attention Vision Transformer
Department of Mathematics; University of California, Irvine; Irvine, CA 92697, USA · Qualcomm AI Research; San Diego, CA 92121, USA
Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention
May 22, 2025Yuang AiEfficient ViTs Linear Attention
University of Chinese Academy of Sciences, Beijing, China
Efficient Concertormer for Image Deblurring and Beyond
Apr 9, 2024Pin-Hung Kuo, Jinshan Pan, Shao-Yi Chien +1Image Deblurring Transformer Attention

Feb 2, 2026J/K move · Enter open · S save

STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs

Weikang Meng, Liangyu Huo, Yadan Luo, Jiawen Guan+3

SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland+1

LLM Inference Acceleration Linear Attention Attention Mechanisms Hybrid Attention

Open paper AskPDF arXiv

Linearizing pretrained large language models (LLMs) primarily relies on intra-layer hybrid attention mechanisms to alleviate the quadratic complexity of standard softmax attention. Existing methods perform token routing based on sliding-window partitions, resulting in position-based selection and fails to capture token-specific global importance. Meanwhile, linear attention further suffers from distribution shift caused by learnable feature maps that distort pretrained feature magnitudes. Motivated by these limitations, we propose STILL, an intra-layer hybrid linearization framework for efficiently linearizing LLMs. STILL introduces a Self-Saliency Score with strong local-global consistency, enabling accurate token selection using sliding-window computation, and retains salient tokens for sparse softmax attention while summarizing the remaining context via linear attention. To preserve pretrained representations, we design a Norm-Preserved Feature Map (NP-Map) that decouples feature direction from magnitude and reinjects pretrained norms. We further adopt a unified training-inference architecture with chunk-wise parallelization and delayed selection to improve hardware efficiency. Experiments show that STILL matches or surpasses the original pretrained model on commonsense and general reasoning tasks, and achieves up to a 86.2% relative improvement over prior linearized attention methods on long-context benchmarks. Code is available at this URL.