Overview

Linear attention studies how to process long sequences without forming the full quadratic attention matrix, reducing computation or memory growth as context length increases. Main approaches include kernelized feature maps, recurrent gated and delta-rule memories, chunkwise parallel scans, and compact latent or addressable states; hybrids often retain softmax attention for selected tokens or layers. These methods are used in long-context language modeling and retrieval, as well as high-resolution vision, video, and time-series tasks. The central tension is whether compressed memory can preserve precise recall and expressive token interactions while remaining efficient, and current work explores better state updates, selective retrieval, and hardware-aware hybrids to close that gap.

Latest papers

CardsList

142 papers11 papers in the last four weeks, down 54% on the four weeks before. 0.1% of all new papers.

  1. The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale

    Nov 17, 2026Fırat Öncel, Cem Subakan, Mirco Ravanelli +1Efficient AttentionLinear Attention

  2. Bilateral Log-Linear Attention

    Nov 13, 2026Kazi Ashik Islam, Guojing CongMemory-Efficient AttentionLong-Context Language Modeling

  3. LiSA: Linear Stochastic Attention for Long-Range Visual Modeling

    Nov 3, 2026Sahar Dastani, Fereshteh Shakeri, Samuel Barbeau +4Linear AttentionEfficient ViTs

  4. Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models

    Nov 1, 2026Wenqi Zeng, Chen Feng, YuAng ChenKnowledge Retrieval in Language ModelsHybrid Language Models

  5. Solvable dynamical model of in-context learning

    Oct 31, 2026Ruoran Bai, Haiping HuangIn-Context LearningTransformer Training Dynamics

  6. DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention

    Oct 30, 2026Zhangchi Feng, Harry Jiang, Yangyifei Luo +2Long-Context RetrievalLinear Attention

  7. SparseGDN: Chunk Sparse in Linear Attention

    Oct 30, 2026Jiedong Zhuang, Peng Zhao, Jialin Tang +1LLM Inference AccelerationGated Delta Networks

  8. Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

    Oct 28, 2026Yixiao Qian, Yaxin Zhang, Song Chen +3Linear AttentionDelta-Rule Linear Attention

  9. Multi-State Linear Attention with Generalized Second-Order Routing

    Oct 27, 2026Hee Seung Hwang, Sanghyuk Chun, Xindi Wu +2Long-Context Language ModelingLinear Attention

  10. LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization

    Oct 13, 2026Lingyi Long, Congyue Zhao, Xiaoou Li +2Diffusion Model Inference AccelerationVideo Diffusion Models

  11. In-context classification: Calibration and noise in linear attention with softmax readout

    Oct 12, 2026Arman Rysmakhanov, Mary Letey, Yue M. Lu +2In-Context LearningLinear Attention

  12. Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry

    Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-AttentionLayer Normalization

  13. History Replays Itself: Unleashing Long-Context Capability in Linear Attention

    Oct 11, 2026Li Nanbo, Firas Laakom, Yilan Zhang +10Long-Context ModelingMemory-Augmented Language Models

  14. Preserving Updates in Low-Bit Recurrent State Quantization

    Oct 11, 2026Haotian Li, Liran Dong, Kaicheng Xiao +1Recurrent-State QuantizationLow-Bit Quantization

  15. Learnable Low-Rank Polynomial Sketch for Effective Linear Attention

    Oct 9, 2026Zongyue Qin, Haoran Deng, Jason Cong +1Transformer AttentionLinear Attention

  16. Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

    Oct 9, 2026Weiqi Sun, Qianyue Wu, Wenjia Wang +1Attention ApproximationEfficient Attention

  17. What can linear attention learn from nonlinear teachers in-context?

    Oct 7, 2026Mary Letey, Arman Rysmakhanov, Yue M. Lu +2In-Context LearningLinear Attention

  18. Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position

    Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language ModelingHybrid Attention

  19. PHBA: Prefix-State Hybrid Block Attention

    Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context ModelingLinear Attention

  20. HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

    Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language ModelingLinear Attention

  21. HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models

    Oct 5, 2026Zhuokun Chen, Feng Chen, Xi Lin +4World Model LearningLong-Horizon Video Generation

  22. Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention

    Oct 2, 2026Ruijie Li, Shengnan Ding, Weimin Zhang +6Linear AttentionDelta-Rule Linear Attention

  23. Switching Linear Attention

    Sep 30, 2026Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3Efficient AttentionLinear Attention

  24. Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

    Sep 29, 2026Oliver Sieberling, Bharat Runwal, David Jin +3Linear AttentionLong-Range Dependency Modeling

  25. CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

    Sep 28, 2026Yixiao Chen, Shuojin Yang, Shi-Min HuLinear RNNsLinear Attention