Linear Attention

Follow this topic Ask about this topic

Linear attention studies how to process long sequences without forming the full quadratic attention matrix, reducing computation or memory growth as context length increases. Main approaches include kernelized feature maps, recurrent gated and delta-rule memories, chunkwise parallel scans, and compact latent or addressable states; hybrids often retain softmax attention for selected tokens or layers. These methods are used in long-context language modeling and retrieval, as well as high-resolution vision, video, and time-series tasks. The central tension is whether compressed memory can preserve precise recall and expressive token interactions while remaining efficient, and current work explores better state updates, selective retrieval, and hardware-aware hybrids to close that gap.

Latest papers

131 papers20 papers in the last four weeks, up 43% on the four weeks before. 0.1% of all new papers.

Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention
Oct 28, 2026Yixiao Qian, Yaxin Zhang, Song Chen +3Linear Attention Delta-Rule Linear Attention
College of Control Science and Engineering, Zhejiang University · Polytechnic Institute, Zhejiang University · Department of Mathematics, National University of Singapore +1
Multi-State Linear Attention with Generalized Second-Order Routing
Oct 27, 2026Hee Seung Hwang, Sanghyuk Chun, Xindi Wu +2Long-Context Language Modeling Linear Attention
Princeton University · Google DeepMind
LinMorph: Learning to Morph Weights for Efficient Video DiT Linearization
Oct 13, 2026Lingyi Long, Congyue Zhao, Xiaoou Li +2Diffusion Model Inference Acceleration Video Diffusion Models
Shanghai Jiao Tong University · VUI Labs Research
In-context classification: Calibration and noise in linear attention with softmax readout
Oct 12, 2026Arman Rysmakhanov, Mary Letey, Yue M. Lu +2In-Context Learning Linear Attention
Applied Mathematics, Harvard University · Williams College · Kempner Institute, Harvard University +1
Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry
Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-Attention Layer Normalization
New York University Abu Dhabi
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
Oct 11, 2026Li Nanbo, Firas Laakom, Yilan Zhang +10Long-Context Modeling Memory-Augmented Language Models
King Abdullah University of Science and Technology (KAUST)
Preserving Updates in Low-Bit Recurrent State Quantization
Oct 11, 2026Haotian Li, Liran Dong, Kaicheng Xiao +1Recurrent-State Quantization Low-Bit Quantization
The Chinese University of Hong Kong
When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs
Oct 10, 2026Yuhang JiangRecurrent-State Quantization LLM Quantization
Independent Researcher
Learnable Low-Rank Polynomial Sketch for Effective Linear Attention
Oct 9, 2026Zongyue Qin, Haoran Deng, Jason Cong +1Transformer Attention Linear Attention
Department of Computer Science, University of California Los Angeles, CA, USA
Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
Oct 9, 2026Weiqi Sun, Qianyue Wu, Wenjia Wang +1Attention Approximation Efficient Attention
The Chinese University of Hong Kong, Shenzhen · Beijing Normal University, Zhuhai · National University of Singapore
Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
Oct 8, 2026Kaicheng Xiao, Liran Dong, Haotian Li +1Memory-Augmented Neural Networks KV-Cache Quantization
The Chinese University of Hong Kong
What can linear attention learn from nonlinear teachers in-context?
Oct 7, 2026Mary Letey, Arman Rysmakhanov, Yue M. Lu +2In-Context Learning Linear Attention
Applied Mathematics, Harvard University Kempner Institute, Harvard University · Williams College · Center for Brain Science, Harvard University Society of Fellows, Harvard University
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language Modeling Hybrid Attention
Fudan University · Shanghai Innovation Institute · OpenMOSS Team
PHBA: Prefix-State Hybrid Block Attention
Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context Modeling Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Independent researcher
HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language Modeling Linear Attention
Monash University · Zhejiang University
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
Oct 5, 2026Zhuokun Chen, Feng Chen, Xi Lin +4World Model Learning Long-Horizon Video Generation
Monash University · The University of Adelaide · Zhejiang University
Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention
Oct 2, 2026Ruijie Li, Shengnan Ding, Weimin Zhang +6Linear Attention Delta-Rule Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
Switching Linear Attention
Sep 30, 2026Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3Efficient Attention Linear Attention
Stanford University · Unconventional AI. Work done at Stanford University.
Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling
Sep 29, 2026Oliver Sieberling, Bharat Runwal, David Jin +3Linear Attention Long-Range Dependency Modeling
Massachusetts Institute of Technology · MIT-IBM Computing Research Lab
CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory
Sep 28, 2026Yixiao Chen, Shuojin Yang, Shi-Min HuLinear RNNs Linear Attention
Tsinghua University
SMat-Attention: Structured Long-Context Sequence Modeling
Sep 28, 2026Emile Anand, Abdullah Ateyeh, Archer Wang +1Long-Context Modeling Linear Attention
Massachusetts Institute of Technology
MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
Sep 28, 2026Prasoon Dev, Anirudh Sankar, Vasudeva VarmaLong-Context Language Modeling Gated Attention
Language Technologies Research Center International Institute of Information Technology Hyderabad Hyderabad, Telangana, India
How Linear Attention Remembers
Sep 27, 2026Kichang Lee, JaeYeon Park, Songkuk Kim +1Linear Attention Memorization in Language Models
KAIST · Yonsei University · Dankook University
SketchSSM: Write to the Full State, Read from a Compact Sketch
Sep 27, 2026Omin Kwon, JoongWon Shin, Minseo Kim +3LLM Inference Acceleration Linear Attention
Seoul National University · UC Berkeley · KAIST
On-Policy Attention Linearization
Sep 25, 2026Arian Raje, Anupam Nayak, Anthony Fei +3Linear Attention Hybrid Transformer Architectures
Carnegie Mellon University · Cornell University
GTR: Gated Token Recurrence for Efficient Dense Prediction
Sep 22, 2026Zhe Feng, Longfei Liu, Wei Liu +7Efficient Neural Network Inference Gated Attention
Didi International Business Group · Intellindust AI Lab · Institute of Automation, Chinese Academy of Sciences +2
Rethinking Vision Architectures with Gated Linear Attention and KAN
Sep 18, 2026Ali Mehizel, Oussama KhaldiVisual Representation Learning Vision Transformer
Department of Applied Statistics, ENSSEA, Kolea, Tipaza, Algeria
Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Sep 17, 2026Haocheng Xi, Yiming Xie, Hexu Zhao +8Video Diffusion Models Streaming Video Generation
University of California, Berkeley · Impossible, Inc. · University of Texas at Austin
Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis
Sep 13, 2026Bhargav Lad, Yifan HaoFine-Tuning RAG Robustness
Department of Mathematics Indian Institute of Technology Delhi New Delhi, India · Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL 61801
SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
Sep 13, 2026Zian Liu, Yiwen Hu, Zican Dong +5Long-Context Language Modeling LLM Context Management
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Microsoft Research Asia

Oct 28, 2026J/K move · Enter open · S save

Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

Yixiao Qian, Yaxin Zhang, Song Chen, Jiaxu Liu+2

College of Control Science and Engineering, Zhejiang University · Polytechnic Institute, Zhejiang University · Department of Mathematics, National University of Singapore+1

Linear Attention Delta-Rule Linear Attention Language Modeling

Open paper AskPDF arXiv

Linear attention and state space models (SSMs) offer efficient alternatives to softmax attention by maintaining a fixed-size recurrent state, but this design requires careful management to preserve useful historical associations and limit interference. The delta rule helps reduce interference through correction of the value associated with each key, but it does not explicitly account for how related value features are distributed across the state. We observe that a state

\boldsymbol{S}\in\mathbb{R}^{d_k\times d_v}

can be viewed as a bank of

d_k

addresses storing

d_v

-dimensional values. While a key-side read

\boldsymbol{S}^\top\boldsymbol{k}

retrieves the value associated with

\boldsymbol{k}

, a value-side read

\boldsymbol{S}\boldsymbol{e}

aggregates addresses whose stored value features match the selector

\boldsymbol{e}

. Based on this observation, we propose Dual DeltaNet (DDN), which extends Kimi Delta Attention (KDA) with a two-sided delta rule. The key-side delta rule reads with the current key to correct its associated value, while the complementary value-side delta rule reads with selected features of the current value to reorganize the addresses it occupies. We derive a chunkwise parallel training algorithm and a rank-2 recurrence that increases arithmetic intensity during inference. Compared with KDA, GDN-2, and Mamba-3, DDN achieves the strongest overall performance across language modeling, zero-shot downstream tasks, and retrieval, while incurring modest training and inference overhead relative to KDA. Code: https://github.com/ReichtumQian/DualDeltaNet