Delta-Rule Linear Attention

Follow this topic Ask about this topic

Delta-rule linear attention studies sequence models that store key–value associations in a fixed-size matrix, updating it with prediction errors as new tokens arrive. Gated DeltaNet and Kimi Delta Attention add controls over forgetting and writing, while preconditioning, momentum, and query-aware updates aim to improve learning and recall; chunkwise WY algorithms make these recurrences parallelizable during training. Work evaluates language modeling, associative-recall and needle-in-a-haystack benchmarks, long-context generalization, and increasingly the memory and speed of model serving. A central tension is how to make recurrent memory more expressive and reliable without losing stable updates, efficient parallel computation, or the compact-state advantage.

Latest papers

35 papers7 papers in the last four weeks, down 12% on the four weeks before. 0.0% of all new papers.

Kronecker Delta Rule for Long Context Linear Attention
Oct 19, 2026Anamaria-Roberta Hartl, Levente Zólyomi, Michael List +3Delta-Rule Linear Attention Attention Mechanisms
Institute for Machine Learning, Johannes Kepler University, Linz · NXAI GmbH, Linz
DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time
Oct 10, 2026Todd Morrill, Richard ZemelNeural Network Memorization Predictive Coding
Computer Science, Columbia University New York, NY, USA
HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language Modeling Linear Attention
Monash University · Zhejiang University
Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention
Oct 2, 2026Ruijie Li, Shengnan Ding, Weimin Zhang +6Linear Attention Delta-Rule Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM Quantization LLM Inference Acceleration
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
Adapting Linear-Time Architectures for Tabular In-Context Learning
Sep 28, 2026David Schnurr, Felix Sarnthein, Thomas Hofmann +1Tabular ICL Tabular Foundation Models
ETH Zürich · ELLIS Institute Tübingen · MPI-IS +1
DRAM: Delta-rule Recurrent Associative Memory for Robot Manipulation Policies
Sep 26, 2026Xinyu Zhao, Yixiang Shan, Tao Yang +5Long-Horizon Robotic Manipulation Delta-Rule Linear Attention
Simplexity Robotics
Constant-Memory Recall: Learned Associations in a Fixed Matrix State
Sep 23, 2026Samuel LarsonAssociative Memory Memory-Efficient Inference
Pebble ML
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length Generalization Delta-Rule Linear Attention
♡University of Freiburg · ♢Microsoft Research · ELLIS Institute Tübingen +9
SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization
Sep 13, 2026Zian Liu, Yiwen Hu, Zican Dong +5Long-Context Language Modeling LLM Context Management
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Microsoft Research Asia
Kalman Delta Networks: Uncertainty-aware Associative Memory
Sep 7, 2026Ngoc Bui, Tinglin Huang, Rex YingMemory-Augmented Neural Networks Linear Attention
Department of Computer Science, Yale University
Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate
Sep 6, 2026Vishwajith RameshLLM Unlearning Machine Unlearning Evaluation
Vy Labs, Inc.
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Sep 3, 2026Sergii Kozyrev, Davyd MaiborodaLLM Quantization 4-Bit Quantization
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV Caching Memory-Efficient Inference
East China Normal University · Meituan · South China University of Technology
TreeWY: Speculative Verification for Gated DeltaNet Hybrids
Aug 21, 2026Sneha Murthy GhantasalaSpeculative Decoding LLM Inference Acceleration
Thomson Reuters
The Query Knows What to Forget: A Second Erase Direction for Linear Attention
Aug 13, 2026Dhruman Gupta, Aritra Das, Debayan GuptaDelta-Rule Linear Attention Long-Context Retrieval
Truth Audit Labs
Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Aug 3, 2026Ronglong BaoDelta-Rule Linear Attention Language Model Distillation
DT-Project
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Linear Attention Delta-Rule Linear Attention
1ETH Zurich, D-INFK · 2ETH AI Center, ETH Zurich
The Key to Going Linear: Analysis-Driven Transformer Linearization
Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiSoftmax Attention LLM Inference Acceleration
Qualcomm AI Research∗ · Qualcomm AI Research
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Long-Context Retrieval Memory-Augmented Language Models
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
Jul 2, 2026Wanyun CuiHybrid Attention Long-Context Retrieval
Shanghai University of Finance and Economics
CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
Jun 25, 2026Sayak DuttaLinear Attention Delta-Rule Linear Attention
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Memory-Augmented Language Models Long-Context Language Modeling
1Qwen Team · 2Nanjing University · 3Zhejiang University
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
Jun 12, 2026Tiantian ZhangEfficient Attention Linear Attention
Columbia University
Q-Delta: Beyond Key-Value Associative State Evolution
Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkLinear Attention Delta-Rule Linear Attention
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers
May 20, 2026Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros +4Efficient Transformer Inference AI Accelerator Inference
Computing Systems Lab Huawei Technologies2026 Switzerland
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Transformer Attention Linear Attention
Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China

Oct 19, 2026J/K move · Enter open · S save

Kronecker Delta Rule for Long Context Linear Attention

Anamaria-Roberta Hartl, Levente Zólyomi, Michael List, Günter Klambauer+2

Institute for Machine Learning, Johannes Kepler University, Linz · NXAI GmbH, Linz

Delta-Rule Linear Attention Attention Mechanisms Long-Context Language Modeling Associative Memory

Open paper AskPDF arXiv

We propose KronLSTM, a linear attention architecture based on the Kronecker Delta Rule, a novel update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism that corrects targeted associations while modifying the memory only along a single direction in matrix space. This minimally invasive memory revision reduces interference with previously stored information compared with delta-rule updates, making it particularly well suited for long-context language modeling. The resulting memory dynamics can be mathematically described as a Markov chain that converges to a steady-state distribution, while the expected norm of the memory matrix remains uniformly bounded throughout the entire input sequence. Hence, KronLSTM ensures stable memory dynamics without requiring additional normalization and stabilization mechanisms for the memory state. Experiments show that KronLSTM is competitive with, and often outperforms, existing linear attention architectures, including models of the (Gated) DeltaNet family, with particularly strong performance on long context benchmarks. These results establish the Kronecker Delta Rule as an effective and analytically tractable foundation for stable associative memory in recurrent linear attention architectures. Our code (https://github.com/NX-AI/KronLSTM) and models (https://huggingface.co/collections/NX-AI/kronlstm) are open-source.