Delta-Rule Linear Attention

Overview

Delta-rule linear attention studies sequence models that store key–value associations in a fixed-size matrix, updating it with prediction errors as new tokens arrive. Gated DeltaNet and Kimi Delta Attention add controls over forgetting and writing, while preconditioning, momentum, and query-aware updates aim to improve learning and recall; chunkwise WY algorithms make these recurrences parallelizable during training. Work evaluates language modeling, associative-recall and needle-in-a-haystack benchmarks, long-context generalization, and increasingly the memory and speed of model serving. A central tension is how to make recurrent memory more expressive and reliable without losing stable updates, efficient parallel computation, or the compact-state advantage.

Latest papers

CardsList

39 papers5 papers in the last four weeks, down 44% on the four weeks before. 0.0% of all new papers.

  1. Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout

    Nov 5, 2026Tushaar Gangavarapu, Gil Tabak, Clemens JS SchaeferDelta-Rule Linear AttentionSequence Modeling

  2. SparseGDN: Chunk Sparse in Linear Attention

    Oct 30, 2026Jiedong Zhuang, Peng Zhao, Jialin Tang +1LLM Inference AccelerationGated Delta Networks

  3. Dual DeltaNet: Value-Side Memory Reorganization for Linear Attention

    Oct 28, 2026Yixiao Qian, Yaxin Zhang, Song Chen +3Linear AttentionDelta-Rule Linear Attention

  4. Kronecker Delta Rule for Long Context Linear Attention

    Oct 19, 2026Anamaria-Roberta Hartl, Levente Zólyomi, Michael List +3Delta-Rule Linear AttentionAttention Mechanisms

  5. DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time

    Oct 10, 2026Todd Morrill, Richard ZemelNeural Network MemorizationPredictive Coding

  6. HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

    Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language ModelingLinear Attention

  7. Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention

    Oct 2, 2026Ruijie Li, Shengnan Ding, Weimin Zhang +6Linear AttentionDelta-Rule Linear Attention

  8. STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

    Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM QuantizationLLM Inference Acceleration

  9. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

    Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM QuantizationLLM Inference Acceleration

  10. Adapting Linear-Time Architectures for Tabular In-Context Learning

    Sep 28, 2026David Schnurr, Felix Sarnthein, Thomas Hofmann +1Tabular ICLTabular Foundation Models

  11. Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

    Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length GeneralizationDelta-Rule Linear Attention

  12. SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

    Sep 13, 2026Zian Liu, Yiwen Hu, Zican Dong +5Long-Context Language ModelingLLM Context Management

  13. Kalman Delta Networks: Uncertainty-aware Associative Memory

    Sep 7, 2026Ngoc Bui, Tinglin Huang, Rex YingMemory-Augmented Neural NetworksLinear Attention

  14. DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

    Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV CachingMemory-Efficient Inference

  15. Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

    Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Linear AttentionDelta-Rule Linear Attention

  16. The Key to Going Linear: Analysis-Driven Transformer Linearization

    Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiSoftmax AttentionLLM Inference Acceleration

  17. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Ilze Amanda Auzina +7Long-Context RetrievalMemory-Augmented Language Models

  18. Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

    Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Memory-Augmented Language ModelsLong-Context Language Modeling

  19. Q-Delta: Beyond Key-Value Associative State Evolution

    Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkLinear AttentionDelta-Rule Linear Attention