Overview

Gated Delta Networks process sequences by repeatedly updating a fixed-size matrix state: delta-rule updates write new information while gates regulate what is retained or erased. Chunkwise and WY-style algorithms parallelize training, while recurrent decoding offers linear sequence cost and constant-size state instead of growing a full attention cache. Current work explores finer-grained erase and write controls, normalized update steps, larger or structured memories, and hybrids with attention; evaluations often focus on language modeling, associative recall, and long-context retrieval benchmarks such as needle-in-a-haystack tasks. A central challenge is balancing memory capacity and retrieval accuracy against efficiency, while keeping recurrent states numerically stable and reliable beyond the training context length.

Latest papers

CardsList

25 papers4 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

  1. The State Is Not the Bottleneck: A Paged-State Gated-DeltaNet Tile on a Tri-Mode Multicast Mesh, Bit-Exact from RTL to Silicon

    Nov 13, 2026Cong Xu, Ravi SankarLLM Inference AccelerationFPGA-Based Neural Network Acceleration

  2. DAMI: Dual-Axis Memory Initialization for Enhanced Long-Context Retrieval in Gated Linear Attention

    Oct 30, 2026Zhangchi Feng, Harry Jiang, Yangyifei Luo +2Long-Context RetrievalLinear Attention

  3. SparseGDN: Chunk Sparse in Linear Attention

    Oct 30, 2026Jiedong Zhuang, Peng Zhao, Jialin Tang +1LLM Inference AccelerationGated Delta Networks

  4. Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory

    Oct 5, 2026Parsa Hejabi, Morteza Dehghani, Payam PirayKalman FilteringBayesian Filtering

  5. Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention

    Oct 2, 2026Ruijie Li, Shengnan Ding, Weimin Zhang +6Linear AttentionDelta-Rule Linear Attention

  6. Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

    Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length GeneralizationDelta-Rule Linear Attention

  7. Visual Representation and History Modeling for Navigation World Models

    Aug 26, 2026Guangfu Guo, Xiaoqian Lu, Rui Liu +3World ModelsWorld Model Learning

  8. SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

    Aug 23, 2026Qingwen Lin, Boyan Xu, Xiao Liu +2Recurrent Neural NetworksGated Delta Networks

  9. DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows

    Aug 2, 2026Guangfu Guo, Xiaoqian Lu, Linsey Pang +4Gated Delta NetworksDiffusion Language Models

  10. Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

    Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Linear AttentionDelta-Rule Linear Attention

  11. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Ilze Amanda Auzina +7Long-Context RetrievalMemory-Augmented Language Models

  12. Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

    Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network InitializationLinear Attention

  13. SinkRec: Mitigating Semantic State Sink in Long Sequence Recommendation with Memory-Conditioned Gated Delta Networks

    Jun 3, 2026Zhuang Zhuang, Zhipeng Wei, Ji Dai +4Memory-Augmented Neural NetworksSequential Recommendation

  14. Unlocking Feature Learning in Gated Delta Networks at Scale

    Jun 2, 2026Yifeng Liu, Quanquan GuNeural Network OptimizationHyperparameter Transfer

  15. Patched-DeltaNet: Token-Level Event-Driven Memory for Linear-Time Anomaly Detection

    May 27, 2026Tae-Gyun Lee, Junyoung Park, Kyu Won HanGated Delta NetworksPatch-Based Time Series Modeling

  16. Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders

    May 15, 2026Tomasz SteiferGated AttentionRecurrent Transformers

  17. Kaczmarz Linear Attention

    May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language ModelingRecurrent Transformers