Gated Delta Networks

Follow this topic Ask about this topic

Gated Delta Networks process sequences by repeatedly updating a fixed-size matrix state: delta-rule updates write new information while gates regulate what is retained or erased. Chunkwise and WY-style algorithms parallelize training, while recurrent decoding offers linear sequence cost and constant-size state instead of growing a full attention cache. Current work explores finer-grained erase and write controls, normalized update steps, larger or structured memories, and hybrids with attention; evaluations often focus on language modeling, associative recall, and long-context retrieval benchmarks such as needle-in-a-haystack tasks. A central challenge is balancing memory capacity and retrieval accuracy against efficiency, while keeping recurrent states numerically stable and reliable beyond the training context length.

Latest papers

21 papers2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory
Oct 5, 2026Parsa Hejabi, Morteza Dehghani, Payam PirayKalman Filtering Bayesian Filtering
University of Southern California
Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention
Oct 2, 2026Ruijie Li, Shengnan Ding, Weimin Zhang +6Linear Attention Delta-Rule Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
Constant-Memory Recall: Learned Associations in a Fixed Matrix State
Sep 23, 2026Samuel LarsonAssociative Memory Memory-Efficient Inference
Pebble ML
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length Generalization Delta-Rule Linear Attention
♡University of Freiburg · ♢Microsoft Research · ELLIS Institute Tübingen +9
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Sep 3, 2026Sergii Kozyrev, Davyd MaiborodaLLM Quantization 4-Bit Quantization
Visual Representation and History Modeling for Navigation World Models
Aug 26, 2026Guangfu Guo, Xiaoqian Lu, Rui Liu +3World Models World Model Learning
Clemson University Clemson, SC, USA · University of Ottawa Ottawa, ON, Canada
SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models
Aug 23, 2026Qingwen Lin, Boyan Xu, Xiao Liu +2Recurrent Neural Networks Gated Delta Networks
School of Computer Science, Guangdong University of Technology · Yuanshi Intelligence · College of Science, Shantou University
TreeWY: Speculative Verification for Gated DeltaNet Hybrids
Aug 21, 2026Sneha Murthy GhantasalaSpeculative Decoding LLM Inference Acceleration
Thomson Reuters
The Query Knows What to Forget: A Second Erase Direction for Linear Attention
Aug 13, 2026Dhruman Gupta, Aritra Das, Debayan GuptaDelta-Rule Linear Attention Long-Context Retrieval
Truth Audit Labs
DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows
Aug 2, 2026Guangfu Guo, Xiaoqian Lu, Linsey Pang +4Gated Delta Networks Diffusion Language Models
1Clemson University · University of Ottawa · 3PayPal +1
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Linear Attention Delta-Rule Linear Attention
1ETH Zurich, D-INFK · 2ETH AI Center, ETH Zurich
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Long-Context Retrieval Memory-Augmented Language Models
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory
Jun 23, 2026Habibullah AkbarHybrid Attention Long-Context Language Modeling
Kreasof AI Jakarta, Indonesia
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network Initialization Linear Attention
1The University of Sydney · 2Together AI · 5Microsoft +2
SinkRec: Mitigating Semantic State Sink in Long Sequence Recommendation with Memory-Conditioned Gated Delta Networks
Jun 3, 2026Zhuang Zhuang, Zhipeng Wei, Ji Dai +4Memory-Augmented Neural Networks Sequential Recommendation
Kuaishou Technology, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China · Independent Researcher
Unlocking Feature Learning in Gated Delta Networks at Scale
Jun 2, 2026Yifeng Liu, Quanquan GuNeural Network Optimization Hyperparameter Transfer
University of California, Los Angeles
Patched-DeltaNet: Token-Level Event-Driven Memory for Linear-Time Anomaly Detection
May 27, 2026Tae-Gyun Lee, Junyoung Park, Kyu Won HanGated Delta Networks Time-Series Anomaly Detection
Electronics and Telecommunications Research Institute, South Korea
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
May 15, 2026Tomasz SteiferGated Attention Recurrent Transformers
Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
FG $^2$ -GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Long-Context Modeling Linear Attention
Meituan · Beijing, China

Oct 5, 2026J/K move · Enter open · S save

Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory

Parsa Hejabi, Morteza Dehghani, Payam Piray

University of Southern California

Kalman Filtering Bayesian Filtering Memory-Augmented Language Models Gated Delta Networks

Open paper AskPDF arXiv

Recurrent sequence models must decide how strongly to overwrite their memory at each token. Read as Bayesian filtering, this write is the gain of a Kalman update, set by uncertainty from two sources that pull it in opposite directions: volatility, how quickly the underlying associations change, and stochasticity, how noisy each observation of them is. First, we show that the update of gated delta-rule memories is the form this filter takes under isotropic uncertainty. Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. A dense covariance would have to be propagated token by token, ruling out the parallel training these models depend on. We therefore give two assumed-density approximations, diagonal and quasi-diagonal, both of which leave the memory update in delta-rule form and reuse its chunked kernels. On controlled recall tasks in which associations change and observations are corrupted, Voltic leads all baselines. On the task combining volatility and stochasticity, its margin over the strongest baseline is larger at both extrapolation sizes than at the training sizes. In 45M-parameter language models it leads an eight-task reasoning average and achieves higher retrieval accuracy beyond the training context length than gated baselines, at throughput close to those baselines. Deriving the write from an uncertainty recursion therefore makes memory more responsive to change.