Delta-Rule Linear Attention

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 25

All topics

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language Modeling Linear Attention
Monash University · Zhejiang University
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM Quantization LLM Inference Acceleration
Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM Quantization LLM Inference Acceleration
UC Berkeley · University of Washington · MIT +2
DRAM: Delta-rule Recurrent Associative Memory for Robot Manipulation Policies
Sep 26, 2026Xinyu Zhao, Yixiang Shan, Tao Yang +5Long-Horizon Robotic Manipulation Delta-Rule Linear Attention
Simplexity Robotics
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Sep 21, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +8Length Generalization Delta-Rule Linear Attention
Kalman Delta Networks: Uncertainty-aware Associative Memory
Sep 7, 2026Ngoc Bui, Tinglin Huang, Rex YingMemory-Augmented Neural Networks Linear Attention
Department of Computer Science, Yale University
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Sep 3, 2026Sergii Kozyrev, Davyd MaiborodaLLM Quantization 4-Bit Quantization
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV Caching Memory-Efficient Inference
East China Normal University · Meituan · South China University of Technology
Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Aug 3, 2026Ronglong BaoDelta-Rule Linear Attention Language Model Distillation
DT-Project
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Linear Attention Delta-Rule Linear Attention
1ETH Zurich, D-INFK · 2ETH AI Center, ETH Zurich
The Key to Going Linear: Analysis-Driven Transformer Linearization
Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiSoftmax Attention LLM Inference Acceleration
Qualcomm AI Research∗ · Qualcomm AI Research
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Long-Context Retrieval Memory-Augmented Language Models
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets
Jul 2, 2026Wanyun CuiLong-Context Retrieval Memory-Augmented Language Models
Shanghai University of Finance and Economics
CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
Jun 25, 2026Sayak DuttaLinear Attention Delta-Rule Linear Attention
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Memory-Augmented Language Models Long-Context Language Modeling
1Qwen Team · 2Nanjing University · 3Zhejiang University
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
Jun 12, 2026Tiantian ZhangEfficient Attention Linear Attention
Columbia University
Q-Delta: Beyond Key-Value Associative State Evolution
Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkLinear Attention Delta-Rule Linear Attention
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers
May 20, 2026Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros +4Efficient Transformer Inference AI Accelerator Inference
Computing Systems Lab Huawei Technologies2026 Switzerland
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Transformer Attention Linear Attention
Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Momentum Methods Linear Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language Modeling Recurrent Neural Networks
1MIT · 2Liquid AI
FG $^2$ -GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Long-Context Modeling Linear Attention
Meituan · Beijing, China
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Dec 14, 2025Jingdi Lei, Di Zhang, Soujanya PoriaDynamical Systems Linear Attention
Nanyang Technological University · Fudan University

Oct 5, 2026, cs.CLJ/K move · Enter open · S save

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Zhuokun Chen, Xi Lin, Xiyu Wu, Jiahao He+2

Monash University · Zhejiang University

Long-Context Language Modeling Linear Attention Delta-Rule Linear Attention Attention Mechanisms

Open paper AskPDF arXiv

Linear attention enables efficient long-context autoregressive decoding by compressing history into recurrent states, but this compression can make selective access to sparse and distant information difficult. Existing chunk-based extensions increase memory capacity, yet learned chunk-mixing coefficients may remain fixed with respect to input content and therefore cannot adapt historical access to each query. We introduce \emph{Hybrid Linear Attention} (HLA), a query-dependent chunk-level attention mechanism for Gated DeltaNet (GDN). HLA represents each completed chunk as an exact affine state transition and computes content-dependent routing gates from compact, self-attentively pooled representatives. Each gate interpolates the corresponding historical transition with the identity map, controlling both the chunk's additive memory and its transformation of earlier states. Effective-support regularization further encourages concentrated routing for sparse inference. We evaluate HLA under both pretrained adaptation and from-scratch training. Across Qwen3.5 models from 0.8B to 9B, HLA consistently improves over native GDN and fixed chunk mixing, with gains of up to 5.57 percentage points on LongBench-V2 and 3.97 points on RULER. In a controlled from-scratch 1.3B setting trained for 100B tokens with a 4K context, HLA also improves RULER performance from 4K to 32K, with gains increasing from 0.83 points at 4K to 4.22 points at 32K. These results demonstrate that query-dependent composition of recurrent memory improves long-context modeling and remains effective beyond the training context while using compact per-chunk affine summaries. Project page: https://caesarhhh.github.io/hla/

Momentum

Related topics

Latest papers 25