Gated Deltanet

Follow this topic Ask about this topic

Momentum

8 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 6Week of Sep 21

Latest papers 43

All topics

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10Large Language Model Quantization Kimi Delta Attention
UC Berkeley · University of Washington · MIT +2
DeltaSeek: Toward Active Perception in Evolving Construction Environments
Sep 27, 2026Sanjay Acharjee, Md Nazmus SakibViewpoint-Dependent Active Perception Robot Systems
Department of Civil Engineering, University of Texas at Arlington, TX 76019, USA.
DeltaWAM: Delta World Action Models for Bimanual Manipulation
Sep 23, 2026Han Yan, Zishang Xiang, Haokai Jiang +6Recent World-Action Models World Models
School of Computer Science, Peking University · AI2 Robotics
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models
Sep 20, 2026Zhiyuan MaGated Deltanet Block Sparse Flash Attention
DeltaSelect: Affordable A/B Testing for Coding Agents
Sep 17, 2026Nicholas J. ConnAgentic Benchmarks Benchmark Saturation
Conn Castle Studios
DiffLUT-Net: Differentiable Training of FPGA LUT Networks with Learnable Connectivity
Sep 10, 2026Jiaqi Ye, Xinrui Gong, Jingcun Wang +3Field-Programmable Gate Arrays Neural Network Inference
TU Darmstadt · TU Ilmenau
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
Sep 3, 2026Sergii Kozyrev, Davyd MaiborodaLarge Language Model Quantization Fp8
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4Kimi Delta Attention Recurrent State
East China Normal University · Meituan · South China University of Technology
Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs
Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2Kimi Delta Attention Linear Attention
Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Aug 27, 2026Tao Zhang, Jianchao Tan, Pingwei Sun +6Recurrent State Gated Deltanet
South China University of Technology · Meituan · East China Normal University
Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference
Aug 5, 2026Zheng Liu, Zeyu Guo, Zihan Liu +9Diffusion-Based Vision-Language-Actions Embodied Artificial Intelligence
Shanghai Jiao Tong University Shanghai, China · Shanghai Jiao Tong University, Shanghai Qi Zhi Institute Shanghai, China · ICT, Chinese Academy of Sciences Beijing, China
Zero-Observation User Reactivation with Gap-Driven Dimensional Gating
Jul 22, 2026Jiandong Ding, Tianying Liu, Fuyuan Liu +2Sequential Recommendation Discontinuous Gating
Fudan University Shanghai, China · Huawei Technologies Shanghai, China
DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation
Jul 15, 2026Son Ha Xuan, Xuan-Bach Le, Phat T. Tran-TruongLarge Language Model Adaptation Motion-Language Alignment
RMIT University, Ho Chi Minh City, Vietnam · Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), VNU-HCM, Ho Chi Minh City, Vietnam
DeltaV: Thinking with Visual State Updates in Unified Large Multimodal Models
Jul 9, 2026Pengjie Wang, Linger Deng, Zujia Zhang +6Large Multimodal Models Multimodal Reasoning
Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc.
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Kimi Delta Attention Linear Attention
1ETH Zurich, D-INFK · 2ETH AI Center, ETH Zurich
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding Drafter
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Kimi Delta Attention Efficient Long-Context Inference
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences
Jun 26, 2026Yankai Yang, Yancheng Long, Bin Wen +4Fine-Grained Video Understanding Video Understanding
1Harbin Institute of Technology, Shenzhen · 2Kuaishou Technology
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Transformer Architectures Gated Deltanet
1The University of Sydney · 2Together AI · 5Microsoft +2
On Subquadratic Architectures: From Applications to Principles
Jun 10, 2026Anamaria-Roberta Hartl, Levente Zólyomi, David Stap +6Sequence Modeling Linear Attention
Johannes Kepler University Linz · NXAI GmbH
When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet
Jun 4, 2026Luoming Zhang, Yuwei Ren, Kui Zhang +7Matrix Multiplication Kimi Delta Attention
Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc.
DELTAMEM: Incremental Experience Memory for LLM Agents via Residual Trees
Jun 2, 2026Haoran Tan, Zeyu Zhang, Zhicheng Cao +2Large Language Model Agents Gated Deltanet
Gaoling School of Artificial, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +1
Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
May 31, 2026Dong Le, Thong Nguyen, Cong-Duy Nguyen +1Kimi Delta Attention Linear Attention
Nanyang Technological University · National University of Singapore · VinUniversity
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
May 27, 2026Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1Asymmetric Grouped-Attention Variate Mixer Sequence Modeling
Carnegie Mellon University · Google Research
DeltaMCP: Incremental Regeneration via Spec-Aware Transformation for MCP servers
May 27, 2026Aditya Pujara, Xiaogang Zhu, Hsiang-Ting ChenModel Context Protocol Gated Deltanet
Microsoft · University of Adelaide
Patched-DeltaNet: Token-Level Event-Driven Memory for Linear-Time Anomaly Detection
May 27, 2026Tae-Gyun Lee, Junyoung Park, Kyu Won HanTime-Series Anomaly Detection Gated Deltanet
Electronics and Telecommunications Research Institute, South Korea
DeltaCam: Differential Intrinsic Camera Modeling for Video Generation
May 24, 2026Debabrata Mandal, Zhihan Peng, Yujie Wang +1Camera Control Long Video Generation
UNC, Chapel Hill, USA
Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs
May 21, 2026Jongseo Lee, Hyuntak Lee, Sunghun Kim +3Spatial Blindness Motion
1Kyung Hee University · 2Princeton University
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzKimi Delta Attention Gated Deltanet
NVIDIA
DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox Checkpoint/Rollback
May 21, 2026Yunpeng Dong, Jingkai He, Shiqi Liu +7Sandbox Rollback
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Engineering Research Center for Domain-specific Operating Systems, Ministry of Education, China · Huawei Technologies Co., Ltd.
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
May 15, 2026Tomasz SteiferKimi Delta Attention Gated Deltanet
Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland
SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer
May 14, 2026Haoyi Zhu, Haozhe Liu, Yuyang Zhao +6Video World Models World Models
NVIDIA
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Kimi Delta Attention Linear Attention
Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1
WriteSAE: Sparse Autoencoders for Recurrent State
May 12, 2026Jack YoungRecurrent State Gated Deltanet
Indiana University
Mixture of Layers with Hybrid Attention
May 10, 2026Ivan Ternovtsii, Yurii BilakTransformer Attention Mixture-Of-Experts
Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
May 10, 2026Rui Liu, Dian Yu, Zhenwen Liang +6Rubric-Based Scoring Visual Reasoning
1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuKimi Delta Attention Recurrent Model
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6Generative Pretrained Transformers Priming
AWS Agentic AI
MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Kimi Delta Attention Linear Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusKimi Delta Attention Recurrent Model
1MIT · 2Liquid AI
FG $^2$ -GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Kimi Delta Attention Linear Attention
Meituan · Beijing, China
DeltaSeg: Tiered Attention and Deep Delta Learning for Multi-Class Structural Defect Segmentation
Apr 20, 2026Enrique Hernandez Noguera, Md Meftahul Ferdaus, Elias Ioup +1U-Net Defect Detection
University of New Orleans, New Orleans, LA, USA · Center for Geospatial Sciences, Naval Research Laboratory, Hancock County, MS, USA
DeltaDeno: Zero-Shot Anomaly Generation via Delta-Denoising Attribution
Nov 21, 2025Chaoran Xu, Chengkan Lv, Qiyu Chen +3Iterative Denoising Methods Gated Deltanet
Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · CASI Vision Technology CO., LTD., Luoyang, China +1

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li+9

UC Berkeley · University of Washington · MIT+2

Large Language Model Quantization Kimi Delta Attention LLM Inference Optimization Recurrent State Gated Deltanet Bottlenecks

Open paper AskPDF arXiv

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision Compensator Tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05--3.70

\times

at the kernel level and 1.47

\times

for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

Momentum

Related topics

Latest papers 43