Linear Attention

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 43% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration
Oct 8, 2026Kaicheng Xiao, Liran Dong, Haotian Li +1Memory-Augmented Neural Networks KV-Cache Quantization
The Chinese University of Hong Kong
What can linear attention learn from nonlinear teachers in-context?
Oct 7, 2026Mary Letey, Arman Rysmakhanov, Yue M. Lu +2In-Context Learning Linear Attention
Applied Mathematics, Harvard University Kempner Institute, Harvard University · Williams College · Center for Brain Science, Harvard University Society of Fellows, Harvard University
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Oct 7, 2026Xiaoran Liu, Ziwei He, Xipeng QiuLong-Context Language Modeling Hybrid Attention
Fudan University · Shanghai Innovation Institute · OpenMOSS Team
PHBA: Prefix-State Hybrid Block Attention
Oct 6, 2026Ruijie Li, Jiaxi Hu, Shiyu Wang +1Long-Context Modeling Linear Attention
The Hong Kong University of Science and Technology (Guangzhou) · Independent researcher
HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language Modeling Linear Attention
Monash University · Zhejiang University
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
Oct 5, 2026Zhuokun Chen, Feng Chen, Xi Lin +4World Model Learning Long-Horizon Video Generation
Monash University · The University of Adelaide · Zhejiang University
Switching Linear Attention
Sep 30, 2026Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3Efficient Attention Linear Attention
Stanford University · Unconventional AI. Work done at Stanford University.
SMat-Attention: Structured Long-Context Sequence Modeling
Sep 28, 2026Emile Anand, Abdullah Ateyeh, Archer Wang +1Long-Context Modeling Linear Attention
Massachusetts Institute of Technology
MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution
Sep 28, 2026Prasoon Dev, Anirudh Sankar, Vasudeva VarmaLong-Context Language Modeling Gated Attention
Language Technologies Research Center International Institute of Information Technology Hyderabad Hyderabad, Telangana, India
SketchSSM: Write to the Full State, Read from a Compact Sketch
Sep 27, 2026Omin Kwon, JoongWon Shin, Minseo Kim +3LLM Inference Acceleration Linear Attention
Seoul National University · UC Berkeley · KAIST
GTR: Gated Token Recurrence for Efficient Dense Prediction
Sep 22, 2026Zhe Feng, Longfei Liu, Wei Liu +7Efficient Neural Network Inference Gated Attention
Didi International Business Group · Intellindust AI Lab · Institute of Automation, Chinese Academy of Sciences +2
Rethinking Vision Architectures with Gated Linear Attention and KAN
Sep 18, 2026Ali Mehizel, Oussama KhaldiVisual Representation Learning Vision Transformer
Department of Applied Statistics, ENSSEA, Kolea, Tipaza, Algeria
Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation
Sep 17, 2026Haocheng Xi, Yiming Xie, Hexu Zhao +8Video Diffusion Models Streaming Video Generation
University of California, Berkeley · Impossible, Inc. · University of Texas at Austin
OneLA: Scaling Linear-Attention Decoding to Large Beams in Generative Recommendation
Sep 14, 2026Xiangrui Yang, Cheng Peng, Yunfeng Zhao +9Linear Attention Autoregressive Decoding
Kuaishou Technology China · Peking University China
RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States
Sep 14, 2026Luca Herranz-Celotti, Vincent GuigueRecurrent Neural Networks Linear Attention
ISIR Sorbonne, France
Kalman Delta Networks: Uncertainty-aware Associative Memory
Sep 7, 2026Ngoc Bui, Tinglin Huang, Rex YingMemory-Augmented Neural Networks Linear Attention
Department of Computer Science, Yale University
RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers
Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Rotary Positional Embeddings Low-Rank Attention
Peking University, Melon Group · Alibaba Group · University of Electronic Science and Technology of China +1
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Positional Embeddings Linear Attention
Institute of Automation, Chinese Academy of Sciences, Beijing, China
Liquid Gated Attention
Aug 31, 2026Yiheng Jiang, Yuanbo Xu, Yongjian YangGated Attention Linear Attention
Mobile Intelligent Computing Laboratory, College of Computer Science and Technology, Jilin University, China
Sliding-window beats linear attention
Aug 28, 2026Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1KV Caching LLM Inference Acceleration
Microsoft Applied Sciences Group (ASG) · Independent
Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Aug 12, 2026Zunhai Su, Bohan Sun, Xialie Zhuang +8Linear Attention Hybrid Attention
Startlux · Tsinghua University · University of Chinese Academy of Sciences +3
MixFormer: Linear Transformer with Mixture of Memory Experts
Aug 10, 2026Yu Guo, Lei DuanMemory-Augmented Neural Networks Transformer
School of Computer Science, Sichuan University Chengdu, China · School of Artificial Intelligence, Sichuan University Chengdu, China
Linearized 2-Simplicial Attention
Aug 10, 2026Aritra Das, Dhruman Gupta, Debayan GuptaLinear Attention Attention Mechanisms
1Truth Audit Labs
HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers
Aug 7, 2026Dong Liu, Yanxuan Yu, Renata Borovica-Gajic +2Efficient ViTs Linear Attention
University of California, Los Angeles, USA · Columbia University, USA · The University of Melbourne, Australia +1
Retrofitting Linear Attention into Diffusion Language Models
Aug 6, 2026Jinha Kim, Younghun Roh, Jaeyeon KimBlockwise Diffusion Language Models Diffusion Language Model Inference
Apple · *Work done at MIT · Google DeepMind +1
A Unified Resolution-Conditioned Framework for Orthogonal Line-Scanning Image Fusion
Aug 4, 2026Yiming Gong, Kai WangLinear Attention Microscopy Image Analysis
Department of Physics, University of Michigan, Ann Arbor, 48109, MI, USA · School of Physics and Astronomy, Sun Yat-sen University and Guangdong Provincial Key Laboratory of Quantum Metrology and Sensing, Zhuhai Campus, Zhuhai, 519082, Guangdong Province, China · Zhuhai Key Laboratory of Optoelectronic Functional Materials and Membrane Technology, Zhuhai, 519082, Guangdong Province, China
Bole: Efficient Tree Speculation for Hybrid-Attention Language Models
Aug 3, 2026Li Wang, Yi Su, Xiabao Wu +9Speculative Decoding LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · Shanghai Jiao Tong University +1
Linear Multi-Timescale Retention as a Memory-Efficient Vision-Language Bridge
Aug 3, 2026Ashfak Yeafi, Mehedi Hasan, Md Khairul IslamMemory-Augmented VLMs Vision-Language Models
Department of Electrical and Electronic Engineering Khulna University of Engineering & Technology Khulna-9203, Bangladesh · Department of Computer Science and Engineering Brac University, Dhaka, Bangladesh · Department of Mathematics and Computer Science Hobart and William Smith Colleges Geneva, NY, USA
Indexing: the Beginning and the End
Jul 24, 2026Alexander Kozachinskiy, Vicente Opazo, Felipe UrrutiaInformation Bottleneck Linear Attention
CENIA · Pontifical Catholic University of Chile
SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation
Jul 23, 2026Junsong Chen, Jincheng Yu, Yitong Li +11Video Diffusion Models Attention Residual Connections
NVIDIA

Oct 8, 2026, cs.LGJ/K move · Enter open · S save

Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migration

Kaicheng Xiao, Liran Dong, Haotian Li, Guoliang Xing

The Chinese University of Hong Kong

Memory-Augmented Neural Networks KV-Cache Quantization Linear Attention

Open paper AskPDF arXiv

KV-cache quantization and linear attention are two representative approaches to tackling the storage and computational costs of Transformers. KV-cache quantization compresses individual KV entries into discrete codes but retains all entries, whereas linear attention recurrently aggregates multiple historical KV contributions into a fixed-size continuous state but can introduce interference. This contrast raises the question of whether per-KV compression and multi-KV aggregation can be bridged within a single mechanism for efficient attention. We identify RAM-Net as such a bridge through soft assignments over a discrete address space. These assignments determine recurrent updates to the continuous slot state associated with each address. Under a restricted RAM-Net construction, we prove that soft address assignments extend hard quantized matching to a separable read-write overlap that locally approximates full-attention similarity and supports recurrent aggregation. These connections further enable Transformer-to-RAM-Net weight migration through a new path based on a soft-quantized intermediate construction. Across nine pretrained Transformer models from 0.3B to 7B parameters, RAM-Net recovers an average of 87.1% of the teachers' accuracy gains over random guessing across six commonsense and knowledge tasks using only a 500M-token budget per model.

Momentum

Related topics

Latest papers 95