Linear Attention

Ask about this topic

Momentum

25 papers in the last four weeks, up 92% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 183

All topics
CardsList

  1. In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners

    Oct 1, 2026Haotian Gu, Yizhou Xu, Lenka ZdeborováIn-Context LearningLinear Attention

    Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · University of Chinese Academy of Sciences · Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

  2. Switching Linear Attention

    Sep 30, 2026Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3Kimi Delta AttentionLinear Attention

    Stanford University · Unconventional AI. Work done at Stanford University.

  3. Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

    Sep 29, 2026Eric Enouen, Sainyam GalhotraPositional EncodingEfficient Long-Context Inference

    Cornell University

  4. Retrieval Capacity of Self-Attention Under Competition

    Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateLinear AttentionCompetition

    University of Luxembourg, Luxembourg

  5. SMat-Attention: Structured Long-Context Sequence Modeling

    Sep 28, 2026Emile Anand, Abdullah Ateyeh, Archer Wang +1Linear AttentionSequence Modeling

    Massachusetts Institute of Technology

  6. MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution

    Sep 28, 2026Prasoon Dev, Anirudh Sankar, Vasudeva VarmaLinear AttentionMulti-Scale Attention

    Language Technologies Research Center International Institute of Information Technology Hyderabad Hyderabad, Telangana, India

  7. Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context

    Sep 28, 2026Muyu He, Yuchen Liu, Ran Tao +1Compact Latent Sensor TokensLinear Attention

    Independent · University of Pennsylvania · Drexel University

  8. CoDrive: Cross-Vehicle World-Consistent Video Generation with Precise Trajectory Control for Cooperative Driving

    Sep 28, 2026Yu Meng, Baining Zhao, Junta Wu +10DrivesCamera Trajectories

    Tsinghua University · Tencent Hunyuan · Zhongguancun Academy

  9. WorldAttention: An Efficient Attention Architecture for Interactive Video World Models

    Sep 28, 2026Zeyu Zhang, Jinyuan Mao, Dakai An +6Video World ModelsDynamic Sparse Attention

    DAMO Academy, Alibaba Group · Zhejiang University · Hong Kong University of Science and Technology +2

  10. DORA: Dynamic Online Reinforcement Agent for Token Pruning in Vision Transformers

    Sep 28, 2026Kaixuan He, Song Chen, Yi KangVisual Token PruningVision Transformer

    University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China

  11. SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval

    Sep 27, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungRecurrent ModelPersistent State

    College of Pharmacy, Chungnam National University Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University Daejeon, Republic of Korea

  12. NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers

    Sep 23, 2026Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang +1Transformer AttentionLinear Attention

    University of Exeter · University of Exeter, Exeter, United Kingdom

  13. DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video

    Sep 23, 2026Taeyoun Kwon, Seungjin Kim, Hyeonyu Kim +1Kimi Delta AttentionLinear Attention

    Maum AI Inc. · Seoul National University · Yonsei University

  14. Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering

    Sep 21, 2026Jingkun Liu, Yue SongLinear AttentionAttractors

    Tsinghua University · Shanghai Jiao Tong University

  15. Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer

    Sep 20, 2026David LitmanChessTransformer Attention

    Computational Neurobiology Laboratory, Salk Institute

  16. Rethinking Vision Architectures with Gated Linear Attention and KAN

    Sep 18, 2026Ali Mehizel, Oussama KhaldiVision TransformerVision Foundation Models

    Department of Applied Statistics, ENSSEA, Kolea, Tipaza, Algeria

  17. Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

    Sep 17, 2026Haocheng Xi, Yiming Xie, Hexu Zhao +8Video Diffusion ModelsVideo Generation

    University of California, Berkeley · Impossible, Inc. · University of Texas at Austin

  18. The Attention Within: Consensus Dynamics in Selective State Space Models

    Sep 16, 2026João Pedro Silvestre, Álvaro Rodríguez Abella, Paulo TabuadaTransformer AttentionState Space Models

    Electrical and Computer Engineering Department, University of California at Los Angeles, Los Angeles, CA 90095 USA · Department of Applied Mathematics, Comillas Pontifical University, Madrid, 28015 - Madrid, Spain

  19. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

    Sep 14, 2026Xingyang Li, Dongyun Zou, Shining Zhang +8Linear AttentionVideo Generation

    2MIT · 1Nunchux AI · 3CMU +3

  20. Rethinking Heterogeneous System Disaggregation for Subquadratic Attention

    Sep 14, 2026Arya Tschand, Yaosheng Fu, Vikram Sharma Mailthody +6Linear AttentionGraphics Processing Unit Memory

    Harvard University Cambridge, Massachusetts, USA · NVIDIA Santa Clara, California, USA

  21. RunningTensor: Generalizing Linear Attention to Higher-Order Recurrent States

    Sep 14, 2026Luca Herranz-Celotti, Vincent GuigueKimi Delta AttentionRecurrent State

    ISIR Sorbonne, France

  22. OneLA: Scaling Linear-Attention Decoding to Large Beams in Generative Recommendation

    Sep 14, 2026Xiangrui Yang, Cheng Peng, Yunfeng Zhao +9Kimi Delta AttentionLinear Attention

    Kuaishou Technology China · Peking University China

  23. ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification

    Sep 9, 2026Jianzong Wang, Chuhang Liu, Botao Zhao +6Confidence EstimationConfidence Calibration

    Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China · Tsinghua Shenzhen International Graduate School, Shenzhen, China · Hubei University of Technology, Hubei, China

  24. Why shared attention vectors fail: a case for outcome-indexed tuning

    Sep 8, 2026Lenard DomeAttention LayersLinear Attention

    Department of Psychiatry and Psychotherapy, Faculty of Medicine University of Tübingen, Tübingen, Germany · German Center for Mental Health (DZPG), Tübingen, Germany

  25. Kalman Delta Networks: Uncertainty-aware Associative Memory

    Sep 7, 2026Ngoc Bui, Tinglin Huang, Rex YingKimi Delta AttentionLinear Attention

    Department of Computer Science, Yale University

  26. RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

    Sep 6, 2026Zekun Zhang, Yixiang Cai, Yuxi Liu +9Video Diffusion ModelsSpatio-Temporal Attention

    Peking University, Melon Group · Alibaba Group · University of Electronic Science and Technology of China +1

  27. Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

    Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiRotary Position EmbeddingTransformer Architectures

    Institute of Automation, Chinese Academy of Sciences, Beijing, China

  28. HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models

    Sep 2, 2026Renjie Xie, Juncheng Yang, Aoting Hu +4Efficient Long-Context InferenceKey-Value Cache

    Nanjing University of Posts and Telecommunications · Tylogi AI Lab / TAIL · Anhui University of Technology +2

  29. Multi-Head Self Attention is a Parameter Identification Mechanism

    Sep 1, 2026W. Ross MorrowMulti-Head AttentionLinear Attention

  30. Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

    Sep 1, 2026Xurui Song, Weishi Wang, Zhongqi Yue +5Long Visual-Token SequencesFaithfulness

    SAP · Microsoft · College of Computing and Data Science, Nanyang Technological University, Singapore

  31. Can Video World Models Track Unobserved World States?

    Aug 31, 2026Joonghyuk Shin, Yicong Hong, Jaesik Park +1Video World ModelsHidden States

    Seoul National University · Roblox

  32. Tail-Replay: Escaping the Curse of Linear Attention in Prefix Caching for Hybrid LLMs

    Aug 31, 2026Yirui Liu, Ruoling Qi, Xuaner Wu +2Kimi Delta AttentionLinear Attention

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University

  33. ConCA: Concentration-Aware Channel Attention for Fine-Grained Visual Recognition

    Aug 31, 2026Yu-Sheng Liu, Yu-Chen TungConvolutional Block Attention ModuleFine-Grained Perception

    Department of Physics, National Kaohsiung Normal University, Kaohsiung, 824, Taiwan

  34. Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention

    Aug 31, 2026Bingde Liu, Wu Ran, Jinglei Zhang +2Depth EstimationVisual Fidelity

    MoE Key Lab of Artificial Intelligence, Institute of AI, Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China

  35. ClusterAttention: A training-free speedup of bidirectional attention

    Aug 27, 2026Kasper Nordenram, Amelie DittmannDynamic Sparse AttentionLinear Attention

    Independent researcher

  36. Ask Self, Ask Others: Relation Is All You Need

    Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieLinear AttentionBlock Sparse Flash Attention

    Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore

  37. Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference

    Aug 10, 2026Burc GokdenLinear AttentionInterleaved Graph Attention

    Fromthesky Research Labs LLC, Oregon, USA

  38. ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Models

    Aug 10, 2026Róisín LuoPositional EncodingLinear Attention

    University of Galway, Ireland

  39. Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

    Aug 9, 2026Qucheng Gao, Zuyi Yang, Xiao ChenAttractorsLinear Attention

    Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA · Department of Physics, Tsinghua University, Beijing 100084, China

  40. Gated Spatial Redundancy Projection for Pathology Transformer Attentions

    Aug 8, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Computational PathologyWhole-Slide Images

    Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, Canada · Axe Cancer, Centre de recherche du CHUM, Université de Montréal, Montréal, Canada · Mila - Quebec AI Institute, Montreal, Canada

  41. Faster Query-Key Learning Sharpens Attention in Self-Attention Models

    Aug 7, 2026Rahul Vashisht, Harish G. RamaswamyLinear AttentionInterpretability

    Department of CSE, IIT Madras, India · Department of DSAI, IIT Madras, India.

  42. HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers

    Aug 7, 2026Dong Liu, Yanxuan Yu, Renata Borovica-Gajic +2Self-Supervised Vision TransformersVision Transformer

    University of California, Los Angeles, USA · Columbia University, USA · The University of Melbourne, Australia +1

  43. Retrofitting Linear Attention into Diffusion Language Models

    Aug 6, 2026Jinha Kim, Younghun Roh, Jaeyeon KimDiffusion Language ModelsKimi Delta Attention

    Apple · *Work done at MIT · Google DeepMind +1

  44. GAUGE: Granularity-Adaptive Counterfactual Gating of Evidence for Incomplete Multimodal Classification

    Aug 6, 2026Yunping Shi, En Yu, Kairui Guo +1Multimodal ClassificationCounterfactual Learning

    Australian Artificial Intelligence Institute (AAII), University of Technology Sydney (UTS), Australia

  45. AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation

    Aug 5, 2026Chengyu Qiu, Xiao Fu, Fengcun Li +6Hardware EfficiencyTime-To-First-Token

    Tsinghua University · Meituan · Beijing, China

  46. Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

    Aug 3, 2026Cunchen Hu, Liangliang Xu, Tian Liu +9Graphics Processing Unit ResourcesRate Scaling

    China Telecom Cloud Computing Research Institute · Xidian University · University of Science and Technology of China +1

  47. Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design

    Aug 2, 2026Sen SongTransformer AttentionTransformer Architectures

    Independent Researcher, Guangzhou, China

  48. AdaHAT: Adaptive Hard Attention to the Task in Task-Incremental Learning

    Aug 2, 2026Pengxiang Wang, Hongbo Bo, Jun Hong +2Class-Incremental LearningCatastrophic Forgetting

    School of Mathematical Sciences, Peking University, Beijing, China · Population Health Sciences Institute, Newcastle University, Newcastle, UK · School of Engineering Mathematics and Technology, University of Bristol, Bristol, UK +1

  49. How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair

    Jul 28, 2026Ramtin Ehsani, Irene Manotas, Saurabh Pujar +2Automated Program RepairBug

    Drexel University Philadelphia, PA, USA · IBM Research Yorktown, NY, USA

  50. What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation

    Jul 24, 2026Minwoo Yu, Young-guk HaLinear AttentionTemporal Graph Neural Networks

    Smart Computing Laboratory, Department of Computer Science & Engineering, Konkuk University, Seoul 05029, Republic of Korea.

  51. SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

    Jul 23, 2026Junsong Chen, Jincheng Yu, Yitong Li +11Video Diffusion ModelsLinear Attention

    NVIDIA

  52. Beyond Transformers: Linear Attention Policy for Open-Vocabulary Object Goal Navigation

    Jul 21, 2026Jiahong Zhang, Yifan Lin, Yandong Zhang +6Vision-Language NavigationNavigation

    Institute of Automation, Chinese Academy of Sciences, Beijing 100045, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China · Beihang University, Beijing 100191, China +2

  53. Norm or Direction? Decoding Vision Mambas for High-Resolution Vision

    Jul 21, 2026Jin Yu, Juyoun ParkMamba-Based ModelsVision State Space Models

    Center for Humanoid Research Korea Institute of Science and Technology

  54. On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

    Jul 20, 2026Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5Transformer ArchitecturesLinear Attention

    University of Wisconsin–Madison, Department of Statistics · University of Maryland, Department of Electrical and Computer Engineering · University of Warsaw, Institute of Applied Mathematics and Mechanics +2

  55. Convolution for Large Language Models

    Jul 20, 2026Yuchuan Tian, Yingte Shu, Wei He +7Linear AttentionConvolutional

    1Peking University · 2Huawei Technologies · 3Tsinghua University

  56. L1 Augmented Attention as an Improved Vector Similarity Metric

    Jul 20, 2026Kurt GoddenTransformer AttentionLinear Attention

    Walsh College, Troy Michigan, USA

  57. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    Jul 19, 2026Ayoub Ghriss, Sourav ChakrabortyKimi Delta AttentionLinear Attention

    Department of Computer Science, University of Colorado Boulder

  58. AVQ-Attention: Adaptive Vector-Quantized Attention

    Jul 14, 2026Winfried van den dool, Patrick Forré, Amir Habibian +2Residual Vector QuantizationLinear Attention

    QUVA Lab, University of Amsterdam, The Netherlands · AMLab, Informatics Institute, University of Amsterdam, The Netherlands · AI4Science Lab, University of Amsterdam, The Netherlands +3

  59. Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning

    Jul 13, 2026Vishwajith RameshLinear AttentionForgetting

    Vy Labs, Inc.

  60. Is Energy Guidance All You Need? Training-Free Norm Injection for Driving World Models

    Jul 12, 2026Xiyan Su, Frank Diermeyer, Markus LienkampVideo World ModelsWorld Models

    Institute of Automotive Technology, Technical University of Munich, Munich Germany