Linear Attention

Ask about this topic

Momentum

10 papers in the last four weeks, up 43% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 95

All topics
CardsList

  1. Beyond Transformers: Linear Attention Policy for Open-Vocabulary Object Goal Navigation

    Jul 21, 2026Jiahong Zhang, Yifan Lin, Yandong Zhang +6Robot NavigationLinear Attention

    Institute of Automation, Chinese Academy of Sciences, Beijing 100045, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China · Beihang University, Beijing 100191, China +2

  2. Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

    Jul 19, 2026Ayoub Ghriss, Sourav ChakrabortyEfficient AttentionLinear Attention

    Department of Computer Science, University of Colorado Boulder

  3. SpecLA: Efficient Speculative Decoding for Linear-Attention Models

    Jul 18, 2026Zhibin Wang, Xuying Han, Zhaohua Yang +5Speculative DecodingLLM Inference Acceleration

    State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China

  4. EMAGN: Efficient Multi-Attention Graph Network via Learned Clustering for Scalable Traffic Forecasting

    Jul 14, 2026Mingxing Xu, Rakesh Chowdary Machineni, Ke Liu +7Efficient Neural Network InferenceGraph Attention Networks

    Shanghai Jiao Tong University · University of Michigan, Ann Arbor · University of California, Berkeley +5

  5. Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

    Jul 8, 2026Tommaso Cerruti, Tim Rieder, George Rowlands +2Linear AttentionDelta-Rule Linear Attention

    1ETH Zurich, D-INFK · 2ETH AI Center, ETH Zurich

  6. LLT: Local Linear Transformer for PDE Operator Learning

    Jul 4, 2026Oded Ovadia, Eli TurkelPDE Surrogate ModelingTransformer Attention

    School of Mathematical Sciences, Tel Aviv University, Tel Aviv, Israel

  7. ELiTeFormer: An Efficient Transformer for FPGAs

    Jul 4, 2026Victor Agostinelli, Nicolas Bohm Agostini, Antonino TumeoLLM QuantizationTernary Quantization

    Oregon State University Corvallis, Oregon, USA · Pacific Northwest National Laboratory Richland, Washington, USA

  8. Self-Gating Attention for Efficient Time Series Forecasting

    Jul 2, 2026Dezheng Wang, Tong Chen, Wei Yuan +3Time Series ForecastingLinear Attention

    School of Automation, Southeast University, 210096, China · School of Electrical Engineering and Computer Science, The University of Queensland, QLD 4072, Australia

  9. Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

    Jul 1, 2026Peilin Liu, Ding-Xuan ZhouLinear AttentionIn-Context Learning

    School of Mathematics and Statistics, University of Sydney, NSW Australia

  10. Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

    Jun 26, 2026Haoran Zhang, Feng ZhouFourier Feature EmbeddingsLong-Context Language Modeling

    Center for Applied Statistics and School of Statistics, Renmin University of China Beijing, China

  11. CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

    Jun 25, 2026Sayak DuttaLinear AttentionDelta-Rule Linear Attention

  12. Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

    Jun 25, 2026Xiao Li, Chengruidong Zhang, Hao Luo +15Memory-Augmented Language ModelsLong-Context Language Modeling

    1Qwen Team · 2Nanjing University · 3Zhejiang University

  13. Enhancing Layer Interaction Using Key-Correlated Layer Attention

    Jun 24, 2026Jianlong Xiong, ChuanBo Xie, Le Yu +2Efficient AttentionLinear Attention

    College of Computer Science, Sichuan University, Chengdu, 610065, China · Department of Ultrasound, Zigong Maternal and Child Health Care Hospital, China

  14. LinStereo: Linear-Complexity Global Attention for Multi-Scale Iterative Stereo Matching

    Jun 24, 2026Yiran Wang, Oliver Turner, Viorela IlaMulti-Scale Feature FusionStereo Matching

    Australian Centre for Robotics (ACFR), School of Aerospace, Mechanical and Mechatronic Engineering (AMME), Faculty of Engineering, The University of Sydney, Sydney, NSW, Australia

  15. TuringViT: Making SOTA Vision Transformers Accessible to All

    Jun 23, 2026Qiman Wu, Hanlin Chen, Lyujie Chen +19Vision Foundation ModelsVision Transformer

    Foundation Model Team, Xpeng Inc.

  16. HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

    Jun 18, 2026Zhentao Tan, Wei Chen, Jingyi Shen +4Softmax AttentionLong-Context Modeling

    Alibaba Group

  17. When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers

    Jun 15, 2026Ravin Raj, Gautam ReddyTransformer InterpretabilityAdaptive Inference

    Joseph Henry Laboratories of Physics, Princeton University, Princeton, New Jersey 08544, USA

  18. Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

    Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network InitializationLinear Attention

    1The University of Sydney · 2Together AI · 5Microsoft +2

  19. Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale

    Jun 13, 2026Ang Li, Ben Liu, Bin Han +215Efficient Language Model TrainingLinear Attention

  20. Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels

    Jun 12, 2026Tiantian ZhangEfficient AttentionLinear Attention

    Columbia University

  21. Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing

    Jun 9, 2026Yongchao Huang, Hassan RazaLong-Context Language ModelingEfficient Attention

  22. Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It

    Jun 9, 2026Xinyu Zhou, Boyu Zhu, Yi Xu +4Long-Context RetrievalSupervised Fine-Tuning

    1LARK, HKUST(GZ) · 2UCL · 3Mistral AI +3

  23. Dynamic Linear Attention

    Jun 9, 2026Xin Wang, Hui Shen, Boyuan Zheng +7Long-Context Language ModelingLinear Attention

    The Ohio State University · University of Michigan · ByteDance Seed

  24. Q-Delta: Beyond Key-Value Associative State Evolution

    Jun 7, 2026Sumin Park, Seojin Kim, Noseong ParkLinear AttentionDelta-Rule Linear Attention

    Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.

  25. When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet

    Jun 4, 2026Luoming Zhang, Yuwei Ren, Kui Zhang +7Efficient Transformer InferenceLLM Inference Acceleration

    Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc.

  26. SinkRec: Mitigating Semantic State Sink in Long Sequence Recommendation with Memory-Conditioned Gated Delta Networks

    Jun 3, 2026Zhuang Zhuang, Zhipeng Wei, Ji Dai +4Memory-Augmented Neural NetworksSequential Recommendation

    Kuaishou Technology, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China · Independent Researcher

  27. Blurry Window Attention

    May 31, 2026Axel Laborieux, Christos Sourmpis, Juan Gabriel Kostelec +1Long-Context Language ModelingLinear Attention

    Huawei, Zurich, Switzerland · Huawei Advanced Computing and Storage Lab, Shenzhen, China

  28. Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

    May 31, 2026Dong Le, Thong Nguyen, Cong-Duy Nguyen +1In-Context RetrievalLong-Context Language Modeling

    Nanyang Technological University · National University of Singapore · VinUniversity

  29. Parallax: Parameterized Local Linear Attention for Language Modeling

    May 27, 2026Yifei Zuo, Dhruv Pai, Zhichen Zeng +3Language Model PretrainingSelf-Attention

    Northwestern University · Tilde Research · University of Washington

  30. Interdomain Attention: Beyond Token-Level Key-Value Memory

    May 23, 2026Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin +4Self-AttentionLong-Context Language Modeling