Linear Attention

Ask about this topic

Momentum

25 papers in the last four weeks, up 92% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 183

All topics
CardsList

  1. HierEdit: Region-Aware Hierarchical Diffusion for Efficient High-Resolution Editing

    May 17, 2026Yuyao Zhang, Alexander Huang-Menders, Yu-Wing TaiDiffusion-Based Image EditingImage Editing

    Dartmouth College

  2. CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection

    May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1PrefillLinear Attention

    Seoul National University

  3. Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion

    May 15, 2026Kunyang Li, Mubarak Shah, Yuzhang ShangAutoregressive Video Diffusion ModelsSpatio-Temporal Attention

    Institute of Artificial Intelligence, University of Central Florida

  4. Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

    May 15, 2026Shuaiyi Li, Zhisong Zhang, Yan Wang +5Dynamic Sparse AttentionLinear Attention

    The Chinese University of Hong Kong · City University of Hong Kong · Tencent +2

  5. WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer

    May 14, 2026Caoliwen Wang, Minghao Guo, Siyuan Chen +13Physics SimulationParticle Dynamics

    University of British Columbia, Canada · MIT CSAIL, USA · Georgia Institute of Technology, USA +7

  6. Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement

    May 14, 2026Hengjie Liu, Zhenya Zhang, Jianjun ZhaoTransformer ArchitecturesRectified Linear Unit

    Kyushu University, Fukuoka, Japan · National Institute of Informatics, Tokyo, Japan

  7. OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention

    May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Kimi Delta AttentionLinear Attention

    Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1

  8. Lower bounds for one-layer transformers that compute parity

    May 12, 2026Daniel HsuLinear AttentionRectified Linear Unit Networks

  9. Composition of Memory Experts for Diffusion World Models

    May 12, 2026Sebastian Stapf, Pablo Acuaviva Huertos, Aram Davtyan +1World ModelsDiffusion Models

    Computer Vision Group Department of Computer Science University of Bern

  10. Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

    May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Neural NetworksTransformer Attention

    School of Future Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300354, China · School of Artificial Intelligence, Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300354, China

  11. Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

    May 11, 2026Vishal Pandey, Gopal SinghKimi Delta AttentionLinear Attention

    Independent Researcher London, UK · Metriqual Athens, GR

  12. Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

    May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer EncoderTransformer Architectures

    Department of Mathematics Friedrich-Alexander-University Erlangen-Nürnberg · Institute of Mathematics CAIDAS University of Würzburg · Mathematical Institute University of Oxford Reuben College +1

  13. Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions

    May 11, 2026Diancheng Kang, Zheyuan Liu, Ningshan Ma +3Linear Activation SteeringModel Activations

    1Southern University of Science and Technology · University of Notre Dame · 3Massachusetts Institute of Technology

  14. Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition

    May 11, 2026Haoren Xu, Guanhua FangLinear AttentionIn-Context Learning

    Fudan University

  15. Explanation-Aware Learning for Enhanced Interpretability in Biomedical Imaging

    May 11, 2026Zubair Faruqui, Rahul DubeyInterpretabilityGradient-Weighted Class Activation Mapping

    Department of Computer Science, Missouri State University

  16. Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging

    May 11, 2026Guisong Liu, Xin Gao, Martin Dresler +2SleepSpatio-Temporal Transformers

    School of Biological Science and Medical Engineering Southeast University Nanjing, China · University of Bath Bath, England · Donders Institute for Brain, Cognition and Behaviour Radboud University Medical Center Nijmegen, The Netherlands +1

  17. Scaling Limits of Long-Context Transformers

    May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Linear AttentionEfficient Long-Context Inference

  18. Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning

    May 8, 2026Zixuan Xie, Xinyu Liu, Claire Chen +3In-Context LearningLinear Attention

    University of Virginia · California Institute of Technology · Purdue University

  19. Federated Nested Learning: Collaborative Training of Self-Referential Memories for Test-Time Adaptation

    May 8, 2026Hong Chen, Pengcheng Wu, Yuanguo Lin +4Federated LearningParameter-Efficient Adaptation

    Nanyang Technological University · Jimei University · Xiamen University

  20. Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

    May 7, 2026Chenyang Zhang, Yuan CaoIn-Context LearningTransformer Architectures

    School of Computing & Data Science, The University of Hong Kong

  21. Metonymy in vision models undermines attention-based interpretability

    May 7, 2026Ananthu Aniraj, Cassio F. Dantas, Dino Ienco +2InterpretabilityVision Foundation Models

    Inria, EVERGREEN, University of Montpellier, 34090 Montpellier, France · INRAE, UMR TETIS, University of Montpellier, 34090 Montpellier, France · University of Trento, Trento, Italy

  22. MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

    May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Kimi Delta AttentionLinear Attention

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

  23. On the Role of Language Representations in Auto-Bidding: Findings and Implications

    May 7, 2026Guanyu Zhu, Jining Luan, Hanwen Du +11BiddingLinear Attention

    City University of Hong Kong · South China Agricultural University · University of Electronic Science and Technology of China +11

  24. Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

    May 6, 2026Dominik Dahlem, Diego Maniloff, Mac MisiuraLinear AttentionSpectral

    Red Hat AI

  25. Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks

    May 5, 2026Yaobo ZhangPositional EncodingRelative Position

    School of Physics, Ningxia University Yinchuan, 750021, China

  26. Cascade Token Selection for Transformer Attention Acceleration

    May 4, 2026Stephen J. ThomasTransformer AttentionLinear Attention

    Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015

  27. Linearizing Vision Transformer with Test-Time Training

    May 4, 2026Yining Li, Dongchen Han, Zeyu Liu +3Self-Supervised Vision TransformersVision Transformer

    Tsinghua University, Beijing, China

  28. PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting

    Apr 30, 2026Hira Saleem, Flora Salim, Cormac PurcellNumerical Weather PredictionParametric Physics-Informed Neural Network

    University of New South Wales, Sydney, NSW, Australia

  29. From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data

    Apr 29, 2026Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa NaeemLarge Language Model HallucinationLarge Language Models Fail

    *Dept. of Computer Science and Engineering, Metropolitan University, Sylhet, Bangladesh

  30. AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving

    Apr 28, 2026Zhongkai Yu, Haotian Ye, Chenyang Zhou +9Graphics Processing Unit MemoryLinear Attention

    University of California, San Diego · Columbia University · Yonsei University +2

  31. Emergent Self-Attention from Astrocyte-Gated Associative Memory Dynamics

    Apr 28, 2026Arnau Vivet, Alex ArenasHopfield NetworksNeurons

    1Departament d’Enginyeria Inform`atica i Matem`atiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain · 2Complexity Science Hub Vienna, Metternichgasse 8, 1030 Vienna, Austria

  32. HBGSA: Hydrogen Bond Graph with Self-Attention for Drug-Target Binding Affinity Prediction

    Apr 25, 2026Junxiao Kong, Chupei Tang, Di Wang +4Interleaved Graph AttentionLinear Attention

    School of Mathematics and Statistics, Lanzhou University, 222 South Tianshui Road, 730000, Gansu, China · Department of Computer Science, City University of Hong Kong, 83 Tat Chee Avenue, Kowloon Tong, Hong Kong 999077, China

  33. LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel

    Apr 22, 2026Zhe Feng, Sen Lian, Changwei Wang +5Linear AttentionTransformer Architectures

    1MAIS,Institute of Automation, Chinese Academy of Sciences · usr 3China Electronics Data Corporation · 5The Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology +4

  34. Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling

    Apr 21, 2026Weijie Zhao, Mingquan Liu, Bolun Wang +4Linear AttentionLanguage Modeling

    Luxi Tech, Shenzhen, China · The School of Computer Science, China University of Geosciences, 388 Lumo Road, Wuhan, 430074, Hubei, China.

  35. Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization

    Apr 21, 2026Andrei Andrusenko, Vladimir Bataev, Lilit Grigoryan +3Automatic Speech RecognitionStreaming

    NVIDIA, Armenia · NVIDIA, USA

  36. FG2^2-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control

    Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Kimi Delta AttentionLinear Attention

    Meituan · Beijing, China

  37. Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning

    Apr 20, 2026Manuel Israel CazaresMathematical ReasoningLLM Reasoning Strategies

    Bytepro AI Mazatlán, Sinaloa, Mexico

  38. Advancing Vision Transformer with Enhanced Spatial Priors

    Apr 20, 2026Qihang Fan, Huaibo Huang, Mingrui Chen +2Self-Supervised Vision TransformersVision Transformer

    MAIS & NLPR, Institude of Automation, Chinese Academic of Science, Beijing 100190, China · School of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China

  39. OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models

    Apr 20, 2026Yiwei Zhang, Xuesong Chen, Jin Gao +5Autonomous Hard Drive DisassemblyAutoregressive Language Models

    State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University +2

  40. Robust Diabetic Retinopathy Grading Using Dual-Resolution Attention-Based Deep Learning with Ordinal Regression

    Apr 19, 2026Afshan HashmiRetinal ImagingFundus Images

    Tuwaiq Academy, Tuwaiq Research and Development Centre, Riyadh, Saudi Arabia

  41. Capacity-Controlled Global Attention for Graph Transformers

    Apr 19, 2026Yang Liu, Dongxin Guo, Tom Zheng +3Transformer AttentionLinear Attention

    Brain Investing Limited Hong Kong, China · The University of Hong Kong Hong Kong, China · Stellaris AI Limited Hong Kong, China

  42. CAM3DNet: Comprehensively mining the multi-scale features for 3D Object Detection with Multi-View Cameras

    Apr 18, 2026Mingxi Pang, Dingheng Wang, Zekun Li +43D Object DetectionMulti-View

    College of Intelligent Science and Technology, National University of Defense Technology, Changsha, 410073, China · Northwest Institute of Mechanical & Electrical Engineering, Xianyang, 712099, China. · School of Aerospace Engineering, Xi’an Jiaotong University, Xi’an, 710049, China

  43. Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information

    Apr 17, 2026Yao Chen, Jiawei Sheng, Wenyuan Zhang +1Dataset DistillationReasoning Skills

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences

  44. LACE: Lattice Attention for Cross-thread Exploration

    Apr 16, 2026Yang Li, Zirui Zhang, Yang Liu +1LLM Reasoning StrategiesCross-Task Interference

    Department of Computer Science Rutgers University · Amazon AGI Labs

  45. Invertible Query-Key Coupling Composes with Attention Mechanisms

    Apr 2, 2026Barak Gahtan, Alex M. BronsteinLinear AttentionKeys And Value

    Technion – Israel Institute of Technology · ISTA – Institute of Science and Technology Austria

  46. Beyond Acoustic Prefixes: Persistent Access to Serialized Acoustic Memory for LLM-Based Multi-Talker Speech Recognition

    Mar 28, 2026Hao Shi, Yuan Gao, Xugang Lu +1AcousticLinear Attention

    Graduate School of Informatics, Kyoto University, Kyoto, Japan · National Institute of Information and Communications Technology, Japan

  47. The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle

    Mar 18, 2026Dibakar SigdelTransformer AttentionEnergy-Based Transformer

    Mindverse Computing LLC, WA 98087

  48. Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention

    Mar 6, 2026Haiqing Hao, Zhipeng Sui, Rong Zou +4Object DetectionDynamic Sparse Attention

    State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China · Robotics and Perception Group, University of Zurich, Zurich, Switzerland · ShanghaiTech University, Shanghai, China

  49. Central Dogma Transformer II: An AI Microscope for Understanding Cellular Regulatory Mechanisms

    Feb 9, 2026Nobuyuki OtaRegulatory NetworksRegulation

    Independent Researcher, Burlingame, CA, USA

  50. Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences

    Feb 6, 2026Alessio Pecilli, Matteo RosatiVariational Quantum AlgorithmsTransformer Attention

    Dipartimento di Ingegneria Civile, Informatica e delle Tecnologie Aeronautiche, Universit`a degli Studi Roma Tre, Via della Vasca Navale 79, 00146 Rome, Italy

  51. Poly-attention: a general scheme for higher-order self-attention

    Feb 2, 2026Sayak Chakrabarti, Toniann Pitassi, Josh AlmanLinear AttentionTransformer Architectures

    Computer Science, Columbia University, New York, NY 10027.

  52. EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

    Dec 25, 2025Sanghyun Jo, Donghwan Lee, Eunji Jung +2Video Object RemovalDiverse Occlusion-And-Revelation Scenarios

    OGQ, Seoul, Korea · Seoul National University, Seoul, Korea

  53. Data-Free Pruning of Self-Attention Layers in LLMs

    Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLarge Language Model CompressionLinear Attention

    School of Computer Science University of St Andrews United Kingdom

  54. Unifying Convolution and Attention via Convolutional Nearest Neighbors

    Nov 18, 2025Mingi Kang, Jeová Farias Sales Rocha NetoConvolutional Block Attention ModuleConvolutional Neural Networks

    Bowdoin College

  55. Controllably Efficient Language Models

    Nov 7, 2025Jatin Prakash, Aahlad Puli, Rajesh RanganathTransformer ArchitecturesLinear Attention

    New York University

  56. How Can Mamba Learn In Context with Outliers and Generalize Provably?

    Oct 1, 2025Hongkang Li, Songtao Lu, Xiaodong Cui +2Mamba-Based ModelsIn-Context Learning

    Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Sha Tin, New Territories, Hong Kong · IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA +1

  57. Policy Gradient with Self-Attention for Model-Free Distributed Nonlinear Multi-Agent Games

    Sep 22, 2025Eduardo Sebastián, Maitrayee Keskar, Eeman Iqbal +3Multi-Agent Reinforcement LearningPolicy Gradient

    Department of Computer Science and Technology, University of Cambridge, UK · Department of Electrical and Computer Engineering, University of California San Diego, USA · RoPeRt group, at DIIS - I3A, Universidad de Zaragoza, Spain

  58. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    Sep 4, 2025Peter RacioppoLinear AttentionState Estimation

    Independent Researcher, Los Angeles, CA, USA.

  59. SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging

    Jun 10, 2025Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4Linear AttentionMamba-Based Models

    Department of Mathematics; University of California, Irvine; Irvine, CA 92697, USA · Qualcomm AI Research; San Diego, CA 92121, USA

  60. Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention

    May 22, 2025Yuang AiImage RestorationSelf-Supervised Vision Transformers

    University of Chinese Academy of Sciences, Beijing, China