Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model Pretraining Softmax Attention
Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Sep 4, 2025Peter RacioppoSelf-Attention Long-Context Language Modeling
Independent Researcher, Los Angeles, CA, USA.
FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel
Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3GPU Kernel Optimization GPU Acceleration
1. The Hong Kong University of Science and Technology · 2. Carnegie Mellon University
MGDFIS: Multi-scale Global-detail Feature Integration Strategy for Small Object Detection
Jun 15, 2025Yuxiang Wang, Xuecheng Bai, Chuanzhi Xu +2Remote Sensing Small Object Detection Small Object Detection
School of Computer Science, The University of Sydney, Sydney, Australia · 2Aviation Traffic Control Technology (SHENZHEN) Co., Ltd., Shenzhen, China
SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging
Jun 10, 2025Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4Self-Attention Vision Transformer
Department of Mathematics; University of California, Irvine; Irvine, CA 92697, USA · Qualcomm AI Research; San Diego, CA 92121, USA
ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation
May 27, 2025Sanghyun Jo, Wooyeol Lee, Ziseok Lee +3Diffusion Model Guidance T2I Generation
OGQ, Seoul, Korea · Seoul National University, Seoul, Korea
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta
Ordinary Least Squares as an Attention Mechanism
Apr 13, 2025Philippe Goulet CoulombeTransformer Attention Linear Regression
Département des Sciences Économiques, Université du Québec à Montréal
You Do Not Fully Utilize Transformer's Representation Capacity
Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Representation Collapse Memory-Augmented Language Models
1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University
Attention is All You Need Until You Need Retention
Jan 15, 2025M. Murat YasliogluMemory-Augmented Language Models Agent Memory Poisoning
Istanbul University, School of Business
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Oct 3, 2024Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva +5Self-Attention Multiple-Choice Question Answering
Skolkovo Institute of Science and Technology · AI Foundation and Algorithm Lab · Moscow Institute of Physics and Technology +2
GFPack++: Attention-Driven Gradient Fields for Optimizing 2D Irregular Packing
Jun 9, 2024Tianyang Xue, Lin Lu, Yang Liu +5Combinatorial Optimization Constrained Optimization
Shandong University, China · Microsoft Research Asia, China · Peking University, China
A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models
Date pendingSoyoung Oh, Vera DembergLLM Interpretability Mechanistic Interpretability
Saarland University · Max Planck Institute for Informatics
Attention by Synchronization in Coupled Oscillator Networks
Date pendingFabio Pasqualetti, Taosha GuoSoftmax Attention Dynamical Systems
Electrical Engineering and Computer Science University of California, Irvine
Relevance Is Not Permission: Localizing and Controlling Metric-Facing Attention Contributions
Date pendingMinwoo Yu, Young-guk HaFeature Attribution Evidence Selection
Smart Computing Laboratory, Department of Computer Science and Engineering, Konkuk University, 120 Neungdong-ro, Gwangjin-gu, Seoul 05029, Republic of Korea
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Date pendingShaowen Wang, Yuke Zheng, Tansheng Zhu +4Rotary Positional Embeddings Self-Attention
Tsinghua University · Hunyuan Team, Tencent · Xiongan AI Institute +1
Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
Date pendingArman Adibi, Alireza Jafari, Mohammad Ghavamzadeh +1Transformer Diffusion Sampling
School of Computer and Cyber Sciences, Augusta University · Department of Computer Science, University of Virginia · Qualcomm AI Research

Sep 12, 2025, cs.LGJ/K move · Enter open · S save

Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining

Rupert Mitchell, Kristian Kersting

Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany+1

Language Model Pretraining Softmax Attention Self-Attention Long-Context Language Modeling Transformer Attention Sparse Attention+1

Open paper AskPDF arXiv

Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs. We present Multipole Semantic Attention (MuSe), which accelerates 64k-context pretraining by 36% while matching baseline loss, requiring no architectural changes. MuSe clusters queries and keys separately in representation space. This yields query-specific summaries that substantially outperform spatial blocking at matched sparsity, while also enabling drop-in compatibility with existing pretrained models; we validate on Llama 3.1-8B and 3.2-1B without retraining. We pretrain language models up to 1B parameters at 64k context on code and scientific documents, confirming that MuSe preserves quality and long-context utilization during training.

Momentum

Related topics

Latest papers 378