Self-Attention

Ask about this topic

Overview

Self-attention lets each token build a context-dependent representation by weighting information from other tokens, and is central to language, vision, audio, and multimodal Transformers. Common approaches include multi-head softmax attention with positional encodings such as RoPE, sparse and block-sparse attention, and linear or recurrent variants that reduce the cost of processing long sequences. Researchers evaluate these methods on language-model perplexity, long-context retrieval and reasoning, and vision tasks, while also studying attention heads, information flow, and phenomena such as attention sinks. A central tension is how to cut computation and memory for long contexts without losing relevant information or the accuracy and flexibility of full attention.

Latest papers

CardsList

225 papers11 papers in the last four weeks, up 22% on the four weeks before. 0.1% of all new papers.

  1. Data-Free Pruning of Self-Attention Layers in LLMs

    Dec 3, 2025Dhananjay Saikumar, Blesson VargheseLLM Inference AccelerationLLM Compression

    School of Computer Science University of St Andrews United Kingdom

  2. Unifying Convolution and Attention via Convolutional Nearest Neighbors

    Nov 18, 2025Mingi Kang, Jeová Farias Sales Rocha NetoHybrid CNN-Transformer ArchitecturesSelf-Attention

    Bowdoin College

  3. NOSA: Native and Offloadable Sparse Attention

    Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache OffloadingGPU Acceleration

    NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1

  4. Accelerating Attention with Basis Decomposition

    Oct 2, 2025Jialin ZhaoSelf-AttentionTransformer Attention

    Department of Computer Science Tsinghua University

  5. Policy Gradient with Self-Attention for Model-Free Distributed Nonlinear Multi-Agent Games

    Sep 22, 2025Eduardo Sebastián, Maitrayee Keskar, Eeman Iqbal +3Policy GradientSelf-Attention

    Department of Computer Science and Technology, University of Cambridge, UK · Department of Electrical and Computer Engineering, University of California San Diego, USA · RoPeRt group, at DIIS - I3A, Universidad de Zaragoza, Spain

  6. Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining

    Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model PretrainingSoftmax Attention

    Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1

  7. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    Sep 4, 2025Peter RacioppoSelf-AttentionLong-Context Language Modeling

    Independent Researcher, Los Angeles, CA, USA.

  8. FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel

    Aug 25, 2025Ran Yan, Youhe Jiang, Zhuoming Chen +3GPU Kernel OptimizationGPU Acceleration

    1. The Hong Kong University of Science and Technology · 2. Carnegie Mellon University

  9. Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception

    Aug 15, 2025Junjie Wang, Keyu Chen, Yulin Li +4Open-Vocabulary Object DetectionOpen-Vocabulary Semantic Segmentation

    Harbin Institute of Technology (Shenzhen) · University of Hong Kong

  10. MILAAP: Mobile Link Allocation via Attention-based Prediction

    Jun 24, 2025Yung-Fu Chen, Anish AroraSelf-AttentionWireless Communications

    The Ohio State University, Columbus, OH 43210, USA

  11. SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging

    Jun 10, 2025Nhat Thanh Tran, Fanghui Xue, Shuai Zhang +4Self-AttentionVision Transformer

    Department of Mathematics; University of California, Irvine; Irvine, CA 92697, USA · Qualcomm AI Research; San Diego, CA 92121, USA

  12. Recurrent Self-Attention Dynamics: An Energy-Agnostic Perspective from Jacobians

    May 26, 2025Akiyoshi Tomihari, Ryo KarakidaSelf-AttentionAttention Mechanisms

    Artificial Intelligence Research Center, AIST, Japan · Department of Computer Science, The University of Tokyo, Japan · RIKEN Center for Advanced Intelligence Project

  13. The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

    Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer InferenceLLM Evaluation

    University of Edinburgh · Cohere · Meta

  14. You Do Not Fully Utilize Transformer's Representation Capacity

    Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Representation CollapseMemory-Augmented Language Models

    1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University

  15. Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models

    Oct 3, 2024Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva +5Self-AttentionMultiple-Choice Question Answering

    Skolkovo Institute of Science and Technology · AI Foundation and Algorithm Lab · Moscow Institute of Physics and Technology +2