Multi-Head Attention

Ask about this topic

Overview

Multi-head attention studies how Transformers use parallel query-key-value computations to select and combine information across tokens or features. Researchers probe head specialization with activation analysis, ablation, and circuit discovery, while theoretical work examines expressivity, learning dynamics, and the number of heads needed for particular computations. Architectural approaches include sparse and grouped attention, cross-head communication, and compressed projections, evaluated in settings such as long-context retrieval, language modeling, vision, and reasoning benchmarks. A central tension is that specialized heads can support useful capabilities, but their roles vary by task and model, and reducing attention cost does not always preserve accuracy.

Latest papers

CardsList

74 papers9 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

  1. ExoNet: Calibrated Multimodal Deep Learning for TESS Exoplanet Candidate Vetting using Phase-Folded Light Curves, Stellar Parameters, and Multi-Head Attention

    Apr 16, 2026Md. Rashadul IslamTime Series ClassificationAI-Assisted Scientific Research

    Dept. of Computer Science and Engineering Daffodil International University Dhaka, Bangladesh

  2. Expressivity of Transformers: A Tropical Geometry Perspective

    Apr 16, 2026Ye Su, Yong LiuTransformer ExpressivityTransformer

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · University of Chinese Academy of Sciences, Beijing 101407, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China

  3. A Multi-head-based architecture for effective morphological tagging in Russian with open dictionary

    Apr 3, 2026K. Skibin, M. Pozhidaev, S. SuschenkoMorphosyntactic TaggingMulti-Head Attention

    National Research Tomsk State University

  4. Homogenized Transformers

    Apr 2, 2026Hugo Koubbi, Borjan Geshkovski, Philippe RigolletTransformerMulti-Head Attention

  5. Tucker Attention: A generalization of approximate attention mechanisms

    Mar 31, 2026Timon Klein, Jonas Kusch, Sebastian Sager +2Tucker DecompositionAttention Mechanisms

    Department of Mathematics; Otto von Guericke University Magdeburg; 39106 Magdeburg; Germany. · Scientific Computing, Norwegian University of Life Sciences, ˚As, Norway · Max Planck Institute for Dynamics of Complex Technical Systems; 39106 Magdeburg; Germany. +1

  6. MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation

    Mar 19, 2026Zuher Jahshan, Ben Ben Ishay, Leonid YavitsAttention MechanismsMulti-Head Attention

    Bar Ilan University, Ramat Gan, Israel

  7. Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers

    Mar 9, 2026Shubham Aggarwal, Lokendra KumarWalsh-Hadamard TransformMulti-Head Attention

  8. Specialization of softmax attention heads: insights from the high-dimensional single-location model

    Mar 4, 2026M. Sagitova, O. Duranthon, L. ZdeborováSoftmax AttentionMulti-Head Attention

    Statistical physics of computation laboratory, École Polytechnique Fédérale de Lausanne, Switzerland

  9. Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads

    Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language ModelsCross-Lingual Reasoning in Language Models

    New York University

  10. Interleaved Head Attention

    Feb 24, 2026Sai Surya Duvvuri, Chanakya Ekbote, Rachit Bansal +6Multi-Head AttentionLLM Reasoning

    UT Austin · MIT · Harvard University +2

  11. Incremental Learning of Sparse Attention Patterns in Transformers

    Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-AttentionTransformer Training Dynamics

    TML Lab, EPFL

  12. Block Sparse Flash Attention

    Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU AccelerationSelf-Attention

    Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel

  13. Accelerating Attention with Basis Decomposition

    Oct 2, 2025Jialin ZhaoSelf-AttentionTransformer Attention

    Department of Computer Science Tsinghua University

  14. Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models

    Oct 3, 2024Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva +5Self-AttentionMultiple-Choice Question Answering

    Skolkovo Institute of Science and Technology · AI Foundation and Algorithm Lab · Moscow Institute of Physics and Technology +2