Multi-Head Attention

Follow this topic Ask about this topic

Multi-head attention studies how Transformers use parallel query-key-value computations to select and combine information across tokens or features. Researchers probe head specialization with activation analysis, ablation, and circuit discovery, while theoretical work examines expressivity, learning dynamics, and the number of heads needed for particular computations. Architectural approaches include sparse and grouped attention, cross-head communication, and compressed projections, evaluated in settings such as long-context retrieval, language modeling, vision, and reasoning benchmarks. A central tension is that specialized heads can support useful capabilities, but their roles vary by task and model, and reducing attention cost does not always preserve accuracy.

Latest papers

74 papers9 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

Head Dimension Matters in Language Models for Long-Context Retrieval
Oct 23, 2026Seiya Tokui, Shoichiro Yamaguchi, Sosuke KobayashiLong-Context Retrieval Attention Head Analysis
Preferred Networks, Inc., Tokyo, Japan
Formally Verified Autoresearch for Analyzing the Expressivity of Multi-Head Attention
Oct 18, 2026Karthik Viswanathan, Plamen Dimitrov, Stefano Rocca +1Transformer Expressivity Multi-Head Attention
Verified Mechanisms · MATS Research · University of California, Berkeley
Can Capability Enhancement be Localized?
Oct 15, 2026Anh Tuan Nguyen, Anna Bair, Zico KolterAttention Head Analysis Multi-Head Attention
Carnegie Mellon University
Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale Study
Oct 14, 2026Arjun Boya, Charan Sai Bandaru, Masthanaiah Pillimetla +1Multi-Head Attention
Institute of Cognitive Science University of Osnabrück · Department of Social Sciences and Technology Technical University of Munich
Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
Oct 9, 2026Claudio AltafiniSelf-Attention Attention Mechanisms
Division of Automatic Control, Dept. of Electrical Engineering, Linköping University, SE-58183, Linköping, Sweden
Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers
Oct 9, 2026Zhenke Duan, Xin Li, Jiqun Pan +3Multi-Head Attention Transformer
School of Statistics and Mathematics, Zhongnan University of Economics and Law, Wuhan, China · CodeSoul.Co, Hangzhou, China · Innovation and Talent Base for Digital Technology and Finance, ZUEL +1
Gaussian Equivalence for Multi-Head Self-Attention
Oct 7, 2026Tomohiro Hayase, Ryo KarakidaMulti-Head Attention
Artificial Intelligence Research Center (AIRC), AIST · RIKEN AIP
Let the Heads Talk: Beyond Diagonal Graph Attention
Oct 1, 2026Riccardo Ali, Alessio Borgi, Mario Severino +4Graph Attention Networks Sheaf Neural Networks
Department of Computer Science and Technology, University of Cambridge · Department of Computer, Control and Management Engineering, Sapienza University of Rome · Department of Information Engineering, University of Padua +1
Learned Queries and Keys Are All You Need: Replacing the Value Projection with Structured Transforms
Sep 29, 2026Ene Meco, Emadeldeen Hamdan, A. Enis CetinTransformer Attention Efficient ViTs
University of Illinois Chicago, Chicago, IL 60607
MorphAtt: A Neuromorphic Accelerator for Efficient Multi-Head Attention Processing in Spiking Vision Transformers
Sep 27, 2026Rachmad Vidya Wicaksana Putra, Amirhesam Jafari Rad, Muhammad ShafiqueAI Accelerator Inference Spiking Transformers
eBRAIN Lab, New York University (NYU) Abu Dhabi, Abu Dhabi, United Arab Emirates · University of Tehran, Tehran, Iran
CoWindow Attention: Full Causal Coverage Is a Collective Property
Sep 26, 2026Jingze Shi, Zhangyang Peng, Xianduo Li +6Efficient Attention Causal Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Beijing Academy of Artificial Intelligence, Beijing, China. · Université Paris Cité, Paris, France.
Transformer Heads Looking for Order
Sep 22, 2026Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser +4Transformer Expressivity Multi-Head Attention
IMC UC · CENIA · Centre for Credible AI +1
The Head Complexity of Boolean Functions in Single-Layer Attention
Sep 3, 2026Rajmohan Rajaraman, Ravi Sundaram, Amanuel TesfayeSelf-Attention Attention Mechanisms
Northeastern University.
Multi-Head Self Attention is a Parameter Identification Mechanism
Sep 1, 2026W. Ross MorrowSoftmax Attention Parameter Identifiability
Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras
Sep 1, 2026Jiming Feng, Junliang LiAttention Head Analysis Transformer Attention
College of Computer Science, Beijing University of Technology Beijing, China · School of Mathematics, Statistics and Mechanics, Beijing University of Technology Beijing, China
Inherent Trade-offs when Enforcing Correct Data Symmetries in Multi-head Self-attention
Aug 31, 2026Tīkun Ông, Georg BökmanAttention Head Analysis Equivariant Neural Networks
Mahalanobis-Based Multi-Head Attention for Complex State Propagation
Aug 25, 2026Xiaohe LiAttention Mechanisms Multi-Head Attention
GuangDong Police College
Matrix Zonotopic Attention: A Context-Adaptive Value Projection for Set Transformers
Aug 5, 2026Zhen Zhang, Amr AlanwarSet Prediction Transformer Attention
School of Computation, Information and Technology Technical University of Munich, Germany
Attention-based representations for multi-task computation
Aug 4, 2026Daniel Hsu, Mingyue XuRepresentation Learning Multi-Task Learning
Columbia University · Purdue University
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Aug 4, 2026Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2Language Model Pretraining Long-Context Retrieval
Institute for Applied Informatics at Leipzig University (InfAI) · ScaDS.AI Dresden/Leipzig · Seltz +1
Provably Learning Multi-Head Attention with Queries
Aug 4, 2026Sunyeop Kim, Insung Kim, Jian GuoSoftmax Attention Transformer
Korea University, Republic of Korea · Nanyang Technological University, Singapore
Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
Jul 25, 2026Dhruvil S, Fenil Sojitra, Ravirajsinh ChauhanTransformer Interpretability Low-Rank Attention
1Indian Institute of Technology Madras · 2P P Savani University
Multi-Head Attention Residuals
Jul 22, 2026Cheng Luo, Zefan Cai, Junjie HuTransformer Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
Jul 12, 2026Binbin Lin, Wei Chen, Yalun Li +3Self-Attention Transformer Attention
School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China
Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention
Jul 7, 2026Li HengyuRotary Positional Embeddings Attention Head Analysis
Institute for Solid State Physics, The University of Tokyo
Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Jul 1, 2026Aryo Pradipta Gema, Beatrice Alex, Pasquale MinerviniLong-Context Retrieval Attention Head Analysis
QUniversity of Edinburgh · KHeriot-Watt University · V Miniml.AI
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
Jun 25, 2026Shubham AggarwalHierarchical Representation Learning Transformer Attention
DnA: Denoising Attention for Visual Tasks
Jun 25, 2026Ron Campos, Subhajit Maity, Xin Li +2Visual Attention Vision Transformer
University of Central Florida, USA · University of North Carolina at Charlotte, USA
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Jun 18, 2026Vishesh Tripathi, Abhay KumarGrouped-Query Attention Transformer Attention
FrontiersMind
HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization
Jun 18, 2026Zhentao Tan, Wei Chen, Jingyi Shen +4Softmax Attention Long-Context Modeling
Alibaba Group

Oct 23, 2026J/K move · Enter open · S save

Head Dimension Matters in Language Models for Long-Context Retrieval

Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi

Preferred Networks, Inc., Tokyo, Japan

Long-Context Retrieval Attention Head Analysis Multi-Head Attention Long-Context Language Modeling

Open paper AskPDF arXiv

Multi-head attention retrieves and combines context representations through query--key interactions and value aggregation, with head dimension constraining the per-head representation space. In long-context processing, it remains unclear whether allocating a fixed parameter budget to fewer, wider heads improves information retrieval. In this paper, we investigate how head dimension affects the performance of long-context retrieval tasks. Our empirical studies reveal that pretraining loss and short-context non-retrieval task performance are not affected much by head dimension, while long-context retrieval performance improves with wider heads at fixed model sizes. We find that models with wider heads have a higher proportion of retrieval heads, associating more parameters with retrieval. Our ablation studies also show that the retrieval performance improves with wider heads even when the projection weights in attention modules are frozen during long-context continual pretraining, suggesting that wider heads not only benefit updates in attention modules but lead to more effective adaptation of components such as embeddings and MLPs.