Multi-Head Attention

Follow this topic Ask about this topic

Multi-head attention studies how Transformers use parallel query-key-value computations to select and combine information across tokens or features. Researchers probe head specialization with activation analysis, ablation, and circuit discovery, while theoretical work examines expressivity, learning dynamics, and the number of heads needed for particular computations. Architectural approaches include sparse and grouped attention, cross-head communication, and compressed projections, evaluated in settings such as long-context retrieval, language modeling, vision, and reasoning benchmarks. A central tension is that specialized heads can support useful capabilities, but their roles vary by task and model, and reducing attention cost does not always preserve accuracy.

Latest papers

74 papers9 papers in the last four weeks, up 29% on the four weeks before. 0.1% of all new papers.

Multi-Head Attention-Based Feature Extractor Integration with Soft Actor-Critic for Porosity Prediction and Process Parameter Optimization in Additive Manufacturing
Jun 18, 2026Kianoush Aqabakee, Leonardo StellaMaterials Property Prediction Soft Actor-Critic
Department of Electrical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran, and Department of Mechanical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran · School of Computer Science, University of Birmingham, Birmingham B15 2TT, United Kingdom
Explaining Attention with Program Synthesis
Jun 17, 2026Amiri Hayes, Belinda Z Li, Jacob AndreasTransformer Interpretability Attention Head Analysis
NJIT · MIT EECS
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh +2Rotary Positional Embeddings Transformer
National University of Singapore · Center for AI Research, VinUniversity · Technical University of Munich +1
Conservation Laws for Modern Neural Architectures
Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Tan Lai Ngoc +3Neural Network Training Dynamics Multi-Head Attention
National University of Singapore · Center for AI Research, VinUniversity · Independent Researcher +1
QK-Normed MLA: QK normalization without full key caching
Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2KV Caching Transformer Attention
School of Data Science, The Chinese University of Hong Kong, Shenzhen.
Where Computation Lives Inside TabPFN: Causal Localisation of Attention Head Function
Jun 11, 2026Atharva Gupta, Dhruv Kumar, Murari Mandal +1Attention Head Specialization Attention Head Analysis
Department of Computer Science, Birla Institute of Technology and Science, Pilani, India · School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India · Birla AI Labs, Office of Ananya Birla, Aditya Birla Group, India
Closure-Validated Circuit Discovery in Attention Heads: Co-activation Proposes, Ablation Disposes
Jun 8, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer Interpretability Vision Transformer
Kempner Institute Harvard University Cambridge, MA
How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models
Jun 5, 2026Hongxing Wang, Harenome Razanajato, Zhen Zhang +2Grouped-Query Attention Long-Context Language Modeling
When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
Jun 1, 2026Yongzhong XuLanguage Model Pretraining Transformer Interpretability
Riemannian Gradient Descent for Low-Rank Architectures
Jun 1, 2026Nicholas KnightRiemannian Optimization Low-Rank Approximation
NVIDIA
Leaf Spectral Reflectance Prediction Using Multi-Head Attention Neural Networks
May 31, 2026Parastoo Farajpoor, Alireza Pourreza, Mohammadreza Narimani +2Agricultural Remote Sensing Remote Sensing
Digital Agriculture Laboratory, Department of Biological and Agricultural Engineering, University of California, Davis, CA, USA · Department of Botany and Plant Sciences, University of California, Riverside, CA, USA · Department of Viticulture and Enology, University of California, Davis, CA, USA
Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization
May 29, 2026Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3Rotary Positional Embeddings Attention Head Analysis
CENIA & Faculty of Mathematics UC Santiago, Chile · CENIA Santiago, Chile · IMC UC & CENIA Santiago, Chile
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity
May 18, 2026Ernest FokouéSelf-Attention Ensemble Learning
School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Self-Attention Long-Context Language Model Inference
Nanjing University · Alibaba Group
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
May 15, 2026Alex Massucco, Leonardo Del Grande, Marcello Carioni +2Transformer Transformer Training Dynamics
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · Department of Mathematics, University of Twente, Enschede, Netherlands
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaMulti-Head Attention Self-Attention
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089
Lower bounds for one-layer transformers that compute parity
May 12, 2026Daniel HsuTransformer Expressivity Self-Attention
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Attention Approximation Softmax Attention
Apple
A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models
May 10, 2026Djamel BouchaffraLLM Pruning Attention Head Analysis
DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France
When and Why Grouping Attention Heads Accelerates Muon Optimization
May 9, 2026Hongtao Zhang, Wenjie Zhou, Wei Chen +1Attention Head Analysis Muon Optimizer
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · University of Chinese Academy of Sciences
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
May 7, 2026Amrit NidhiEfficient Transformer Inference Self-Attention
Independent Researcher
A Multi-Head Attention Approach for SLA Compliance Monitoring in Data Centers
May 6, 2026Omanshu ThapliyalMulti-Head Attention
Researcher, Strategic Data Solutions Lab Hitachi America Ltd. Santa Clara, USA
Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention
May 5, 2026Ayan PendharkarSoftmax Attention Attention Head Analysis
Borrowed Geometry: Cross-Distribution Head-Importance Fingerprints of Frozen Pretrained Gemma 4 31B
May 1, 2026Abay BektursunCross-Modal Learning Attention Head Analysis
Independent research
Attention Is Where You Attack
Apr 30, 2026Aviral Srivastava, Sourav PandaSelf-Attention Adversarial Attacks
Amazon · The Pennsylvania State University
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
Apr 26, 2026Peize He, Yaodi Luo, Xiaoqian Liu +7Audio Token Compression Self-Attention
1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2
Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
Apr 26, 2026Giansalvo CirrincioneTransformer Interpretability Representation Collapse
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France
HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
Apr 19, 2026Juyuan Wang, Chenxing Wang, Yuchen Fang +8Attention Head Analysis Self-Attention
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China

Jun 18, 2026J/K move · Enter open · S save

Multi-Head Attention-Based Feature Extractor Integration with Soft Actor-Critic for Porosity Prediction and Process Parameter Optimization in Additive Manufacturing

Kianoush Aqabakee, Leonardo Stella

Department of Electrical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran, and Department of Mechanical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran · School of Computer Science, University of Birmingham, Birmingham B15 2TT, United Kingdom

Materials Property Prediction Soft Actor-Critic Additive Manufacturing Multi-Head Attention

Open paper AskPDF arXiv

Additive manufacturing process optimization requires precise parameter control to minimize defects such as porosity. Traditional reinforcement learning (RL) approaches using discrete action spaces suffer from slow convergence and susceptibility to local optima, limiting their effectiveness for high-precision manufacturing tasks. This study addresses these limitations by employing a continuous action space combined with a novel architecture that integrates a multi-head attention mechanism with the Soft Actor-Critic (SAC) algorithm. The attention-based feature extractor enhances the agent's ability to capture subtle variations in low-dimensional input features, enabling more effective exploration-exploitation balance for navigating value spaces with local minima. We validate our approach on porosity prediction and process parameter optimization in laser powder bed fusion, demonstrating faster convergence and higher final reward values compared to standard RL methods including DQN, PPO, TD3, and vanilla SAC. The proposed methodology achieves a convergence value of 322.79 within 14 episodes, outperforming existing approaches while maintaining stability throughout training.