Multi-Head Attention

Ask about this topic

Momentum

7 papers in the last four weeks, up 17% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 49

All topics
CardsList

  1. Let the Heads Talk: Beyond Diagonal Graph Attention

    Oct 1, 2026Riccardo Ali, Alessio Borgi, Mario Severino +4Multi-Head AttentionInterleaved Graph Attention

    Department of Computer Science and Technology, University of Cambridge · Department of Computer, Control and Management Engineering, Sapienza University of Rome · Department of Information Engineering, University of Padua +1

  2. Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts

    Sep 30, 2026Dong Gyun Kang, Megha Thukral, Kwangsoo KimTransformer AttentionDynamic Attention

    College of Computing, Georgia Institute of Technology · Department of Transdisciplinary Medicine, Seoul National University Hospital

  3. QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

    Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5Key-Value Cache QuantizationMulti-Head Attention

    The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1

  4. QTrans: A Quantum Transformer for Sentiment Classification

    Sep 14, 2026Ren-Xin Zhao, Xinjie Huang, Yahong Liu +4Hybrid Quantum-Classical PipelineSentiment

    School of Computer Science, Xiangtan University, Xiangtan 411105, China · School of Electronic Information, Central South University, Changsha 410083, China · Department of Otorhinolaryngology-Head and Neck Surgery, the Third Xiangya Hospital, Central South University, Changsha 410013, China +1

  5. Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction

    Sep 8, 2026Kunhan GuoMulti-Head AttentionSpurious Correlations

    The University of Hong Kong

  6. BioSync: Transformer-Based Cross-Modal Fusion for a Multimodal Physiological Digital Biomarker

    Sep 7, 2026Seyed Mahmoud Sajjadi MohammadabadiPhysiological SignalsBiomarker

    Department of Computer Science and Engineering, University of Nevada, Reno, 1664 N. Virginia St., Reno, 89557, NV, USA · Connected Future Labs, Reno, NV, USA

  7. RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving

    Sep 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +10Multi-Head AttentionDeepseek

    Xiaohongshu Inc., China · Huawei Cloud · Shanghai Jiao Tong University +2

  8. The Head Complexity of Boolean Functions in Single-Layer Attention

    Sep 3, 2026Rajmohan Rajaraman, Ravi Sundaram, Amanuel TesfayeMulti-Head AttentionStructural Complexity

    Northeastern University.

  9. VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branch

    Sep 3, 2026WenJie FanKey-Value CacheCache

    Yotta Labs

  10. Multi-Head Self Attention is a Parameter Identification Mechanism

    Sep 1, 2026W. Ross MorrowMulti-Head AttentionLinear Attention

  11. No Equivariant Architecture Covers All Equivariant Attention

    Aug 31, 2026Tīkun ÔngMulti-Head AttentionEquivariance

  12. Toward a First-Principles Update Geometry for the Language-Model Head

    Aug 23, 2026Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2Spectral NormMulti-Head Attention

    MBZUAI

  13. When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation

    Aug 13, 2026Shuhan Wang, Yilin Luo, Nan Xu +1Post-Training QuantizationRotary Position Embedding

  14. Matrix Zonotopic Attention: A Context-Adaptive Value Projection for Set Transformers

    Aug 5, 2026Zhen Zhang, Amr AlanwarMulti-Head AttentionTransformer Architectures

    School of Computation, Information and Technology Technical University of Munich, Germany

  15. Attention-based representations for multi-task computation

    Aug 4, 2026Daniel Hsu, Mingyue XuMulti-Head AttentionMulti--Task Learning

    Columbia University · Purdue University

  16. Provably Learning Multi-Head Attention with Queries

    Aug 4, 2026Sunyeop Kim, Insung Kim, Jian GuoMulti-Head AttentionRectified Linear Unit

    Korea University, Republic of Korea · Nanyang Technological University, Singapore

  17. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    Jul 29, 2026Weiye Shi, Fanxu Meng, Muhan ZhangSpeculative DecodingLLM Inference Optimization

    Institute for Artificial Intelligence, Peking University

  18. Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

    Jul 25, 2026Dhruvil S, Fenil Sojitra, Ravirajsinh ChauhanMulti-Head AttentionDeepseek

    1Indian Institute of Technology Madras · 2P P Savani University

  19. Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings

    Jul 13, 2026Boda Xiao, Xiran Xu, Songyi Li +4Neural RecordingsDirect Feature Fusion

    Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies,Peking University, Beijing, China · Speech and Hearing Research Center, School of Intelligence Science and Technology,Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Beijing, China +1

  20. From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

    Jul 12, 2026Binbin Lin, Wei Chen, Yalun Li +3Transformer AttentionLinear Attention

    School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China

  21. Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

    Jul 7, 2026Li HengyuMulti-Head AttentionSpectral

    Institute for Solid State Physics, The University of Tokyo

  22. Dual Sparse Aggregation Transformer for Multispectral Object Detection

    Jun 30, 2026Wencong Wu, Xiuwei Zhang, Hanlin Yin +2Detection TransformerFeature Pyramid

    School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China · School of Computer Science, the Shaanxi Provincial Key Laboratory of Speech and Image Information Processing and the National Engineering Laboratory for Integrated Aerospace-Ground-Ocean Big Data Application Technology, Northwestern Polytechnical University, Xi’an 710072, China · School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China

  23. Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing

    Jun 25, 2026Shubham AggarwalMulti-Head AttentionTransformer Architectures

  24. Learning the Koopman Operator using Attention Free Transformers

    Jun 22, 2026Mohammed Nagdi, Evangelos-Marios Nikolados, Alexey Yermakov +3Koopman OperatorLatent Dynamics

    Institute for Bioengineering, School of Engineering and Centre for Engineering Biology, University of Edinburgh, Edinburgh, United Kingdom · Electrical and Computer Engineering and Applied Mathematics, University of Washington, Seattle, WA, USA · Electrical and Computer Engineering and Computer Science & Engineering, University of Washington, Seattle, WA, USA

  25. Does RoPE Prevent or Degrade Retrieval Heads? A Mechanistic Analysis Across Model Families

    Jun 19, 2026Cengizhan BayramRotary Position EmbeddingMulti-Head Attention

    Independent Researcher

  26. Conservation Laws for Modern Neural Architectures

    Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Tan Lai Ngoc +3OverparameterizationGradient

    National University of Singapore · Center for AI Research, VinUniversity · Independent Researcher +1

  27. Reconfigurable Computing Challenge: Transformer for Jet Tagging on Versal AI Engines

    Jun 16, 2026Gram Koski, Sean Lipps, Zhenghua Ma +2Transformer ArchitecturesReconfiguration

    Department of Computer Science and Engineering University of California San Diego La Jolla, CA, USA

  28. QK-Normed MLA: QK normalization without full key caching

    Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2Multi-Head AttentionBatch Normalization

    School of Data Science, The Chinese University of Hong Kong, Shenzhen.

  29. 3D Oral Modelling with Improved Vertex Distribution Using Matching-Based Learning

    Jun 6, 2026Jihun Cho, Soo-Yeon Jeong, Eun-Jeong Bae +1Occlusal ConstraintMobilenetv2

    Dept. of Computer Engineering Pai Chai University Daejeon, Korea · Division of Software Engineering Pai Chai University Daejeon, Korea · Dept. of Dental Technology Bucheon University Bucheon, Korea

  30. Riemannian Gradient Descent for Low-Rank Architectures

    Jun 1, 2026Nicholas KnightRiemannian OptimizationMulti-Head Attention

    NVIDIA

  31. CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability

    May 31, 2026Chad A. CappsRecurrent ModelTransformer Architectures

    Independent Researcher

  32. Leaf Spectral Reflectance Prediction Using Multi-Head Attention Neural Networks

    May 31, 2026Parastoo Farajpoor, Alireza Pourreza, Mohammadreza Narimani +2ReflectanceCanopy

    Digital Agriculture Laboratory, Department of Biological and Agricultural Engineering, University of California, Davis, CA, USA · Department of Botany and Plant Sciences, University of California, Riverside, CA, USA · Department of Viticulture and Enology, University of California, Davis, CA, USA

  33. TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detection

    May 31, 2026Bohan Yang, Yijun Gong, Zhi Zhang +3Citation Hallucination DetectionHallucination Detection

    Binjiang Institute of Zhejiang University · Beijing Normal-Hong Kong Baptist University · Great Bay University +1

  34. VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion

    May 28, 2026Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral +4Multi-Head AttentionKey-Value Cache

    1Virginia Tech · 2fal

  35. Hurwitz Quaternion Multiplicative Quantization for KV Cache Compression

    May 26, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +3Key-Value Cache QuantizationKey-Value Cache Compression

    Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA.

  36. Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity

    May 18, 2026Ernest FokouéMulti-Head AttentionIntrinsic Dimensionality

    School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA

  37. GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

    May 14, 2026Fanxu MengMulti-Head AttentionLarge Language Model Decoding

    Institute for Artificial Intelligence, Peking University

  38. On the global convergence of gradient descent for wide shallow models with bounded nonlinearities

    May 11, 2026Romain Petit, Clarice Poon, Gabriel PeyréGradient DescentFlat Minima

    CNRS and ENS, PSL Universit´e · Mathematics Institute, University of Warwick

  39. A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models

    May 10, 2026Djamel BouchaffraMulti-Head AttentionFree Energy Principle

    DAVID Lab, University of Paris-Saclay, UVSQ Campus, 78035 Versailles, France

  40. When and Why Grouping Attention Heads Accelerates Muon Optimization

    May 9, 2026Hongtao Zhang, Wenjie Zhou, Wei Chen +1Multi-Head AttentionMuon

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · University of Chinese Academy of Sciences

  41. Revisiting Transformer Layer Parameterization Through Causal Energy Minimization

    May 8, 2026Jin Xu, Camille Couturier, Victor Rühle +2Transformer ArchitecturesMulti-Head Attention

    Microsoft · Microsoft Research Cambridge

  42. Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention

    May 5, 2026Ayan PendharkarMulti-Head AttentionTransformer Attention

  43. Deepfake Audio Detection Using Self-supervised Fusion Representations

    May 5, 2026Khalid Zaman, Qixuan Huang, Muhammad Uzair +1Audio UnderstandingSelf-Supervised Representations

    Graduate School of Advanced Science and Technology · Japan Advanced Institute of Science and Technology · 1-1 Asahidai, Nomi, Ishikawa, Japan 923-1292

  44. Transformer as an Euler Discretization of Score-based Variational Flow

    Apr 26, 2026Huadong LiaoTransformer ArchitecturesMulti-Head Attention

  45. Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers

    Apr 26, 2026Giansalvo CirrincioneMulti-Head AttentionTransformer Architectures

    aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France

  46. Adaptive Head Budgeting for Efficient Multi-Head Attention

    Apr 24, 2026Bilal Faye, Abdoulaye Mbaye, Hanane Azzag +1Multi-Head AttentionTransformer Architectures

  47. Expressivity of Transformers: A Tropical Geometry Perspective

    Apr 16, 2026Ye Su, Yong LiuTransformer AttentionMulti-Head Attention

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · University of Chinese Academy of Sciences, Beijing 101407, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China

  48. FGAA-FPN: Foreground-Guided Angle-Aware Feature Pyramid Network for Oriented Object Detection

    Feb 11, 2026Jialin MaFeature PyramidObject Detection

    International school of BUPT, Beijing University Of Posts And Telecommunications, Beijing,China.

  49. Attention Enhanced Entity Recommendation for Intelligent Monitoring in Cloud Systems

    Oct 23, 2025Fiza Husain, Anson Bastos, Anjaly Parayil +4CloudLong-Range Temporal Dependencies

    Microsoft