Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics

Improved Belief-Attention in Vision Task
May 22, 2026Guoqiang ZhangVisual Attention Self-Attention
University of Exeter
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
May 21, 2026Hangyue Zhao, Paul Caillon, Erwan Fagnou +1Self-Attention Structured Sparsity
ESPCI PSL, Paris, France · LAMSADE, Université Paris Dauphine - PSL, Paris, France
Can Transformers Learn to Verify During Backtracking Search?
May 21, 2026Yin Jun Phua, Tony Ribeiro, Tuan Nguyen +1Tree Search Transformer Attention
Institute of Science Tokyo, 2-12-1 Ookayama, Meguro-ku, Tokyo 152-8550, Japan · Centrale Nantes, CNRS, Laboratoire des Sciences du Num´erique de Nantes, LS2N, UMR 6004, F-44000 Nantes, France · National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan +2
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Musical Attention Transformer: Music Generation Using a Music-Specific Attention Model
May 20, 2026Shinnosuke Takasuka, Hideo MukaiSymbolic Music Generation Transformer Attention
Department of Computer Science, School of Science and Technology, Meiji University, 1-1-1 Higashimita, Tama-ku, Kawasaki, Kanagawa 214-8571, Japan
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
The General Theory of Localization Methods
May 20, 2026Congwei SongTransformer Attention Local Learning
Beijing Institute of Mathematical Sciences and Applications
Multi-Head Attention as Ensemble Nadaraya-Watson Estimation: Variance Reduction, Decorrelation, and Optimal Head Diversity
May 18, 2026Ernest FokouéSelf-Attention Ensemble Learning
School of Mathematics and Statistics, College of Science Rochester Institute of Technology Rochester, New York 14623, USA
Function graph transformers universally approximate operators between function spaces
May 18, 2026Takashi Furuya, David Mis, Ivan Dokmanić +2Transformer Attention Neural Network Approximation Theory
Doshisha University, RIKEN AIP · Rice University · University of Basel +2
InfoFlow: A Framework for Multi-Layer Transformer Analysis
May 18, 2026Penghao Yu, Haotian Jiang, Zeyu Bao +1Transformer Expressivity Transformer
Department of Mathematics National University of Singapore · Institute for Functional Intelligent Materials National University of Singapore · National University of Singapore +1
Hamiltonian-Inspired Attention Mechanism for Scalable RF Transmitter Fingerprinting
May 17, 2026Chitraksh Singh, Monisha Dhanraj, Akram SheriffTransformer Attention Physics-Informed ML
Frondeur Labs · Cisco Systems
VGGT-Occ: Geometry-Grounded and Density-Aware Gated Fusion for 3D Occupancy Prediction
May 16, 2026Xun Chen, Tianchen Deng, Rui Wang +53D Semantic Segmentation 3D Occupancy Prediction
Nanyang Technological University · Shanghai Jiao Tong University · ETH Zurich
Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
May 15, 2026Alex Massucco, Leonardo Del Grande, Marcello Carioni +2Transformer Wasserstein Gradient Flows
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK · Department of Mathematics, University of Twente, Enschede, Netherlands
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
May 15, 2026Yufeng Du, Phillip Harris, Minyang Tian +5Rotary Positional Embeddings Self-Attention
University of Illinois at Urbana-Champaign · USA · University of Bonn +3
Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
May 14, 2026Mahdi Naser Moghadasi, Faezeh GhaderiTransformer Inference Transformer Attention
Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX · University of Texas at Arlington, Arlington, TX
GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding
May 14, 2026Fanxu Meng, Zhixin PanGrouped-Query Attention GPU Acceleration
Institute for Artificial Intelligence, Peking University
Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility
May 13, 2026Gergely Szilvasy, Manuel Faysse, Maria Lomeli +5Self-Attention KV Caching
Meta FAIR · MICS, CentraleSupélec
Delta Attention Residuals
May 13, 2026Cheng Luo, Zefan Cai, Junjie HuAttention Residual Connections Residual Learning
Independent Researcher · University of Wisconsin–Madison
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Transformer Attention Linear Attention
Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1
ECG-NAT: A Self-supervised Neighborhood Attention Transformer for Multi-lead Electrocardiogram Classification
May 13, 2026Mahsa Gazeran, Sayvan Soleymanbaigi, Fatemeh Daneshfar +2Self-Supervised Time Series Representation Learning Transformer Attention
Department of Computer Engineering, University of Kurdistan, Sanandaj, Iran · Department of Mathematics and Operational Research, University of Mons, Mons, Belgium
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaAttention Head Analysis Self-Attention
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089
Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores
May 12, 2026Alan Z. Song, Yinjie Chen, Mu Nan +3Visual Representation Learning Vision Transformer
Carnegie Mellon University · University of Hong Kong
Lower bounds for one-layer transformers that compute parity
May 12, 2026Daniel HsuTransformer Expressivity Self-Attention
Bin Latent Transformer (BiLT): A shift-invariant autoencoder for calibration-free spectral unmixing of turbid media
May 12, 2026Martin HohmannAutoencoders Transformer Attention
Institute of Photonic Technologies (LPT), Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Konrad-Zuse-Straße 3/5, 91052 Erlangen, Germany
Inducing Spatial Locality in Vision Transformers through the Training Protocol
May 11, 2026Eduardo Santiago Toledo, Asael Fabian MartínezVision Transformer Transformer Attention
Universidad Autónoma Metropolitana
Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
May 11, 2026Albert Alcalde, Leon Bungert, Konstantin Riedl +1Transformer Inference Transformer
Department of Mathematics Friedrich-Alexander-University Erlangen-Nürnberg · Institute of Mathematics CAIDAS University of Würzburg · Mathematical Institute University of Oxford Reuben College +1
Remember to Forget: Gated Adaptive Positional Encoding
May 11, 2026Riccardo Ali, Alessio Borgi, Mario Severino +2Rotary Positional Embeddings Self-Attention
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Department of Computer, Control and Management Engineering, Sapienza University, Italy · Department of Information Engineering, University of Padova, Italy
Complex-Valued Phase-Coherent Transformer
May 11, 2026Leona HiokiSoftmax Attention Complex-Valued Neural Networks
Attention Drift: What Autoregressive Speculative Decoding Models Learn
May 11, 2026Doğaç Eldenk, Payal Mohapatra, Yigitcan Comlek +3Long-Context Language Model Inference Speculative Decoding
Northwestern University · GE Aerospace · fal +1
Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging
May 11, 2026Guisong Liu, Xin Gao, Martin Dresler +2Sleep Stage Classification Transformer
School of Biological Science and Medical Engineering Southeast University Nanjing, China · University of Bath Bath, England · Donders Institute for Brain, Cognition and Behaviour Radboud University Medical Center Nijmegen, The Netherlands +1

May 22, 2026J/K move · Enter open · S save

Improved Belief-Attention in Vision Task

Guoqiang Zhang

University of Exeter

Visual Attention Self-Attention Vision Transformer Transformer Attention

Open paper AskPDF arXiv

Recently, Belief-Attention \cite{Guoqiang25BeliefAttention} has been proposed by first performing an orthogonal projection of the softmax-based weighted summation of

V

vectors with respect to the original

V

vectors and then taking the perpendicular component as the residual signal in Transformer for performance improvement. In this paper, we first conduct an ablation study showing the projected component also carries information about the token correlation, which should not be ignored. We then propose to extend Belief-Attention by making use of both the perpendicular and projected components. In particular, the projected component goes through certain activation function and then a linear mapping before merging with the considered token. Conceptually speaking, the neural block for the projected component can be viewed as a two-layer feedforward network (FFN) within the new attention block. It is also noted that standard attention captures the token correlation via the inner-product matrix

QK^T

. We propose to introduce an additional inner-product matrix

ZZ^T

QK^T

to capture richer token correlation. We refer to the new module as Belief2-Attention. It can be easily shown that Belief2-Attention is more expressive than standard Attention. We then verify the effectiveness of Belief2-Attention for vision tasks of image classification and segmentation.