Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers

331 papers35 papers in the last four weeks, up 6% on the four weeks before. 0.2% of all new papers.

Exploiting Biases in Attention for Transformers
Nov 3, 2026Charles Herr, Gabriele Spadaro, Enzo TartaglioneTransformer Attention Attention Mechanisms
LTCI, Télécom Paris, Institut Polytechnique de Paris, France · University of Turin, Italy · CNR-ISASI, Italy
Theoretical Analysis of Muon on Transformers for Next-Token Prediction
Nov 3, 2026Shengwei Wu, Yixuan Yang, Gao Huang +1Transformer Training Dynamics Muon Optimizer
School of Mathematical Sciences, Zhejiang University, Hangzhou, Zhejiang, China · Department of Mathematics, National University of Singapore, Singapore
FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
Nov 2, 2026Fedor Velikonivtsev, Mikhail Koltakov, Denis KuznedelevEfficient Attention Transformer Attention
Yandex Research
GatedV: Self-Gated Values for Transformer Language Models
Nov 1, 2026Li-Ming Yu, Hongyu ZhouTransformer Attention Gated Attention
Independent Researcher · Nanyang Technological University, Singapore
Meta-memorization and memorization scaling laws in transformers
Nov 1, 2026Alex Nguyen, Kenneth A. Norman, Gautam ReddyNeural Network Memorization Transformer Training Dynamics
Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA · Princeton Neuroscience Institute, Department of Psychology, Princeton University, Princeton, NJ, USA · Department of Physics, Princeton University, Princeton, NJ, USA
Conic Certification of Transformer Attention
Oct 31, 2026Masoud AtaeiTransformer Attention Certified Robustness
Department of Mathematical and Computational Sciences University of Toronto, Canada.
A Positional Realizability Theory of Length Generalization in Single-Head Attention
Oct 30, 2026Jiasong Han, Yunkai Chai, Jintao WangLength Generalization Positional Encoding
Department of Electronic Engineering, Tsinghua University · School of Information and Communication Engineering, Shanghai Jiao Tong University
Emergence of the transformer architecture as the score of entropic laws of random measures
Oct 29, 2026Giacomo GrecoTransformer Attention Autoregressive Language Modeling
Where Position Matters: Profiling-Guided Layer-Wise RoPE Allocation in Transformers
Oct 29, 2026Zhiwei Liu, Boao Kong, Bing LiuRotary Positional Embeddings Transformer Attention
SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers
Oct 26, 2026Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan +5Transformer Attention Attention Head Analysis
Georgia Institute of Technology · Dolby Laboratories
Transformers as Generic Deep Neural Networks: Exact Compilation, the Cost of Removing Activation-Activation Products, and Multi-Agent Attention Clustering
Oct 20, 2026Sergey Salishev, Zeev Volkovich, Oleg GranichinEfficient Transformer Inference Transformer Attention
St. Petersburg State University, Saint Petersburg 199034, Russia · Department of Software Engineering, Braude College of Engineering, Karmiel 2161002, Israel
RACE: RoPE-Aligned Attention Compression Based on Query-Key Activation Energies
Oct 20, 2026Irene Tenison, Abdulrahman Alabdulkareem, Daniel Karl I. Weidele +2KV-Cache Compression Transformer Attention
MIT CSAIL · MIT-IBM Computing Research Lab
FTFM: Factorized Tabular Foundation Models
Oct 18, 2026Josif GrabockaTabular Foundation Models Tabular ICL
University of Technology Nuremberg
Transformers Learn Two Competing Sorting Algorithms, and Only One Length-Generalizes
Oct 18, 2026Khashayar Gatmiry, Nathan W. Henry, Chloe H. Su +5Length Generalization Transformer Training Dynamics
UC Berkeley & Apple · UC Berkeley · Kempner Institute, Harvard University +5
Transformers as Adaptive Kernel Learners across Tasks for In-Context Learning
Oct 17, 2026Zhaiming Shen, Alexander Cloninger, Rongjie Lai +1Transformer Attention In-Context Learning
Department of Mathematics and Statistics, Utah State University, Logan, UT 84322 · Department of Mathematics, Purdue University, West Lafayette, IN 47907 · School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332
A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt
Oct 16, 2026Shuyang XiangTransformer Attention Transformer Training Dynamics
Independent researcher
Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches
Oct 15, 2026Adam Deng, Shicheng Liu, Yada ZhuKV-Cache Reuse Multi-Agent LLM Systems
MIT ORC · MIT CSAIL · IBM Research
Continuum Attention
Oct 14, 2026Afshin KhadangiCausal Attention Transformer Attention
SnT, University of Luxembourg
Training Dynamics of Transformers on Unbounded Inputs: Attention Learning Can Lag Output Layer Learning
Oct 14, 2026Yaomengxi Han, Debarghya GhoshdastidarTransformer Training Dynamics Neural Network Training Dynamics
Technical University of Munich
Learning Source Structure: Bayesian Prediction with Transformers
Oct 13, 2026Meir Feder, Ruediger UrbankeTransformer Sequence Modeling
School of Electrical and Computer Engineering, Tel Aviv University, Tel Aviv, Israel · School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
The Effect of Layer Ordering in Hybrid Architectures
Oct 13, 2026John Cooper, Mingchen Ma, Jiaqi Mo +4Hybrid Transformer Architectures Hybrid Attention
Department of Computer Sciences, University of Wisconsin-Madison · Department of Computer Sciences, John Hopkins University
Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer Reparameterization
Oct 13, 2026Bum Jun KimDeep Learning Optimization Transformer
The University of Tokyo
On the Statistical Mechanics of Attention
Oct 12, 2026Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su +1Transformer Attention Attention Mechanisms
Center for Foundation Models and Generative AI, Northwestern University Department of Computer Science, Northwestern University · Department of Computer Science, College of Connected Computing, Vanderbilt University · Department of Physics, National Taiwan University +1
CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoff
Oct 11, 2026Haomiao Qiu, Yining Tang, Zijia Song +3KV-Cache Transfer LLM Serving
Harbin Institute of Technology, Shenzhen · Great Bay University · Zhejiang University
HodgePhaseNet: Discrete Hodge Decomposition of Semantic Flows in Transformer Attention
Oct 10, 2026Kiyotaka Kasubuchi, Kazuo FukiyaTransformer Attention Hodge Decomposition
Learnable Low-Rank Polynomial Sketch for Effective Linear Attention
Oct 9, 2026Zongyue Qin, Haoran Deng, Jason Cong +1Transformer Attention Linear Attention
Department of Computer Science, University of California Los Angeles, CA, USA
REFLECT: Role-Aware Detection of Sentiment, Emotion, Politeness, and Formality
Oct 9, 2026Sangpil Youm, Annabel Stocks Natalias, Hayden Heninger +4Emotion Recognition in Conversations Sentiment Analysis
University of Florida
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Oct 9, 2026Yuhong Chou, Zehao Liu, Yuqi Pan +4Transformer Attention Attention Mechanisms
The Hong Kong Polytechnic University · University of Chinese Academy of Sciences · Independent Researcher +1
Neural expectation-maximization and meta-models
Oct 8, 2026Zhuolin FuTransformer Transformer Attention
Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization
Oct 8, 2026Bahareh Nikpour, Hossein Rajabzadeh, Maryam Dialameh +2Transformer Attention KV-Cache Compression
Huawei Technologies, Toronto, Canada · University of Waterloo, Waterloo, Canada · Huawei Noah’s Ark Lab

Nov 3, 2026J/K move · Enter open · S save

Exploiting Biases in Attention for Transformers

Charles Herr, Gabriele Spadaro, Enzo Tartaglione

LTCI, Télécom Paris, Institut Polytechnique de Paris, France · University of Turin, Italy · CNR-ISASI, Italy

Transformer Attention Attention Mechanisms Efficient ViTs Transformer Pruning Token Pruning

Open paper AskPDF arXiv

Attention is known to be sparse in vision transformers. In particular, we analyze the unequal relevance of attention-matrix columns to the final computation and show that this phenomenon becomes increasingly pronounced with network depth. This can be interpreted as saying that tokens have an absolute importance for the attention process, independently of queries. Inspired by this, we propose a reformulation of the attention formula that makes this phenomenon explicit by adding an input-dependent bias to the query-key dot product. We show that this new term gives each token a scalar value indicating its anticipated influence on other tokens during the attention process, and can thus serve as a proxy for pruning attention maps. Across multiple transformer model sizes, this allows us to substantially reduce the computational cost of attention, in terms of both FLOPs and inference latency, while retaining most of the original classification accuracy. We publicly released the code and models at https://github.com/EIDOSLAB/AttentionWithBias.