Attention Mechanisms

Follow this topic Ask about this topic

Attention mechanisms let a model select and combine information from other tokens, image regions, or data points, supporting tasks such as language understanding, visual recognition, and retrieval. The standard approach uses query-key-value comparisons with softmax, while sparse, linear or kernel-based, local, and hybrid methods aim to reduce the cost of processing long sequences; positional encodings and cross-attention adapt interactions to order, geometry, or another input. Researchers also study attention heads and circuits to understand model behavior, alongside mathematical analyses of expressivity, training dynamics, and representation collapse. A central challenge is balancing accuracy and generalization against memory and computation, while determining when attention weights provide reliable explanations rather than merely useful internal signals.

Latest papers

517 papers61 papers in the last four weeks, up 30% on the four weeks before. 0.4% of all new papers.

Ordinary Least Squares as an Attention Mechanism
Apr 13, 2025Philippe Goulet CoulombeTransformer Attention Linear Regression
Département des Sciences Économiques, Université du Québec à Montréal
You Do Not Fully Utilize Transformer's Representation Capacity
Feb 13, 2025Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky +2Representation Collapse Memory-Augmented Language Models
1T-Tech · 2Moscow Institute of Physics and Technology · 3HSE University
Attention is All You Need Until You Need Retention
Jan 15, 2025M. Murat YasliogluMemory-Augmented Language Models Agent Memory Poisoning
Istanbul University, School of Business
A Mechanistic Study of Transformers Training Dynamics
Oct 31, 2024Ambroise Odonnat, Wassim Bouaziz, Vivien CabannesTransformer Interpretability Attention Mechanisms
Inria, Univ. Rennes 2 · Mistral AI; work done while at Meta. · FAIR at Meta.
WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development
Oct 24, 2024Shanchao Liang, Nan Jiang, Shangshu Qian +1Image-to-Code Generation Web Application Generation
Purdue University
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Oct 3, 2024Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva +5Self-Attention Multiple-Choice Question Answering
Skolkovo Institute of Science and Technology · AI Foundation and Algorithm Lab · Moscow Institute of Physics and Technology +2
GFPack++: Attention-Driven Gradient Fields for Optimizing 2D Irregular Packing
Jun 9, 2024Tianyang Xue, Lin Lu, Yang Liu +5Combinatorial Optimization Constrained Optimization
Shandong University, China · Microsoft Research Asia, China · Peking University, China

Apr 13, 2025J/K move · Enter open · S save

Ordinary Least Squares as an Attention Mechanism

Philippe Goulet Coulombe

Département des Sciences Économiques, Université du Québec à Montréal

Transformer Attention Linear Regression Attention Mechanisms Tabular Regression

Open paper AskPDF arXiv

I show that ordinary least squares (OLS) predictions can be rewritten as the output of a restricted attention module, akin to those forming the backbone of large language models. The connection comes from viewing OLS as a similarity-based prediction rule in a learned embedding space. In this representation, least squares does not estimate coefficients per se. Instead, it selects an embedding that minimizes squared prediction error by matching training and test vectors through inner products. This maps directly onto the query-key-value structure of attention mechanisms. I then discuss extensions to dimensionality reduction, nonlinearity, and time series econometrics. Monte Carlo simulations and real-data experiments on UCI/OpenML benchmarks show that nonlinear Attention Regression performs competitively against standard machine learning baselines. In the reverse direction, I replace the attention sublayer of a transformer for tabular data with an explicit regression on polynomial features. The resulting model performs comparably to the standard transformer at a fraction of its parameter count.