Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics

Stream-CQSA: Exact Out-of-Memory Recovery for Attention
Apr 22, 2026Yiming Bian, Joshua M. AkeySelf-Attention Memory-Efficient Inference
Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540
Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity
Apr 22, 2026Pranava Madhyastha, Dagmar AdamcovaTransformer Attention Language Modeling
△City, University of London · ∞The Alan Turing Institute · ⊕Grounded Machines
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
Apr 21, 2026Yuqi Zhang, Di ZhangRetrieval-Augmented Generation Self-Attention
Xi’an Jiaotong-Liverpool University
Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling
Apr 21, 2026Weijie Zhao, Mingquan Liu, Bolun Wang +4Transformer Language Model Scaling Laws
Luxi Tech, Shenzhen, China · The School of Computer Science, China University of Geosciences, 388 Lumo Road, Wuhan, 430074, Hubei, China.
Advancing Vision Transformer with Enhanced Spatial Priors
Apr 20, 2026Qihang Fan, Huaibo Huang, Mingrui Chen +2Vision Transformer Transformer Attention
MAIS & NLPR, Institude of Automation, Chinese Academic of Science, Beijing 100190, China · School of Artificial Intelligence, University of Science and Technology Beijing, Beijing 100083, China
DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs
Apr 19, 2026Lexuan Liang, Tao Zou, Xuxiang Ta +1Graph Attention Networks Transformer Attention
School of Computer Science and Engineering Beihang University Beijing, China 100191
Capacity-Controlled Global Attention for Graph Transformers
Apr 19, 2026Yang Liu, Dongxin Guo, Tom Zheng +3Gated Attention Transformer Attention
Brain Investing Limited Hong Kong, China · The University of Hong Kong Hong Kong, China · Stellaris AI Limited Hong Kong, China
Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension
Apr 17, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuSpiking Transformers Transformer Attention
*The University of Hong Kong, Hong Kong, China · †Brain Investing Limited, Hong Kong, China
Dispatch-Aware Ragged Attention for Pruned Vision Transformers
Apr 16, 2026Seifeldin Abdellatif, Ahmad AlmasriEfficient Transformer Inference Visual Attention
College of Engineering Al Ain University Abu Dhabi, United Arab Emirates
Gating Enables Curvature: A Geometric Expressivity Gap in Attention
Apr 16, 2026Satwik Bathula, Anand A. JoshiInformation Geometry Neural Representation Geometry
Department of Electrical and Computer Engineering University of Southern California
Ordinary Least Squares is a Special Case of Transformer
Apr 15, 2026Xiaojun Tan, Yuchen ZhaoTransformer Transformer Attention
Center for Peak Performance, Department of Psychology and Behavioral Sciences, Zhejiang University · Hangzhou Higgs Asset Management Co., Ltd. · College of Control Science and Engineering, Zhejiang University
Subcritical Signal Propagation at Initialization in Normalization-Free Transformers
Apr 13, 2026Sergey AlekseevTransformer Neural Network Initialization
Department of Physics and Astronomy Stony Brook University Stony Brook, NY 11794
Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification
Apr 13, 2026Patrick Lutz, Themistoklis Haris, Arjun Chandra +2In-Context Learning Transformer Interpretability
Boston University, Departments of Computer Science and Electrical & Computer Engineering.
Winner-Take-All Spiking Transformer for Language Modeling
Apr 13, 2026Chenlin Zhou, Sihang Guo, Jiaqi Wang +6Spiking Transformers Transformer Attention
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University · Pengcheng Laboratory · Harbin Institute of Technology +1
Transformers Learn Latent Mixture Models In-Context via Mirror Descent
Apr 12, 2026Francesco D'Angelo, Nicolas FlammarionIn-Context Learning Transformer Attention
TML Lab, EPFL, Switzerland
Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V
Apr 12, 2026Chirag ShindeTransformer Attention Attention Residual Connections
Independent Researcher
Tracing the Thought of a Grandmaster-level Chess-Playing Transformer
Apr 11, 2026Rui Lin, Zhenyu Jin, Guancheng Zhou +7Transformer Interpretability Mechanistic Interpretability
Shanghai Innovation Institute, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China · School of Mathematics and Statistics, Xi’an Jiaotong University.
Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Apr 11, 2026Zunhai Su, Hengyuan Zhang, Wei Wu +24Attention Sinks Transformer Attention
Tsinghua University · Meituan LongCat Team · The University of Hong Kong +6
Hierarchical Kernel Transformer: Multi-Scale Attention with an Information-Theoretic Approximation Analysis
Apr 10, 2026Giansalvo CirrincioneTransformer Attention Attention Mechanisms
Laboratoire LTI, Université de Picardie Jules Verne, 33 rue Saint-Leu, Amiens, 80039, France
SAT: Selective Aggregation Transformer for Image Super-Resolution
Apr 9, 2026Dinh Phu Tran, Thao Do, Saad Wazir +3Image Super-Resolution Efficient ViTs
School of Computing, KAIST, Republic of Korea
Sinkhorn doubly stochastic attention rank decay analysis
Apr 9, 2026Michela Lapenna, Rita Fioresi, Bahman GharesifardSoftmax Attention Self-Attention
Department of Physics and Astronomy, University of Bologna, Bologna, Italy · Department of Pharmacy and Biotechnologies, University of Bologna Bologna, Italy · Department of Mathematics and Statistics, Queen’s University, Kingston, ON, Canada
Trilinear Compute-in-Memory Architecture for Energy-Efficient Transformer Acceleration
Apr 8, 2026Md Zesun Ahmed Mia, Jiahui Duan, Kai Ni +1Compute-in-Memory Efficient Transformer Inference
The Pennsylvania State University, Department of Electrical Engineering, University Park, PA, USA · University of Notre Dame, Department of Electrical Engineering, Notre Dame, IN, USA
k-Maximum Inner Product Attention for Graph Transformers and the Expressive Power of GraphGPS
Apr 4, 2026Jonas De Schouwer, Haitz Sáez de Ocáriz Borde, Xiaowen DongTransformer Attention Graph Transformer
Stanford University · University of Oxford
Cosine-Normalized Attention for Hyperspectral Image Classification
Apr 2, 2026Muhammad Ahmad, Manuel MazzaraHyperspectral Image Classification Transformer Attention
SDAIA-KFUPM, Joint Research Center for Artificial Intelligence (JRCAI), King Fahd University of Petroleum and Minerals, Dhahran, 31261, Saudi Arabia · Institute of Software Development and Engineering, Innopolis University, Innopolis, 420500, Russia
Invertible Query-Key Coupling Composes with Attention Mechanisms
Apr 2, 2026Barak Gahtan, Alex M. BronsteinInvertible Neural Networks Self-Attention
Technion – Israel Institute of Technology · ISTA – Institute of Science and Technology Austria
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
QUEST: A robust attention formulation using query-modulated spherical attention
Mar 31, 2026Hariprasath Govindarajan, Per Sidén, Jacob Roll +1Transformer Attention Attention Mechanisms
Linköping University, Sweden · Qualcomm Auto Ltd Sweden Filial
Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction
Mar 31, 2026L. Ghiringhelli, A. Zambon, G. TianaProtein Structure Prediction AI-Assisted Scientific Research
Department of Physics, via Celoria 16, 20133 Milano, Italy · Department of Physics and INFN, via Celoria 16, 20133 Milano, Italy
VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference
Mar 31, 2026Anmin Liu, Ruixuan Yang, Huiqiang Jiang +5Transformer Attention Sparse Attention
SCS, Peking University, Beijing, China · Key Lab of HCST (PKU), MOE, Beijing, China · Fudan University, Shanghai, China +2
Physics-Guided Transformer (PGT): Physics-Aware Attention Mechanism for PINNs
Mar 30, 2026Ehsan Zeraatkar, Rodion Podorozhny, Jelena TešićPhysics-Informed ML Neural PDE Solvers
Computer Science, Texas State University, San Marcos, Texas, US

Apr 22, 2026J/K move · Enter open · S save

Stream-CQSA: Exact Out-of-Memory Recovery for Attention

Yiming Bian, Joshua M. Akey

Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540

Self-Attention Memory-Efficient Inference Transformer Attention

Open paper AskPDF arXiv

Long-context large language models are limited not only by attention cost but also by out-of-memory (OOM) failures. A selected attention call may not fit in available device memory even when the kernel is optimized. Exact and approximate attention methods reduce memory use, but every fixed implementation still has a device-specific capacity boundary. We introduce Stream-CQSA, an attention-level OOM recovery framework based on CQS decomposition, derived from the theory of cyclic quorum sets (CQS). Stream-CQSA recursively partitions an infeasible attention call into independent subsequence tasks, executes each with a compatible inner kernel, and recomposes the local statistics to recover the full attention output. This recovery is exact relative to the wrapped attention kernel, whether that kernel is exact or approximate. Compared with FlashAttention-2, the major baseline, our native Stream-CQSA kernel improves 16-bit forward-output error relative to a dense float64 reference and matches 16-bit backward-gradient error where FlashAttention-2 fits in the GPU memory. At the longest feasible baseline length, it costs

1.5

1.9\times

the forward runtime and

2.1

2.4\times

the forward--backward runtime. Beyond that sequence length boundary, our method continues to return an output while FlashAttention-2 OOMs. Stream-CQSA is therefore not a faster attention method. Instead, it converts memory-capacity failure into a recoverable execution path by trading extra compute, host-device transfer, and recomposition for completion.