Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics

An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
Jul 20, 2026Cheng Huan, Hongwei YuanTransformer Attention Positional Encoding
Department of Statistics and Data Science, The Chinese University of Hong Kong. · Department of Mathematics, University of Macau.
The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture
Jul 19, 2026Zhihua LiangTransformer Transformer Attention
INFN, Sezione di Cagliari, I-09042 Monserrato (CA), Italy
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
Jul 15, 2026Tuomas Oikarinen, Zixiao Chen, Charlotte Siska +3LLM Fine-Tuning Text Classification
UC San Diego, Microsoft Research · Microsoft Security AI · UC San Diego +1
AVQ-Attention: Adaptive Vector-Quantized Attention
Jul 14, 2026Winfried van den dool, Patrick Forré, Amir Habibian +2Transformer Attention Vector Quantization
QUVA Lab, University of Amsterdam, The Netherlands · AMLab, Informatics Institute, University of Amsterdam, The Netherlands · AI4Science Lab, University of Amsterdam, The Netherlands +3
From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
Jul 12, 2026Binbin Lin, Wei Chen, Yalun Li +3Self-Attention Transformer Attention
School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China
Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Jul 9, 2026Aiwei Liu, Cheng Shi, Chuhan Wu +45Language Model Scaling Laws Transformer Attention
FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention
Jul 8, 2026Athanasios ZerisFrequency-Domain Feature Learning Transformer Attention
Independent Researcher, Athens, Greece.
Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits
Jul 8, 2026Zitong Andrew Chen, Junaid Hasan, Akhil Srinivasan +2Transformer Interpretability Transformer Attention
1UW Math AI Lab, University of Washington.
Legible-by-Construction: Attention and End-to-End Transformers
Jul 5, 2026Mark OskinTransformer Interpretability Transformer Attention
School of Computer Science and Engineering University of Washington
LLT: Local Linear Transformer for PDE Operator Learning
Jul 4, 2026Oded Ovadia, Eli TurkelPDE Surrogate Modeling Transformer Attention
School of Mathematical Sciences, Tel Aviv University, Tel Aviv, Israel
X-LogSMask: Expand Transformer for Graph-Structured Data
Jul 2, 2026Leyan Li, Rennong Yang, Zhenxing Zhang +1Graph Attention Networks Graph Representation Learning
The Wiola Architecture for Efficient Small Language Models
Jul 1, 2026Aryuemaan Kumar Chowdhury, Afreen Shaik, Yaparla Bhargavi +1Rotary Positional Embeddings Token Merging
Research and Development, Oscowl Ai IIT Hyderabad Hyderabad, India · Research and Development, Oscowl Ai Hyderabad, India
Holographic Quantum Transformer: A Generalist Neuro-Symbolic Architecture for Solving Frustrated Systems via Generative Attention
Jul 1, 2026Xingran Guo, Tiaojie Xiao, Jie Liu +1Neural Quantum States Zero-Shot Learning
National University of Defense Technology Changsha, China · State University of New York at New Paltz New Paltz, NY, USA
Generalization Analysis of Transformers in Distribution Regression
Jun 28, 2026Peilin Liu, Ding-Xuan ZhouNeural Network Generalization Transformer Attention
School of Mathematics and Statistics, University of Sydney, Sydney, NSW 2006, Australia
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
Jun 25, 2026Shubham AggarwalHierarchical Representation Learning Transformer Attention
WQ-Fusion: Dynamic Gated Attention for Cross-Domain Audio Representation
Jun 25, 2026Mingda Lin, Lei Ding, Xinyue Zhou +6Audio Representation Learning Gated Attention
School of Electronic Information, Wuhan University, Wuhan, Hubei, China · Tencent AI Lab Seattle, Seattle, USA · CIAIC, Northwestern Polytechnical University, Xi’an, Shaanxi, China +1
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Jun 23, 2026Vatsal Baherwani, Zixi Chen, Shikai Qiu +2Language Model Scaling Laws Transformer Attention
New York University
Parallel Manifold Steering: Efficient Adaptation of Large Associative Memories via Residual Energy Shaping
Jun 23, 2026Kanishk AwadhiyaMemory-Augmented Neural Networks Transformer Attention
Independent Researcher
Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection
Jun 23, 2026Tian Qiu, Jifeng Shen, Xin ZuoMultispectral Object Detection Cross-Modal Alignment
School of Electrical and Information Engineering, Jiangsu University, Zhenjiang, 212013, China · School of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang, 212003, China
Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Jun 18, 2026Vishesh Tripathi, Abhay KumarGrouped-Query Attention Transformer Attention
FrontiersMind
ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence
Jun 17, 2026Ashim Dhor, Rasel Mondal, Pin-Yu ChenTransformer Attention Convolutional Neural Networks
Indian Institute of Science Education and Research Bhopal · IBM Research
Complementary Attention Head Pruning for Efficient Transformers
Jun 17, 2026Yaniv Livertovsky, Shahar Somin, Gonen SingerModel Compression Transformer Attention
Bar-Ilan University Ramat Gan, Israel
Dual Dimensionality for Local and Global Attention
Jun 17, 2026Zhiyuan Wang, Xuan Luo, Sirui Zeng +1LLM Compression Decoder-Only Language Models
UC Santa Barbara
Communication-Efficient Verifiable Attention for LLM Inference
Jun 15, 2026Ziqun Chen, Ming Wu, Michael Heinrich +4LLM Inference Trusted Execution Environments
Nanyang Technological University · Singapore · Zero Gravity Labs +1
QK-Normed MLA: QK normalization without full key caching
Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2KV Caching Transformer Attention
School of Data Science, The Chinese University of Hong Kong, Shenzhen.
Formalizing and Mitigating Structural Distortion in LLM Attention for Graph Reasoning
Jun 14, 2026Donald Loveland, Puja Trivedi, Ari Weinstein +2LLM Reasoning with Graphs Transformer Attention
University of Michigan Ann Arbor, MI, USA · Amazon Palo Alto, CA, USA
Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models
Jun 12, 2026Ravi Ranjan, Utkarsh Grover, Xiaomin Lin +1Transformer Interpretability Explainable Artificial Intelligence
Florida International University · Miami, USA · University of South Florida +1
FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding
Jun 12, 2026Ziwei Wang, Xingyi He, Tianwang Jia +2Hybrid CNN-Transformer Architectures Neural Decoding
Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China
Attention by Synchronization in Coupled Oscillator Networks
Jun 10, 2026Fabio Pasqualetti, Taosha GuoSoftmax Attention Dynamical Systems
Electrical Engineering and Computer Science University of California, Irvine
Boltzmann Attention: Learnable Ising Couplings for Cooperative Attention
Jun 10, 2026Gilhan Kim, Daniel K. ParkQuantum Annealing Ising Model
Department of Statistics and Data Science, Yonsei University, Seoul 03722, Republic of Korea · Department of Applied Statistics, Yonsei University, Seoul 03722, Republic of Korea · Department of Quantum Information, Yonsei University, Seoul 03722, Republic of Korea

Jul 20, 2026J/K move · Enter open · S save

An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers

Cheng Huan, Hongwei Yuan

Department of Statistics and Data Science, The Chinese University of Hong Kong. · Department of Mathematics, University of Macau.

Transformer Attention Positional Encoding Sensitivity Analysis

Open paper AskPDF arXiv

We develop an adjoint-sensitivity framework for positional influence in causal residual Transformers and separate unconditional analytic results from conditional boundary-shape conclusions. The principal unconditional theorem is the residual-to-depth-flow estimate for layer controls converging in

L^1

, complemented by a finite-token-to-Volterra attention estimate that explicitly controls the first cells near the causal endpoint. We define a normalized adjoint-energy influence density and derive its exact evolution along full-batch gradient flow. The adjoint admits an exact generator-term decomposition into residual transmission, nonlocal Volterra, and local channels, including all covariance cross terms. Causal masking can amplify early-position sensitivity and residual identity paths can transmit a right-localized terminal bias, but neither mechanism alone forces a U-shaped profile. We therefore state boundary advantages under independently checkable energy, correlation, and local-channel bounds; these conditions are sufficient rather than necessary. Finite-token influence balancing, positional reweighting, and task-aligned observability are presented as diagnostics or regularizers with explicit differentiation requirements, computational costs, and limitations. Controlled simulations illustrate that each intervention controls its designated surrogate, while observability balance or outer-loop reweighting need not monotonically reduce the influence-based Lost-in-the-Middle diagnostic.