Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers

309 papers41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

Higher-Dimensional Rotary Position Embedding
Aug 30, 2026Yixing Li, Ruobing Xie, Yudong Zhang +3Rotary Positional Embeddings Transformer Attention
The Chinese University of Hong Kong · Tencent Hunyuan · Tsinghua University +1
Importance Scoring of Transformer Attention Heads in Learning Tabular Data
Aug 27, 2026Ahmad Jad Allah, Kazi F. Akhter, Md. Kamrozzaman Bhuiyan +1Attention Head Analysis Transformer Attention
Department of Computer Science Tennessee State University Nashville, TN, USA · Enosis Solutions Dhaka, Bangladesh · Department of Computer Science North Carolina Agricultural and Technical University Greensboro, NC, USA
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
Aug 26, 2026Gerard Conangla PlanesLow-Rank Adaptation Transformer Attention
Aily Labs
Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?
Aug 24, 2026Lanni Bu, Xiulin Yang, Christian Clark +2Transformer Attention Transformer
Georgetown University · The Ohio State University · UC San Diego
FlashAttention for Scalable Vector Architectures
Aug 19, 2026Sonia Rani Gupta, Nikela Papadopoulou, Miquel PericàsEfficient Transformer Inference Efficient Attention
Chalmers University of Technology Gothenburg, Sweden · University of Glasgow Glasgow, UK
WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing
Aug 19, 2026Wenbo Zhang, Xiang RenEfficient Transformer Inference KV Caching
University of Southern California
What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers
Aug 19, 2026Jesse PonnockVision Transformer Transformer Attention
Johns Hopkins University
TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
Aug 18, 2026Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4LLM Inference Acceleration Long-Context Language Model Inference
LLaVi Lab, Computer Science & Engineering, University of North Texas · Computer Science, Saint Louis University · Data Science, University of North Texas
Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention
Aug 18, 2026Emama Nahid, Tahmid Imtiaz Imu, Huayue Gu +3Transformer Attention Efficient Attention
Kennesaw State University Marietta, GA, USA · Miami University Oxford, OH, USA · Georgia State University Atlanta, GA, USA
Certified Mechanistic Interpretability: Lifting Single-Input Findings to Bounded Neighbourhoods
Aug 17, 2026Zhen Zhang, Yanliang Huang, Peng Xie +2Mechanistic Interpretability Neural Network Verification
School of Computation, Information and Technology Technical University of Munich, Germany
Hybrid Gated Attention
Aug 12, 2026Zekun Zhou, Ruobing Xie, Lanrui Wang +1Low-Rank Matrix Decomposition Gated Attention
Tencent Hunyuan · Peking University
PolarSym: Polar Geometry-aware Attention for CAD Floorplan Parsing
Aug 12, 2026Kerui Chen, Yiqing Wang, Kangzhou Xin +2Panoptic Segmentation Transformer Attention
School of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou, China · Third Dimension (Henan) Software Technology, Zhengzhou, China
Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression
Aug 11, 2026Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara +1LLM Compression Attention Head Pruning
University of Central Florida Orlando, Florida, USA · Old Dominion University Norfolk, Virginia, USA
Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So
Aug 10, 2026Mark OskinTransformer Interpretability Transformer Attention
School of Computer Science and Engineering University of Washington
ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Models
Aug 10, 2026Róisín LuoTransformer Attention Positional Encoding
University of Galway, Ireland
Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention
Aug 9, 2026Qucheng Gao, Zuyi Yang, Xiao ChenDynamical Systems Self-Attention
Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA · Department of Physics, Tsinghua University, Beijing 100084, China
Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention
Aug 8, 2026Kasun Dewage, Marianna Pensky, Suranadi De Silva +1Transformer Interpretability Transformer Attention
University of Central Florida Orlando, Florida, USA · Virtusa
Matrix Zonotopic Attention: A Context-Adaptive Value Projection for Set Transformers
Aug 5, 2026Zhen Zhang, Amr AlanwarSet Prediction Transformer Attention
School of Computation, Information and Technology Technical University of Munich, Germany
Provably Learning Multi-Head Attention with Queries
Aug 4, 2026Sunyeop Kim, Insung Kim, Jian GuoSoftmax Attention Transformer
Korea University, Republic of Korea · Nanyang Technological University, Singapore
One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse
Aug 3, 2026Shuxiao Xie, Shuyang Xie, Yuan Cao +3Softmax Attention Transformer Attention
Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China · Fudan University Institute of Systems for Advanced Computing, China · Harbin Institute of Technology, China +1
Feed-Forward Steering in Transformer Residual Dynamics
Aug 3, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer Interpretability Transformer FFNs
University of Luxembourg · 2London Institute of Mathematical Sciences
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Aug 2, 2026Kehan WangAttention Residual Connections Autoregressive Language Modeling
Kimi Team
Mask-Based Priors Are More Persistent than Query-Key Initializations
Aug 1, 2026Mingze Ma, Hemanth Saratchandran, Cameron Gordon +1Transformer Attention Inductive Bias
Australian Institute for Machine Learning · Adelaide University
Phase Structure in Rotary Attention: A Spectral Framework for Semantic Continuity and Execution-Boundary Governance
Jul 28, 2026Abraham ChachamovitsRotary Positional Embeddings Transformer Attention
ENTRUST AI
A Coulomb Particle Model for Learning Kernel Attention in Transformers
Jul 26, 2026Masoud Badiei Khuzani, Sharath Honnaiah, Atiq Islam +2Transformer Attention Efficient Attention
eBay Inc., San Jose, CA, USA
RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory
Jul 23, 2026Zahra Yousefijamarani, Alaa AlameldeenEfficient Transformer Inference AI Accelerator Inference
Simon Fraser University
TF-MossFormer: Integrating Convolution Gated Local-Global Attentions for Enhanced Time-Frequency Domain Monaural Speech Separation
Jul 23, 2026Shengkui Zhao, Zexu Pan, Haoxu Wang +3Hybrid CNN-Transformer Architectures Frequency-Domain Feature Learning
Token Foundry, Alibaba Group, Singapore
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
Jul 22, 2026Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun MoonLow-Rank Attention Transformer Attention
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea · Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea
Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making
Jul 21, 2026Yuyang Shen, Shan Dai, Daimin ChenNon-Stationary RL Sequential Decision Making
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data · University of International Business and Economics
A Controlled Study of Attention-Only Transformers
Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer FFNs Transformer
Cactus Compute, Inc.

Aug 30, 2026J/K move · Enter open · S save

Higher-Dimensional Rotary Position Embedding

Yixing Li, Ruobing Xie, Yudong Zhang, Yushi Bai+2

The Chinese University of Hong Kong · Tencent Hunyuan · Tsinghua University+1

Rotary Positional Embeddings Transformer Attention Positional Encoding

Open paper AskPDF arXiv

Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property. Furthermore, HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters. We have conducted extensive evaluation results demonstrating that HD-RoPE achieves significant performance improvements over standard RoPE across various popular benchmarks and in both long and short contexts.