Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics

Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models
Jun 8, 2026Yuxuan Chen, Haoyuan Yu, Peize HeCross-Attention Audio Diffusion Models
Jilin University · Hunan University · University of Electronic Science and Technology of China
Integrating gene regulatory priors into Transformer attention with scTransformer for interpretable scRNA-seq analysis
Jun 8, 2026Mikele Milia, Louis Fabrice Tshimanga, Henning Mueller +2Transformer Interpretability Transformer Attention
Department of Information Engineering, University of Padova, Padova, Italy · Department of Neuroscience, University of Padova, Padova, Italy · Padova Neuroscience Center, Padova, Italy +3
Hasse Diagrams for Attention: A Partial Order Framework for Designing Transformer Masks
Jun 8, 2026Chentao Li, Han GuoTransformer Attention Attention Mechanisms
School of Mathematics and Statistics Northeast Petroleum University Daqing, Heilongjiang 163318, China
Chiaroscuro Attention: Spending Compute in the Dark
Jun 6, 2026Prateek Kumar SikdarTransformer Transformer Attention
Accenture, India
AttentionCap: Transformer Based Capacitance Matrix Learning Toward Full-Chip Extraction
Jun 6, 2026Jiechen Huang, Hector R. Rodriguez, Dingcheng Yang +3Electronic Design Automation Transformer Attention
Dept. Computer Science & Tech., BNRist, Tsinghua Univ., Beijing, China · School of IC, BNRist, Tsinghua Univ., Beijing, China · School of IC, Peking Univ., Beijing, China
Attention at the Theoretical Minimum: A Mathematics of Arrays Framework for Memory-Optimal Transformer Kernels
Jun 5, 2026Lenore Mullin, Gaetan HainsGPU Kernel Optimization Energy-Efficient ML
University at Albany, SUNY, Albany, NY, USA · LACL, Université Paris-Est Créteil, Créteil, France
The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspaces
Jun 5, 2026Soo Min Kwon, Alec S. Xu, Can Yaras +3OOD Generalization Transformer Attention
Department of Electrical and Computer Engineering, University of Michigan · Department of Statistics, University of California, Davis
Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram
Jun 4, 2026Athanasios ZerisTransformer Interpretability Proper Orthogonal Decomposition
Independent Researcher, Athens, Greece.
ATT-CR: Adaptive Triangular Transformer for Cloud Removal
Jun 4, 2026Yang Wu, Ye Deng, Pengna Li +4Remote Sensing Transformer Attention
Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China · School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, Sichuan 611130, China · Ningbo University of Technology, Ningbo, Zhejiang 315211, China
Depth-Attention: Cross-Layer Value Mixing for Language Models
Jun 3, 2026Boyi Zeng, Yiqin Hao, Zitong Wang +7Efficient Transformer Inference Decoder-Only Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute +2
Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting
Jun 3, 2026Balthazar Courvoisier, Tristan CazenaveTime Series Forecasting Transformer Attention
Queensfield AI Technologies, Paris, France · LAMSADE, Universit´e Paris Dauphine - PSL, Paris, France
AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task
Jun 2, 2026Quentin Fuxa, Dominik MacháčekSpeech Translation Decoder-Only Language Models
Independent Researcher · Charles University, MFF, ÚFAL · & University of Edinburgh
Do Transformers Need Three Projections? Systematic Study of QKV Variants
Jun 1, 2026Ali Kayyam, Anusha Madan Gopal, M Anthony LewisEfficient Transformer Inference Transformer Attention
BrainChip Inc., Laguna Hills, CA, USA.
Do Value Vectors in Deep Layers Need Context from the Residual Stream?
Jun 1, 2026Muyu He, Yuchen Liu, Qingya Huang +1Transformer Attention Language Modeling
1Independent · 2Drexel University
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
Jun 1, 2026Yu Li, Binxu Li, Tian LanKV Caching Long-Context Language Model Inference
George Washington University · Princeton University
FAiT: Frequency-Aware Inverted Transformer for Multivariate Time Series Forecasting
May 31, 2026Peng He, Yao Liu, Yanglei Gan +3Multivariate Time Series Forecasting Frequency-Domain Feature Learning
University of Electronic Science and Technology of China · Chengdu, Sichuan, China
VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference
May 31, 2026Xinze ZhangTransformer Attention Deformable Attention
University of Southern California, Los Angeles, CA 90007, USA
Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics
May 28, 2026Matthew Smart, Soumya Ganguly, Nilava Metya +2Self-Attention Empirical Bayes
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ, USA · Department of Mathematics, Rutgers University, Piscataway, NJ, USA · Department of Physics and Astronomy, Rutgers University, Piscataway, NJ, USA +1
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
May 27, 2026Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1Transformer Attention Long-Context Modeling
Carnegie Mellon University · Google Research
The Attentional White Bear Effect in Transformer Language Models
May 27, 2026Rebecca Ramnauth, Brian ScassellatiTransformer Interpretability LLM Alignment
Yale University New Haven, CT, USA
Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference
May 27, 2026Alan FerrariEfficient Transformer Inference Self-Attention
Knowledge Lab AG Zürich, Switzerland
ChWDTA: Channel-wise Wavelet-Domain Transformer Attention and Entropy Modeling for Learned Image Compression
May 27, 2026Haisheng Fu, Runyu Yang, Feng Ding +5Learned Image Compression Hybrid CNN-Transformer Architectures
Electrical and Computer Engineering Department, The University of British Columbia, Vancouver, BC, Canada · School of Engineering Science, Simon Fraser University, Burnaby, BC, Canada · School of Electronic Science and Technology, Eastern Institute of Technology, Ningbo, China +1
Fine-Tuning Dynamics of In-Context Factual Recall in Transformers
May 26, 2026Ruomin Huang, Eshaan Nichani, Jason D. Lee +1Supervised Fine-Tuning Transformer Attention
Duke University · Princeton University · UC Berkeley
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
May 26, 2026Sridhar MahadevanTransformer Autoregressive Language Modeling
Adobe Research San Jose, CA · Academic affiliation: Research Professor, University of Massachusetts, Amherst
Is an Image Also Worth 16x16=256 Superpixels? A Framework for Attentional Image Classification
May 26, 2026Pedro Henrique da Costa Avelar, Anderson R. Tavares, Luís C. LambGraph Attention Networks Vision Transformer
ufrgs · uom
Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective
May 26, 2026Hyunmin Cho, Woo Kyoung Han, Kyong Hwan JinSelf-Attention Transformer Attention
Department of Electrical Engineering, Korea University, Seoul, South Korea.
Transformers Can Learn Posterior Predictive Distributions In-Context
May 26, 2026Gyeonghun Kang, Changwoo J. Lee, Xiang ChengBayesian Inference Transformer Attention
Department of Statistical Science, Duke University, Durham, NC, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
JetViT: Efficient High-Resolution Vision Transformer with Post-Training Attention Search
May 26, 2026Dongyun Zou, Zhuoyang Zhang, Junyu Chen +8Efficient Transformer Inference Vision Foundation Models
MIT · NVIDIA · University of Pennsylvania +1
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
May 22, 2026Jie Hu, Zixiang Gao, Yutong He +1Video Diffusion Models Diffusion Transformer
1Peking University.

Jun 8, 2026J/K move · Enter open · S save

Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models

Yuxuan Chen, Haoyuan Yu, Peize He

Jilin University · Hunan University · University of Electronic Science and Technology of China

Cross-Attention Audio Diffusion Models Transformer Attention Efficient Attention Audio Source Separation

Open paper AskPDF arXiv

Flow-matching transformers achieve strong audio separation, yet their attention dynamics are opaque. We adapt established causal-intervention principles into a deterministic, inference-time probing protocol for SAM Audio. Orthogonal probing uncovers a dual-pathway text-conditioning mechanism: additive injections control semantic identity, while cross-attention refines acoustic structure. We observe an asynchronous layerwise convergence: stable layers build temporal scaffolds early, whereas fast layers continue resolving artifacts during sampling. The model also attenuates temporal segmentation cues to maintain continuous-flow stability. Using these insights, we propose Layer-Selective Attention Caching (LSAC), a training-free acceleration method that caches attention in stable layers. Across acoustic complexities, LSAC cuts self-attention computation by about ~25% with negligible quality loss and yields up to 6.7x higher quality retention than naive step reduction.