Transformer Attention

Ask about this topic

Overview

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics
CardsList

  1. A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification

    Mar 27, 2026Zhixuan Cao, Yishu Xu, Xuang WUTransformerTransformer Attention

    Tsinghua University · University of California, Berkeley

  2. Decoding Defensive Coverage Responsibilities in American Football Using Factorized Attention Based Transformer Models

    Mar 26, 2026Kevin Song, Evan Diewald, Ornob Siddiquee +4Transformer Attention

    Amazon Web Services, Seattle, WA, USA · National Football League, New York, NY, USA

  3. On the Expressive Power of Transformers for Contextual Relations

    Mar 26, 2026Demián FraimanTransformer ExpressivityTransformer Attention

    Instituto de Cálculo Universidad de Buenos Aires

  4. Hyperspectral Trajectory Image for Multi-Month Trajectory Anomaly Detection

    Mar 26, 2026Md Awsafur Rahman, Chandrakanth Gudavalli, Hardik Prajapati +1Anomaly LocalizationTransformer Attention

    UC Santa Barbara, USA

  5. Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping

    Mar 25, 2026Yao Chen, Yilong Chen, Yinqi Yang +9Adaptive ComputationRecurrent Transformers

    Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.

  6. FISformer: Replacing Self-Attention with a Fuzzy Inference System in Transformer Models for Time Series Forecasting

    Mar 23, 2026Bulent Haznedar, Levent KaracanTime Series TransformersTime Series Forecasting

    Computer Engineering Department, Gaziantep University, Gaziantep, Türkiye

  7. NDT: Non-Differential Transformer and Its Application to Sentiment Analysis

    Mar 21, 2026Soudeep Ghoshal, Himanshu Buckchash, Sarita Paudel +1Transformer AttentionAttention Mechanisms

    Kalinga Institute of Industrial Technology (KIIT), Bhubaneswar, Odisha 751024, India. · IMC University of Applied Sciences Krems, Piaristengasse 1, 3500 Austria.

  8. NeuroGame Transformer: Gibbs-Inspired Attention Driven by Game Theory and Statistical Physics

    Mar 19, 2026Djamel Bouchaffra, Faycal Ykhlef, Hanene Azzag +2Transformer AttentionAttention Mechanisms

    Paris-Saclay University, UVSQ, DAVID Lab, 78035, Versailles, France · TSAM, ASM, Centre for Development of Advanced Technologies, Algiers, Algeria · LIPN, UMR CNRS 7030, Sorbonne Paris Nord University, Villetaneuse, France

  9. HAViT: Historical Attention Vision Transformer

    Mar 19, 2026Swarnendu Banik, Manish Das, Shiv Ram Dubey +1Vision TransformerTransformer Attention

    Computer Vision and Biometrics Lab, Indian Institute of Information Technology, Allahabad

  10. SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds

    Mar 17, 2026Viktor Stein, Wuchen Li, Gabriele SteidlTransformer AttentionAttention Mechanisms

    Institute of Mathematics, Technische Universität Berlin, Straße des 17. Juni 136, 10623 Berlin, Germany · Department of Mathematics, University of South Carolina, Columbia. 1523 Greene St, Columbia, SC 29208, USA

  11. Scaling Attention via Feature Sparsity

    Mar 17, 2026Yan Xie, Tiansheng Wen, Tangda Huang +4Transformer AttentionEfficient Attention

    School of Electronic Engineering, Xidian University · Stony Brook University · TUM +2

  12. Mixture-of-Depths Attention

    Mar 16, 2026Lianghui Zhu, Yuxin Fang, Bencheng Liao +10Transformer AttentionLanguage Model Scaling

    School of EIC, Huazhong University of Science & Technology · ByteDance Seed

  13. Directional Routing in Transformers

    Mar 16, 2026Kevin TaylorTransformerLLM Routing

  14. Learnable Pulse Accumulation for On-Device Speech Recognition: How Much Attention Do You Need?

    Mar 11, 2026Yakov Pyotr ShkolnikovTransformer AttentionAutomatic Speech Recognition

  15. Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning

    Mar 11, 2026Shirin Amiraslani, Xin GaoSelf-AttentionTransformer Attention

    Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada

  16. QV May Be Enough: Toward the Essence of Attention in LLMs

    Mar 11, 2026Zhang EdwardAttention MechanismsTransformer Attention

  17. Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems

    Mar 11, 2026Hailing ChengGenerative RecommendationSequential Recommendation

    Linkedin Inc Mountain View, California, USA

  18. Beyond Linearity in Attention Projections: The Case for Nonlinear Queries

    Mar 11, 2026Marko KarbevskiTransformer AttentionDecoder-Only Language Models

    In Simplicity Technologies

  19. Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias

    Mar 10, 2026Borun D ChowdhuryTransformerTransformer Attention

    Meta, London, W1T 1FB, UK

  20. Efficient Reasoning at Fixed Test-Time Cost via Length-Aware Attention Priors and Gain-Aware Training

    Mar 10, 2026Rian AtriTransformer AttentionEfficient Transformer Inference

    Serval Systems

  21. GIAT: A Geologically-Informed Attention Transformer for Lithology Identification

    Mar 10, 2026Jie Li, Qishun Yang, Nuo LiTransformer AttentionTransformer Interpretability

    College of Petroleum, China University of Petroleum-Beijing at Karamay, Karamay 834000, China

  22. The Dual-Stream Transformer: Channelized Architecture for Interpretable Language Modeling

    Mar 8, 2026J. Clayton Kerce, Alexis FoxTransformerTransformer Interpretability

    Georgia Tech Research Institute

  23. A Miniature Brain Transformer: Thalamic Gating, Hippocampal Lateralization, Amygdaloid Salience, and Prefrontal Working Memory in Attention-Coupled Latent Memory

    Mar 7, 2026Hong JeongMemory-Augmented Neural NetworksCognitive Modeling

    Department of Computer Information Engineering, Inha University in Tashkent, Uzbekistan

  24. Spectral Conditioning of Attention Improves Transformer Performance

    Mar 7, 2026Hemanth Saratchandran, Simon LuceyTransformer Attention

    Australian Institute for Machine Learning, Adelaide University

  25. Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy

    Mar 6, 2026Abdulrahman Alswaidan, Jeffrey D. VarnerHopfield NetworksEnergy-Based Models

    R.F. Smith School of Chemical and Biomolecular Engineering Cornell University, Ithaca, NY 14850

  26. Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions

    Mar 6, 2026Aditya Varre, Mark Rofin, Nicolas FlammarionGradient Descent DynamicsSoftmax Attention

    Theory of Machine Learning Lab, EPFL.

  27. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling

    Mar 5, 2026Ted Zadouri, Markus Hoehnerbach, Jay Shah +3Efficient AttentionGPU Kernel Optimization

    Princeton University · Together AI · Meta +3

  28. Layer by layer, module by module: Choose both for optimal OOD probing of ViT

    Mar 5, 2026Ambroise Odonnat, Vasilii Feofanov, Laetitia Chapel +2Vision TransformerOOD Generalization

    Noah’s Ark Lab, Inria · L’Institut Agro, IRISA · Univ. Rennes 2, IRISA +1

  29. The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology

    Mar 5, 2026Alper YıldırımGrokkingNeural Network Training Dynamics

    Independent Researcher

  30. Data-Aware Random Feature Kernel for Transformers

    Mar 4, 2026Amirhossein Farzam, Hossein Mobahi, Nolan Andrew Miller +1Transformer AttentionAttention Mechanisms

    Google DeepMind