Transformer Attention

Ask about this topic

Overview

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics
CardsList

  1. Kinetic theory for Transformers and the lost-in-the-middle phenomenon

    May 9, 2026Mitia Duerinckx, Borjan Geshkovski, Stefano RossiSelf-AttentionTransformer Attention

  2. When and Why Grouping Attention Heads Accelerates Muon Optimization

    May 9, 2026Hongtao Zhang, Wenjie Zhou, Wei Chen +1Attention Head AnalysisMuon Optimizer

    State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · University of Chinese Academy of Sciences

  3. Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention

    May 8, 2026Peter Súkeník, Cristina López Amado, Christoph H. Lampert +1Self-AttentionTransformer Attention

    Institute of Science and Technology (ISTA) Austria

  4. Convergent Stochastic Training of Attention and Understanding LoRA

    May 8, 2026Zhengkai Sun, Dibyakanti Kumar, Alejandro F Frangi +2Stochastic OptimizationLow-Rank Adaptation

  5. Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers

    May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformerClustering

    1RIKEN Center for Advanced Intelligence Project · 2Imperial College London · 3The University of Tokyo +1

  6. Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning

    May 8, 2026Zixuan Xie, Xinyu Liu, Claire Chen +3Reinforcement LearningTransformer-Based RL

    University of Virginia · California Institute of Technology · Purdue University

  7. Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent

    May 7, 2026Chenyang Zhang, Yuan CaoLogistic RegressionSelf-Attention

    School of Computing & Data Science, The University of Hong Kong

  8. Cubit: Token Mixer with Kernel Ridge Regression

    May 7, 2026Chuanyang Zheng, Jiankai Sun, Yihang Gao +6Kernel Ridge RegressionTransformer Attention

  9. FedFrozen: Two-Stage Federated Optimization via Attention Kernel Freezing

    May 7, 2026Junye Du, Zhenghao Li, Yushi Feng +1Transformer AttentionFederated Learning

    The University of Hong Kong

  10. Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer

    May 6, 2026Alexander Hsu, Zhaiming Shen, Wenjing Liao +1Transformer AttentionIn-Context Learning

    Department of Mathematics, Purdue University · School of Mathematics, Georgia Institute of Technology

  11. CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels

    May 6, 2026Xing Ma, Yangjie Zhou, Wu Sun +6GPU Kernel OptimizationGPU Acceleration

    Shanghai Jiao Tong University · National University of Singapore · Shanghai Qi Zhi Institute

  12. HEXST: Hexagonal Shifted-Window Transformer for Spatial Transcriptomics Gene Expression Prediction

    May 6, 2026Keunho Byeon, Jin Tae KwakComputational PathologySpatial Transcriptomics

    School of Electrical Engineering, Korea University, Seoul

  13. FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning

    May 6, 2026Waleed Razzaq, Yun-Bo ZhaoTransformerTransformer Attention

    School of Automation University of Science and Technology China Hefei, Anhui

  14. Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks

    May 5, 2026Yaobo ZhangRotary Positional EmbeddingsSelf-Attention

    School of Physics, Ningxia University Yinchuan, 750021, China

  15. Transformers with Selective Access to Early Representations

    May 5, 2026Skye Gunasekaran, Téa Wright, Rui-Jie Zhu +1TransformerResidual Learning

    UC Santa Cruz · UC Berkeley

  16. Cascade Token Selection for Transformer Attention Acceleration

    May 4, 2026Stephen J. ThomasSelf-AttentionTransformer Attention

    Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015

  17. Projection-Free Transformers via Gaussian Kernel Attention

    May 4, 2026Debarshi Kundu, Archisman Ghosh, Swaroop Ghosh +1TransformerSelf-Attention

    The Pennsylvania State University, University Park, USA

  18. Linear-Time Global Visual Modeling without Explicit Attention

    May 3, 2026Ruize He, Dongchen Han, Gao HuangTransformerTransformer Attention

    Tsinghua University

  19. Focus and Dilution: The Multi-stage Learning Process of Attention

    May 2, 2026Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1Self-AttentionTransformer Attention

    School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University. +2

  20. Spiking Sequence Machines and Transformers

    May 1, 2026Joy BoseTransformer AttentionSpiking Neural Networks

    Independent Researcher Bengaluru, India

  21. Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

    Apr 29, 2026Andrea Agazzi, Giuseppe Bruno, Eloy Mosig García +2Dynamical SystemsTransformer

    Department of Mathematics and Statistics, University of Bern (CH) · Department of Mathematics, University of Pisa (IT)

  22. Block-Wise Differentiable Sinkhorn Attention: Tail-Refinement Gradients with a Gap-Aware Dustbin Bridge

    Apr 28, 2026Dylan FordeAutomatic DifferentiationTransformer Attention

    Independent Researcher

  23. Kwai Summary Attention Technical Report

    Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Self-AttentionKV Caching

    Kuaishou

  24. Transformer as an Euler Discretization of Score-based Variational Flow

    Apr 26, 2026Huadong LiaoRepresentation LearningTransformer

  25. Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers

    Apr 26, 2026Giansalvo CirrincioneTransformer InterpretabilityRepresentation Collapse

    aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France

  26. A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification

    Apr 26, 2026Peng Chen, Wenxuan He, Feng Qian +2Hybrid CNN-Transformer ArchitecturesTransformer Attention

    College of Engineering, Shantou University, Shantou, 515063, Guangdong, China · School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, China · Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun, 130033, Jilin, China +2

  27. Adaptive Head Budgeting for Efficient Multi-Head Attention

    Apr 24, 2026Bilal Faye, Abdoulaye Mbaye, Hanane Azzag +1TransformerTransformer Attention

  28. HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models

    Apr 24, 2026Abhinaba BasuTransformer AttentionLanguage Modeling

    National Institute of Electronics and Information Technology (NIELIT)

  29. Dissociating Decodability and Causal Use in Bracket-Sequence Transformers

    Apr 24, 2026Aryan Sharma, Cutter Dawes, Shivam RavalTransformer InterpretabilitySelf-Attention

    1Yale University · 2Independent · 3Harvard University

  30. LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs

    Apr 23, 2026Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia +3Transformer AttentionLLM Inference Acceleration

    Openchip & Software Technologies, S.L.