Softmax Attention

Ask about this topic

Overview

Softmax attention turns query–key similarity scores into normalized weights for combining value vectors, enabling Transformers to retrieve context-dependent information. Research studies its expressivity, training dynamics, concentration, stability, and behaviors such as attention sinks, while comparing alternatives including sigmoid, ReLU, and Sinkhorn normalization. For long sequences and constrained hardware, methods such as FlashAttention, sparse summaries, coresets, quantization, and low-precision kernels aim to reduce memory or computation while preserving useful accuracy. A central tension is that exact dense attention is expensive, but approximations that work well on particular models or inputs may lack uniform guarantees or introduce quality and stability costs.

Latest papers

CardsList

75 papers12 papers in the last four weeks, up 9% on the four weeks before. 0.1% of all new papers.

  1. First Passage Attention

    Oct 15, 2026Yuxuan LiuLong-Context Language ModelingSoftmax Attention

    Department of Mathematics, University of Bergen, Bergen, Norway

  2. Continuum Attention

    Oct 14, 2026Afshin KhadangiCausal AttentionTransformer Attention

    SnT, University of Luxembourg

  3. Asymptotic Behavior of Transformers

    Oct 14, 2026Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3Transformer Training DynamicsTransformer

    Google Research

  4. Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry

    Oct 13, 2026Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2Muon OptimizerSoftmax Attention

    Institute of Foundation Models, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

  5. Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry

    Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-AttentionLayer Normalization

    New York University Abu Dhabi

  6. Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must

    Oct 8, 2026Simon Gabet, Etienne Boursier, Claire BoyerClassificationSoftmax Attention

    LMO · LMO, CELESTE · LMO, IUF

  7. The Identifiability and Observability of Deep Normalized Attention

    Oct 7, 2026Pranav Venkata KondaAttention MechanismsRepresentation Identifiability

    Columbia University

  8. Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

    Oct 5, 2026Hong Ha Le, Jackie Lok, Atsushi Nitanda +1Language Model PretrainingSoftmax Attention

    National University of Singapore · Princeton University · Nanyang Technological University

  9. Universal interpolation for deep residual self-attention networks

    Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax AttentionTransformer

    Department of Computer Science New York University

  10. Attention Kernels for Learning Maps Between Heavy-Tailed Measures

    Sep 30, 2026Kailen Hargenrader, Edoardo Calvello, Bohan ChenSoftmax AttentionAttention Mechanisms

    Computing and Mathematical Sciences California Institute of Technology · Lawrence Berkeley National Laboratory University of California, Berkeley ICSI

  11. Efficiently Approximating Attention Is Hard

    Sep 29, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Attention ApproximationEfficient Attention

    Institute for Data Science in Mechanical Engineering, RWTH Aachen University · Autonomous Systems Lab, Stanford University

  12. Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training

    Sep 28, 2026Junlin Chen, Daize Dong, Huanwei Di +9Softmax AttentionEfficient Attention

    Rutgers University · Carnegie Mellon University · Oracle +2

  13. Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration

    Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiEfficient Transformer InferenceSoftmax Attention

    University of Illinois Urbana-Champaign

  14. FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward

    Sep 27, 2026Sriman AchantaSoftmax AttentionLLM Inference Acceleration

    Virginia Commonwealth University

  15. A Comparative Analysis of Attention versus State-Space Models for In-Context Learning

    Sep 26, 2026Enes Arda, Semih Cayci, Atilla EryilmazState Space ModelsIn-Context Learning

    The Ohio State University · RWTH Aachen University

  16. Emergent One-Third Scaling Law as Attention Tries to Concentrate

    Sep 26, 2026Yizhou Liu, Sara Kangaslahti, Jeff GoreLanguage Model Scaling LawsScaling Laws

    Massachusetts Institute of Technology · Harvard University

  17. FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

    Sep 21, 2026Anthony Givans, Michael Crawshaw, Mingrui LiuSoftmax AttentionAutomatic Differentiation

  18. Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

    Sep 18, 2026Richard Zhe WangSoftmax AttentionGated Attention

    St. John Fisher University, Rochester, New York, USA.

  19. Exact Finite Attention Responses From RoPE Derivatives

    Sep 12, 2026Julie Huang, Maggie Chlon, Gregory Gutin +1Attention ApproximationCausal Interventions in Language Models

    Hassana Labs · Department of Computer Science, Royal Holloway, University of London · University of Oxford

  20. RoPE attention is an exact forward-pass gradient step with softmax intact

    Sep 6, 2026Julie Huang, Maggie Chlon, Leon ChlonSoftmax AttentionRotary Positional Embeddings

    Hassana Labs · Department of Information Engineering, Oxford University

  21. Query-Oblivious Coresets for Softmax Attention: Improved Bounds and Efficient Constructions

    Sep 6, 2026Ofek I. CohenCoreset SelectionSoftmax Attention

    Tel Aviv University.

  22. Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

    Sep 2, 2026Ankur Singh, Ashish Gautam, Shruti R. Kulkarni +1Softmax AttentionAnalog Computing

    Department of Electrical and Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA. · ORNL Oak Ridge, Tennessee, USA.

  23. Multi-Head Self Attention is a Parameter Identification Mechanism

    Sep 1, 2026W. Ross MorrowSoftmax AttentionParameter Identifiability

  24. Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

    Aug 28, 2026Yuhe Sui, Jianing Zhang, Yingzhi TangSoftmax AttentionLow-Rank Attention

    Nanyang Technological University, Singapore · Carnegie Mellon University

  25. How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

    Aug 26, 2026Gerard Conangla PlanesAttention ApproximationLow-Rank Adaptation

    Aily Labs

  26. From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics

    Aug 21, 2026Heyang GongAttention MechanismsSelf-Attention

  27. A Quantum Roadmap for Softmax Attention: Exact Born-Rule Analogs for Softmax Attention on the Probability Simplex

    Aug 11, 2026Eric A. F. Reinhardt, Adam J. HauserSoftmax AttentionQuantum Machine Learning

    Department of Physics and Astronomy University of Alabama, Tuscaloosa, AL 35487, USA

  28. Provably Learning Multi-Head Attention with Queries

    Aug 4, 2026Sunyeop Kim, Insung Kim, Jian GuoSoftmax AttentionTransformer

    Korea University, Republic of Korea · Nanyang Technological University, Singapore

  29. One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse

    Aug 3, 2026Shuxiao Xie, Shuyang Xie, Yuan Cao +3Softmax AttentionTransformer Attention

    Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China · Fudan University Institute of Systems for Advanced Computing, China · Harbin Institute of Technology, China +1

  30. ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

    Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangSoftmax AttentionKV Caching

    University of Electronic Science and Technology of China