Softmax Attention

Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 47

All topics
CardsList

  1. Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must

    Oct 8, 2026Simon Gabet, Etienne Boursier, Claire BoyerSoftmax AttentionIn-Context Learning

    LMO · Université Paris-Saclay, CNRS, Inria, Laboratoire de mathématiques d’Orsay, 91405 Orsay, France · LMO, CELESTE +2

  2. The Identifiability and Observability of Deep Normalized Attention

    Oct 7, 2026Pranav Venkata KondaAttention MechanismsRepresentation Identifiability

    Columbia University

  3. Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

    Oct 5, 2026Hong Ha Le, Jackie Lok, Atsushi Nitanda +1Language Model PretrainingSoftmax Attention

    National University of Singapore · Princeton University · Nanyang Technological University

  4. Universal interpolation for deep residual self-attention networks

    Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax AttentionTransformer

    Department of Computer Science New York University

  5. Attention Kernels for Learning Maps Between Heavy-Tailed Measures

    Sep 30, 2026Kailen Hargenrader, Edoardo Calvello, Bohan ChenSoftmax AttentionAttention Mechanisms

    Computing and Mathematical Sciences California Institute of Technology · Lawrence Berkeley National Laboratory University of California, Berkeley ICSI

  6. Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training

    Sep 28, 2026Junlin Chen, Daize Dong, Huanwei Di +9Softmax AttentionEfficient Attention

    Rutgers University · Carnegie Mellon University · Oracle +2

  7. Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration

    Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiEfficient Transformer InferenceSoftmax Attention

    University of Illinois Urbana-Champaign

  8. FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward

    Sep 27, 2026Sriman AchantaSoftmax AttentionLLM Inference Acceleration

    Virginia Commonwealth University

  9. FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

    Sep 21, 2026Anthony Givans, Michael Crawshaw, Mingrui LiuSoftmax AttentionAutomatic Differentiation

  10. Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

    Sep 18, 2026Richard Zhe WangSoftmax AttentionGated Attention

    St. John Fisher University, Rochester, New York, USA.

  11. Multi-Head Self Attention is a Parameter Identification Mechanism

    Sep 1, 2026W. Ross MorrowSoftmax AttentionParameter Identifiability

  12. Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

    Aug 28, 2026Yuhe Sui, Jianing Zhang, Yingzhi TangSoftmax AttentionLow-Rank Attention

    Nanyang Technological University, Singapore · Carnegie Mellon University

  13. A Quantum Roadmap for Softmax Attention: Exact Born-Rule Analogs for Softmax Attention on the Probability Simplex

    Aug 11, 2026Eric A. F. Reinhardt, Adam J. HauserSoftmax AttentionQuantum Machine Learning

    Department of Physics and Astronomy University of Alabama, Tuscaloosa, AL 35487, USA

  14. Provably Learning Multi-Head Attention with Queries

    Aug 4, 2026Sunyeop Kim, Insung Kim, Jian GuoSoftmax AttentionTransformer

    Korea University, Republic of Korea · Nanyang Technological University, Singapore

  15. One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse

    Aug 3, 2026Shuxiao Xie, Shuyang Xie, Yuan Cao +3Softmax AttentionTransformer Attention

    Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC), China · Fudan University Institute of Systems for Advanced Computing, China · Harbin Institute of Technology, China +1

  16. ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

    Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangSoftmax AttentionKV Caching

    University of Electronic Science and Technology of China

  17. Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere

    Jul 27, 2026Hao YeSoftmax AttentionRotary Positional Embeddings

    Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences · University of Chinese Academy of Sciences

  18. The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

    Jul 25, 2026Byeong Hoon YoonSoftmax AttentionTransformer Expressivity

    Independent Researcher

  19. Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

    Jul 21, 2026Subham Singh, Ashutosh Mishra, Subha RautSoftmax AttentionRotary Positional Embeddings

    Department of Mathematics and Statistics, Mississippi State University · Independent Researcher

  20. BMFA: Boundary-Minority Free-Energy Adaptive Screening

    Jul 20, 2026Wenyan Xu, Alizer WongSoftmax AttentionVision Transformer

    1Guangdong University of Technology · School of Computer Science, Peking University · 3ManXis

  21. The Key to Going Linear: Analysis-Driven Transformer Linearization

    Jul 8, 2026Anna Kuzina, Paul N. Whatmough, Babak Ehteshami BejnordiSoftmax AttentionLLM Inference Acceleration

    Qualcomm AI Research∗ · Qualcomm AI Research

  22. HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

    Jul 5, 2026Hui Dong, Yanzhao Li, Jie Gao +5Softmax AttentionLLM Quantization

    Huawei Technologies

  23. Asymptotic Signal Subspace Recovery in Softmax Attention Models

    Jun 21, 2026Lan V. TruongSoftmax AttentionGradient Descent Dynamics

    Faculty of Computer Science and Engineering Ho Chi Minh City University of Technology (HCMUT) · Vietnam National University Ho Chi Minh City (VNU-HCM), Vietnam

  24. HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

    Jun 18, 2026Zhentao Tan, Wei Chen, Jingyi Shen +4Softmax AttentionLong-Context Modeling

    Alibaba Group

  25. Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence

    Jun 10, 2026Itay Lavie, Kirsten Fischer, Andrey Lekov +3Softmax AttentionPhase Transitions

    Racah Institute of Physics, Hebrew University of Jerusalem, Jerusalem 91904, Israel · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA · Institute for Advanced Simulation (IAS-6), Computational and Systems Neuroscience, Jülich Research Center, Jülich 52428, Germany +3

  26. P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8

    Jun 2, 2026Reed LauSoftmax AttentionAttention Sinks

    Tencent

  27. Forget Attention: Importance-Aware Attention Is All You Need

    Jun 1, 2026Suhyeong Shin, Yeongwook YangSoftmax AttentionLong-Context Language Modeling

    Department of Computer Engineering Kangwon National University

  28. A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention

    May 12, 2026Tomohiro Hayase, Ryo KarakidaSoftmax AttentionSelf-Attention

    AIST

  29. Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition

    May 11, 2026Haoren Xu, Guanhua FangSoftmax AttentionSelf-Attention

    Fudan University

  30. Complex-Valued Phase-Coherent Transformer

    May 11, 2026Leona HiokiSoftmax AttentionComplex-Valued Neural Networks