Overview

Softmax attention turns query–key similarity scores into normalized weights for combining value vectors, enabling Transformers to retrieve context-dependent information. Research studies its expressivity, training dynamics, concentration, stability, and behaviors such as attention sinks, while comparing alternatives including sigmoid, ReLU, and Sinkhorn normalization. For long sequences and constrained hardware, methods such as FlashAttention, sparse summaries, coresets, quantization, and low-precision kernels aim to reduce memory or computation while preserving useful accuracy. A central tension is that exact dense attention is expensive, but approximations that work well on particular models or inputs may lack uniform guarantees or introduce quality and stability costs.

Latest papers

CardsList

82 papers6 papers in the last four weeks, down 65% on the four weeks before. 0.0% of all new papers.

  1. The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale

    Nov 17, 2026Fırat Öncel, Cem Subakan, Mirco Ravanelli +1Efficient AttentionLinear Attention

  2. A Geometric Theory for Nonequilibrium Transformer Dynamics

    Nov 7, 2026Aythami Bethencourt de León, Darryl D. HolmTransformer AttentionSoftmax Attention

  3. Fused-Softmax for efficient quantized attention

    Nov 6, 2026Tianxing Wu, Juuso Korhonen, Jesus Carabano BravoSoftmax AttentionGPU Kernel Optimization

  4. Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds

    Nov 4, 2026Zixuan Huang, Guozhang ChenEfficient AttentionSoftmax Attention

  5. Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms

    Nov 2, 2026Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1Softmax AttentionMulti-Head Attention

  6. Spectral Degeneration of Self-Attention under Isotropic Score Geometry

    Nov 2, 2026Mengda Li, Jianfeng YaoSelf-AttentionSoftmax Attention

  7. FlyWheel: Attention Kernels Built for the TPU

    Nov 1, 2026Zishuo Bao, Zeshen Zhang, Linkun Chen +1Neural Network Hardware AccelerationLLM Inference Acceleration

  8. First Passage Attention

    Oct 15, 2026Yuxuan LiuLong-Context Language ModelingSoftmax Attention

  9. Continuum Attention

    Oct 14, 2026Afshin KhadangiCausal AttentionTransformer Attention

  10. Asymptotic Behavior of Transformers

    Oct 14, 2026Katy Blumer, Da-Cheng Juan, Jon Kleinberg +3Transformer Training DynamicsTransformer

  11. Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry

    Oct 13, 2026Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno +2Muon OptimizerSoftmax Attention

  12. Counting with Attention and LayerNorm: Norm Scaling and Representation Geometry

    Oct 12, 2026Ayyub Shaffy, Manoj Dhakal, Safal Shrestha +1Self-AttentionLayer Normalization

  13. Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must

    Oct 8, 2026Simon Gabet, Etienne Boursier, Claire BoyerClassificationSoftmax Attention

  14. Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

    Oct 5, 2026Hong Ha Le, Jackie Lok, Atsushi Nitanda +1Language Model PretrainingSoftmax Attention

  15. Universal interpolation for deep residual self-attention networks

    Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax AttentionTransformer

  16. Attention Kernels for Learning Maps Between Heavy-Tailed Measures

    Sep 30, 2026Kailen Hargenrader, Edoardo Calvello, Bohan ChenSoftmax AttentionAttention Mechanisms

  17. Efficiently Approximating Attention Is Hard

    Sep 29, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Attention ApproximationEfficient Attention

  18. Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training

    Sep 28, 2026Junlin Chen, Daize Dong, Huanwei Di +9Softmax AttentionEfficient Attention

  19. Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration

    Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiEfficient Transformer InferenceSoftmax Attention

  20. A Comparative Analysis of Attention versus State-Space Models for In-Context Learning

    Sep 26, 2026Enes Arda, Semih Cayci, Atilla EryilmazIn-Context RegressionState Space Models

  21. Emergent One-Third Scaling Law as Attention Tries to Concentrate

    Sep 26, 2026Yizhou Liu, Sara Kangaslahti, Jeff GoreLanguage Model Scaling LawsScaling Laws

  22. Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

    Sep 18, 2026Richard Zhe WangSoftmax AttentionGated Attention

  23. Exact Finite Attention Responses From RoPE Derivatives

    Sep 12, 2026Julie Huang, Maggie Chlon, Gregory Gutin +1Attention ApproximationCausal Interventions in Language Models

  24. RoPE attention is an exact forward-pass gradient step with softmax intact

    Sep 6, 2026Julie Huang, Maggie Chlon, Leon ChlonSoftmax AttentionRotary Positional Embeddings

  25. Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

    Sep 2, 2026Ankur Singh, Ashish Gautam, Shruti R. Kulkarni +1Softmax AttentionAnalog Computing