Softmax Attention

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 47

All topics

Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Softmax Attention Self-Attention
Apple
Scaling Limits of Long-Context Transformers
May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Softmax Attention Self-Attention
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
May 8, 2026Mingsong Yan, Dongyang Li, Charles Kulick +1Softmax Attention Transformer Interpretability
Department of Mathematics, University of California, Santa Barbara, CA
Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimization
May 8, 2026Navid Rezazadeh, Arash Gholami DavoodiSoftmax Attention Neural Network Verification
University of California, Irvine · Carnegie Mellon University
Nearly Optimal Attention Coresets
May 7, 2026Edo Liberty, Alexandr Andoni, Eldar KleinerSoftmax Attention Efficient Attention
Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention
May 5, 2026Ayan PendharkarSoftmax Attention Attention Head Analysis
Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models
Apr 29, 2026Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller +1Softmax Attention Self-Attention
Merck & Co., Inc., Cambridge, MA, USA
Emergent Self-Attention from Astrocyte-Gated Associative Memory Dynamics
Apr 28, 2026Arnau Vivet, Alex ArenasSoftmax Attention Dynamical Systems
1Departament d’Enginyeria Inform`atica i Matem`atiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain · 2Complexity Science Hub Vienna, Metternichgasse 8, 1030 Vienna, Austria
QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
Apr 28, 2026Sehyeon Oh, Yongin Kwon, Jemin LeeSoftmax Attention GPU Acceleration
University of Science and Technology, Daejeon, Republic of Korea · Electronics and Telecommunications Research Institute, Daejeon, Republic of Korea · Pusan National University, Busan, Republic of Korea +1
Transformer Approximations from ReLUs
Apr 27, 2026Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1Softmax Attention Neural Network Approximation Theory
Center for Foundation Models and Generative AI, Northwestern University, Evanston, IL 60208, USA · Department of Computer Science, Northwestern University, Evanston, IL 60208, USA · Department of Physics, National Taiwan University, Taipei 10617, Taiwan +1
ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers
Apr 26, 2026Chih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao +1Efficient Transformer Inference Softmax Attention
Advanced Computer Vision Laboratory, National Yang Ming Chiao Tung University
Sinkhorn doubly stochastic attention rank decay analysis
Apr 9, 2026Michela Lapenna, Rita Fioresi, Bahman GharesifardSoftmax Attention Self-Attention
Department of Physics and Astronomy, University of Bologna, Bologna, Italy · Department of Pharmacy and Biotechnologies, University of Bologna Bologna, Italy · Department of Mathematics and Statistics, Queen’s University, Kingston, ON, Canada
Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators
Feb 9, 2026Gunn KimSoftmax Attention
Department of Physics, Sejong University, Seoul 05006, Republic of Korea
Selective Rotary Position Embedding
Nov 21, 2025Sajad Movahedi, Timur Carstensen, Arshia Afzal +3Softmax Attention Rotary Positional Embeddings
ELLIS Institute Tübingen1, LIONS, EPFL2, University of Freiburg3, Max-Planck-Institute for Intelligent Systems4, Prior Labs5
Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective
Oct 4, 2025Ruoxi Yu, Haotian Jiang, Jingpu Cheng +3Softmax Attention Attention Head Analysis
Center for Data Science, Peking University · Institute for Functional Intelligent Materials, National University of Singapore · Department of Mathematics, National University of Singapore +1
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model Pretraining Softmax Attention
Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1
Attention by Synchronization in Coupled Oscillator Networks
Date pendingFabio Pasqualetti, Taosha GuoSoftmax Attention Dynamical Systems
Electrical Engineering and Computer Science University of California, Irvine

May 10, 2026, cs.LGJ/K move · Enter open · S save

Nectar: Neural Estimation of Cached-Token Attention via Regression

João Monteiro, Michal Klein, Pierre Ablin, Marco Cuturi

Apple

Softmax Attention Self-Attention KV Caching Long-Context Language Model Inference Efficient Attention Multi-Head Attention

Open paper AskPDF arXiv

Evaluating softmax attention over a fixed long context requires reading every cached key-value pair for each new query token. For a given context (a book, a manual, a legal corpus) the attention output is a deterministic function of the query. We propose Nectar, which fits a compact neural network to this function for queries drawn from a task-relevant distribution. Nectar fits two networks per layer and KV-head: a target network that predicts the attention output and a score network that predicts the log-normalizer. The pair plugs into the standard masked self-attention at inference time, replacing the

O(n)

attention over the cache with a forward pass whose cost does not depend on

n

. Each module carries on the order of

|θ|

parameters per layer and KV-head, typically much smaller than the

2nd

KV-cache footprint at the same granularity. We report experiments on models from 1.7B to 8B parameters across five long-context datasets. The approximation error tracks the next-token accuracy gap to full attention, and allocating capacity non-uniformly across layers reduces that gap in our ablation. Beyond this analysis of metrics, we check that the text generations (following a question prompt) of a model equipped with a Nectar module match in semantic content those obtained by giving the same model access to the full cache.

Momentum

Related topics

Latest papers 47