Softmax Attention

Softmax attention turns query–key similarity scores into normalized weights for combining value vectors, enabling Transformers to retrieve context-dependent information. Research studies its expressivity, training dynamics, concentration, stability, and behaviors such as attention sinks, while comparing alternatives including sigmoid, ReLU, and Sinkhorn normalization. For long sequences and constrained hardware, methods such as FlashAttention, sparse summaries, coresets, quantization, and low-precision kernels aim to reduce memory or computation while preserving useful accuracy. A central tension is that exact dense attention is expensive, but approximations that work well on particular models or inputs may lack uniform guarantees or introduce quality and stability costs.