Self-Attention
Overview
Self-attention lets each token build a context-dependent representation by weighting information from other tokens, and is central to language, vision, audio, and multimodal Transformers. Common approaches include multi-head softmax attention with positional encodings such as RoPE, sparse and block-sparse attention, and linear or recurrent variants that reduce the cost of processing long sequences. Researchers evaluate these methods on language-model perplexity, long-context retrieval and reasoning, and vision tasks, while also studying attention heads, information flow, and phenomena such as attention sinks. A central tension is how to cut computation and memory for long contexts without losing relevant information or the accuracy and flexibility of full attention.
Latest papers
231 papers4 papers in the last four weeks, down 71% on the four weeks before. 0.0% of all new papers.