cs.LGApr 11, 2026
SaveAttention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation
Organizations: Tsinghua University · Meituan LongCat Team · The University of Hong Kong · University of Michigan · Xiamen University · The Ohio State University · Columbia University · Shanghai Artificial Intelligence Laboratory · The Hong Kong Polytechnic University
Abstract
As the foundational architecture of modern machine learning, Transformers have driven remarkable progress across diverse AI domains. Despite their transformative impact, a persistent challenge across various Transformers is Attention Sink (AS), in which a disproportionate amount of attention is focused on a small subset of specific yet uninformative tokens. AS complicates interpretability, significantly affecting the training and inference dynamics, and exacerbates issues such as hallucinations. In recent years, substantial research has been dedicated to understanding and harnessing AS. However, a comprehensive survey that systematically consolidates AS-related research and offers guidance for future advancements remains lacking. To address this gap, we present the first survey on AS, structured around three key dimensions that define the current research landscape: Fundamental Utilization, Mechanistic Interpretation, and Strategic Mitigation. Our work makes a pivotal contribution by highlighting the key concepts and main trends in the field, guiding researchers through the evolution of AS-related studies. We envision this survey as a valuable resource, empowering researchers to effectively manage AS within the current Transformer paradigm, while simultaneously inspiring innovative advancements for the next generation of Transformers. The paper list of this work is available at https://github.com/ZunhaiSu/Awesome-Attention-Sink.
Figures & tables
Figure 1: Overview of the survey structure.
Figure 3: Cumulative publication count and temporal trends in AS research from 2023 to 2026. Early research focused on Fundamental Utilization of AS, followed by studies investigating Mechanistic Interpretation , and most recently, efforts targeting Strategic Mitigation to address AS and improve model robustness.
Figure 4: Architecture of the standard Transformer and an illustration of typical AS, where sink tokens exhibit exceptionally high attention scores.
Figure 5: AS in BERT. Each point corresponds to the average attention a particular BERT attention head puts toward a token type. Left: heads often attend to “special” tokens. Early heads attend to [CLS] , middle heads attend to [SEP] , and deep heads attend to periods and commas. Often more than half of a head’s total attention is to these tokens. Right: heads attend to [SEP] tokens even more when the current token is [SEP] itself. The figure is adapted from [ 66 ] .
Figure 6: Visualization of average attention logits across Llama-2-7B. Two distinct structural patterns are observed: (i) The initial layers (layers 0 and 1) exhibit a "local" attention distribution, where attention is predominantly allocated to the most recent context. (ii) In subsequent deeper layers, the model demonstrates a consistent and pronounced concentration of attention toward the initial token across all heads. The figure is adapted from [ 25 ] .
Figure 7: Structural overview of a representative decoder-only LLM. Adapted from [ 29 ] .
Figure 8: Decoder Architecture of MoE LLM. The figure is adapted from [ 43 ] .
Figure 9: Expert router score distributions for sink and non-sink tokens. Sink tokens receive particularly high scores in super experts, whereas non-sink tokens have more evenly distributed scores across all experts. The figure is adapted from [ 43 ] .
Figure 10: Visualization and characterization of Visual Attention Sinks in MLLMs. Semantically irrelevant visual tokens (indicated by red boxes) exhibit Massive Activations within specific dimensions of their hidden states. In contrast, task-relevant visual tokens (indicated by blue boxes) maintain stable activation profiles without such numerical anomalies. This phenomenon mirrors the behavior of established text AS, suggesting a consistent underlying mechanism of AS across both visual and textual modalities. The figure is adapted from [ 49 ] .
Figure 11: A summary of outlier and AS analysis for ViT. (a) An input image. (b) Outliers in the output of layer 11. (c) Cumulative attention weight spent on every patch, showing that attention is concentrated on background patches. (d) Corresponding matrix of attention probabilities. (e) Average magnitude of values for outlier and non-outlier patches, indicating that patches with high attention scores have low value magnitudes. The figure is adapted from [ 30 ] .
Figure 12: Ablation studies on rolling diffusion window, mixed training strategy, and AS in Rolling Forcing . AS allows the model to preserve key-value states of initial frames as a global context anchor, thereby enhancing long-term global consistency in long-horizon streaming video generation tasks. The figure is adapted from [ 94 ] .
Figure 13: StreamingLLM retains the AS alongside recent tokens for stable attention computation. This approach enables efficient and stable performance on extended texts. The figure is adapted from [ 25 ] .
Figure 14: The three sparse attention patterns in MInference , with sink token protection incorporated. The figure is adapted from [ 79 ] .
Figure 15: Visualization of attention maps in the Llama-2-7B model. Streaming heads primarily focus on initial and recent tokens without emphasizing past contextual relevance. The figure is adapted from [ 111 ] .
Figure 16: Overview of Visual Attention Redistribution (VAR) . (a) Image-centric heads are selected based on the visual non-sink ratio; heads satisfying are designated as image-centric heads. (b) VAR reallocates surplus attention from sink tokens to visual non-sink tokens. The attention budget accumulates a fraction of the attention scores from sink tokens, which is then distributed to visual non-sink tokens. The figure is adapted from [ 49 ] .
Figure 17: Visualization of average attention logits comparing models pre-trained without (left) and with (right) a sink token. Both maps show the same layers and heads. Key observations: (1) Without a sink token, models exhibit local attention in lower layers and increased attention to initial tokens in deeper layers. (2) With a sink token, clear attention is directed to it across all layers, effectively collecting redundant attention. (3) With the presence of the sink token, less attention is given to other initial tokens, supporting the benefit of designating a sink token to enhance streaming performance. The figure is adapted from [ 25 ] .
Figure 18: Activation magnitudes in LLaMA2-7B before and after applying CushionCache . By inserting and tuning several prefix tokens that act as AS, CushionCache mitigates activation outliers in subsequent tokens, enabling effective activation quantization with coarse granularities. The figure is adapted from [ 53 ] .
Figure 19: Visualization of attention maps with and without register tokens. Without registers, attention maps are noisy and often focus on background patches. With registers, attention becomes cleaner and more focused on foreground objects, demonstrating that register tokens effectively absorb attention artifacts. The figure is adapted from [ 52 ] .
Figure 20: Illustration of AS in MLLM responses. The sink token exhibits a columnar high-attention pattern. Hallucinated responses are highlighted in indigo. The figure is adapted from [ 55 ] .
Figure 21: Schematic overview of backdoor attacks in LLM unlearning. (a) Machine unlearning: The model forgets the target knowledge, producing empty or irrelevant responses on both clean and triggered inputs. (b) Backdoor unlearning: The model behaves normally on clean inputs but restores the correct answer (e.g., “The Golden Snitch”) when the trigger appears. (c) AS indicate “where” to backdoor: Because AS emerge on shallow tokens near the sequence start, prefix triggers align with these sinks, concentrate attention, and enable recovery; infix or suffix placements misalign and fail. (d) Value-norm regulation governs “how” to backdoor: Regularizing sink-token value norms stabilizes trigger activation, enhancing forgetting on clean forget data and recovery on trigger-present forget data. The figure is adapted from [ 34 ] .
Figure 22: Visualization of self-attention patterns in BERT-base, showing attention probabilities (left), value magnitudes (middle), and their product (right) for attention head 3. Sink tokens such as [SEP] receive high attention but exhibit small value outputs, consistent with the no-op behavior predicted by the theory. The figure is adapted from [ 30 ] .
Figure 23: Analysis of sink token properties. (a) High cosine similarity of QK states. (b), (c), and (e) illustrate QKV states, showing that sink tokens exhibit significantly smaller value magnitudes. (f) Visualizes the attention output, demonstrating the minimal residual contribution of sink tokens. The figure is adapted from [ 29 ] .
Figure 24: (Left) Comparison of attention maps using Softmax versus Softpick and overall sink rate of the 340M models. (Right) Largest hidden state activation per layer of the 340M models. Softpick significantly mitigates both AS and large activations. The figure is adapted from [ 59 ] .
Figure 25: Systematic outliers in LLaMA2-7B. Outliers are identified in four locations: activations (layer outputs and down-projection inputs ), weights (down-projection matrices ), and attention (attention weights ). The figure is adapted from [ 57 ] .
Figure 26: The emergence of activation outliers from weight outliers. The figure is adapted from [ 57 ] .
Figure 27: The spread of attention outliers from activation outliers (AS). Activation outliers influence the self-attention mechanism. The figure is adapted from [ 57 ] .
Figure 28: Systematic outlier mechanism in Qwen3-30B-A3B MoE LLM. The figure is adapted from [ 43 ] .
Figure 29: Cross-layer evolution of extreme activation outliers in LLaMA2-7B. Activation outliers and AS exhibit a systematic and stable interaction. The figure is adapted from [ 29 ] .
Figure 30: Value updates from AS tokens are essentially the same. The figure is adapted from [ 47 ] .
Figure 31: (a) depicts the average cosine similarity of across all tokens for each head on LLaMA2-7B, showing that the values are consistently close to one across different tokens. (b) visualizes the attention biases for several example heads, where remains nearly constant. The figure is adapted from [ 29 ]
Figure 32: PCA visualization of positional vectors. After the first layer, only the initial tokens (e.g., positions 1–4) exhibit distinct positional vectors, whereas later tokens converge to similar representations. The figure is adapted from [ 137 ] .
Figure 33: Cosine similarity of normalized hidden states across layers. (a)-(b) Sink token maintains high similarity even between distant layers. (c)-(d) Another token shows similarity only between adjacent layers. The red boundary indicates layers after . These results highlight the static geometric nature of the sink token. The figure is adapted from [ 138 ] .
Figure 34: The presence of AS modulates information flow between tokens, making Transformer models more robust to perturbations in input prompts. This figure illustrates how a perturbation in the second token’s input representation (highlighted in red) propagates to other token embeddings throughout the model, both without (left) and with (right) a sink token (e.g., ⟨BOS⟩ ). The sink token diverts attention away from other tokens, limiting the spread of the perturbed information and resulting in more stable embeddings. Adapted from [ 28 ] .
Figure 35: Evolution of attention patterns in Pythia 410M, highlighting representative heads at layers 0, 16, and 23. Early layers exhibit diffuse attention that facilitates broad information mixing. Middle layers display sink patterns that restrict mixing, while late layers show sharp positional patterns enabling selective refinement. Adapted from [ 41 ] .
Figure 36: A schematic illustration of gated attention. Adapted from [ 30 ] .
Figure 37: Gating position exploration and performance comparison. Left: Investigated positions for applying gating operations. Middle: Performance of 15B MoE models. Gating after SDPA (G1) yields the best overall results; gating after the Value layer (G2) also improves performance, particularly in perplexity. Right: Training loss over 3.5T tokens for baseline vs. SDPA-gated 1.7B dense models. Gating reduces final loss and enhances stability by mitigating loss spikes, enabling higher learning rates and better scaling. The figure is adapted from [ 27 ] .
Figure 38: AS mitigation with Gated Attention . Left: Proportion of attention allocated to the initial token per layer. The baseline model devotes 46.7% of attention scores (averaged across layers) to the first token; gating reduces this to 4.8%. Right: Average attention map weights per head. In layer 21, the baseline AS (83% on the first token) drops to 4% with gating. The figure is adapted from [ 27 ] .
Figure 39: Architecture of Value-State Gated Attention . Unlike vanilla attention or input-state gated attention, VGA introduces a value-state gating mechanism to modulate the attention output. The figure is adapted from [ 44 ] .
Figure 40: Top: Mean attention map across all heads and layers of GPT2-Medium (baseline): the first token dominates attention (red box). Mean hidden state across layers: outlier activations emerge in specific feature dimensions (red box); the first token position exhibits the most extreme outliers (red circle). Bottom: Replacing canonical Softmax with Softmax-1 eliminates first-token dominance. The figure is adapted from [ 58 ] .
Figure 41: Attention maps of Softmax and Softpick . Using Softpick effectively eliminates AS. The figure is adapted from [ 59 ] .
Figure 42: Activation distribution in 1.4B models trained on 100B tokens. Three optimization strategies: (a) Adam, (b) Muon, (c) OSP . Muon alone provides insufficient outlier mitigation; OSP eliminates outliers. Adapted from [ 42 ] .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Paper | § 3 Utilization | § 4 Interpretation | § 5 Mitigation | § 6 Applications | Venue | Year | Link |
|---|---|---|---|---|---|---|---|
| Classical Language Models | |||||||
| [ 183 ] | Attention Redistribution | - | - | Mechanism Interpretability | ArXiv | 2026 | Link |
| [ 72 ] | - | Geometric Anchoring | - | Mechanism Interpretability | NeurIPS | 2025 | Link |
| [ 123 ] | Learnable Prefix Tokens | Geometric Anchoring | - | General Capability Enhancement | ArXiv | 2025 | Link |
| [ 67 ] | Attention Redistribution | - | - | Model Tuning Mechanism Interpretability | COLM | 2024 | Link |
| [ 30 ] | - | Softmax Limitations & No-Op Theory Outlier Circuits | Gated Attention Modified Softmax | Model Pre-training Efficient Inference | NeurIPS | 2023 | Link |
Table 1: Summary of Surveyed Papers. Each paper is annotated with tags corresponding to specific aspects of Fundamental Utilization (§ 3 ), Mechanistic Interpretation (§ 4 ), and Strategic Mitigation (§ 5 ) of AS. As most studies do not target all three key aspects, the symbol “-” denotes the absence of a particular dimension in a given work.