cs.AIMar 5, 2026
SaveThe Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks
Organizations: New York University.
Abstract
We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.
Figures & tables
Figure 1: Top-3 channel magnitudes across depth in Llama 2 7B and Qwen3 8B (post-residuals vs. block outputs). In both models, early blocks inject massive activations that persist through most of the network before being neutralized by late blocks.
| Model | # Blocks | Step-Up | Step-Down |
| Llama 2 7B | 64 | 4 | 62 |
| Llama 2 13B | 80 | 8 | 78, 79 |
| Llama 3 8B | 64 | 4 | 64 |
| Qwen2.5 7B | 56 | 8, 10 | 54, 55 |
| Qwen2.5 14B | 96 | 10 | 90, 92, 94, 95 |
| Qwen3 8B | 72 | 14 | 70, 72 |
Table 1: Step-up and step-down block indices across models. Step-up blocks appear near the beginning of the network and step-down blocks near the end, confining massive activations to intermediate layers. Odd and even indices denote attention and feed-forward blocks, respectively.
Figure 2: Input-output characteristics of in step-up and step-down blocks of Llama 2 7B. Based on randomly sampled sentences from C4 dataset ( Raffel et al., 2020 ) , we plot the cosine similarity and norm ratio for each token. Points are colored by the maximum magnitude of the block output. For spike tokens (red points), both direction and norm remain largely unchanged, indicating that the gate operates in a near-identity regime.
Figure 3: Frobenius norms for the quadratic forms in Llama 2 7B. Spike channels align with matrices that have substantially larger norms than typical channels. These high-norm coordinates appear exclusively in step-up and step-down blocks.
Figure 4: Eigenvalue spectra of for spike vs. non-spike channels in Llama 2 7B. Spike channels exhibit a single dominant eigenvalue that is orders of magnitude larger than the remainder of the spectrum; non-spike channels show no such outlier.
| Model | # Vocab | # Spike Token | Ratio |
| Llama 2 7B | 32,000 | 31,887 | 99.65% |
| Llama 2 13B | 32,000 | 31,889 | 99.65% |
| Llama 3 8B | 128,256 | 127,956 | 99.77% |
| Qwen 2.5 7B | 152,064 | 149,587 | 98.40% |
| Qwen 2.5 14B | 152,064 | 149,645 | 98.40% |
| Qwen 3 8B | 151,936 | 151,830 | 99.93% |
Table 2: Ubiquity of initial spikes across diverse LLMs. For nearly all evaluated models, positional occupancy at the initial position induces massive activations in intermediate layers, independent of the token’s semantic identity.
Figure 5: Cosine similarity among spike tokens before and after step-up block in Llama 2 7B. Pre-step-up representations vary across spike tokens, but post-step-up representations collapse to nearly identical directions, empirically validating the near-constant approximation.
Figure 6: t-SNE visualization of query and key vectors for a representative sink head (left) and non-sink head (right). In the sink head, sink keys lie closer to than non-sink keys , creating large logit gaps. In the non-sink head, and are approximately equidistant from , preventing the formation of a privileged sink position.
| Setup | Perplexity | Sink Ratio | Spike |
| Base Learning Rate | |||
| 11.8 | 18.6% | 1447 | |
| 10.7 | 31.8% | 2251 | |
| 10.1 | 46.0% | 3818 | |
| 10.0 | 51.5% | 3773 | |
| 10.2 | 39.2% | 2723 | |
Table 3: Optimization Hyperparameters ablations. We observe that the sink ratio could serve as a proxy for optimization health. Conversely, the magnitude of massive activations varies largely independently of both perplexity and the sink ratio. Highlighted rows denote the baseline result.
| Setup | Perplexity | Sink Ratio | Spike |
| Feed-Forward Block | |||
| GeLU | 10.1 | 69.3% | 3369 |
| Linear | 12.5 | 58.9% | 688 |
| Attention | 10.8 | 73.9% | 637 |
| SwiGLU | 10.1 | 46.0% | 3818 |
Table 4: Feed-forward block design ablations. Massive activations and attention sinks emerge across all evaluated designs, including attention-only and single linear configurations. Notably, GeLU and SwiGLU architectures yield significantly higher spike magnitudes by acting as efficient amplifiers. In contrast, linear and attention-only blocks exhibit much lower spikes as they require gradual accumulation across multiple layers. Highlighted rows denote the baseline result.
| Setup | Perplexity | Sink Ratio | Spike |
| Normalization | |||
| Sandwich | 9.8 | 44.7% | 520 |
| Sandwich (QK) | 10.0 | 42.0% | 92 |
| DynamicTanh | 10.0 | 61.0% | 153 |
| Pre-Norm | 10.1 | 46.0% | 3818 |
Table 5: Normalization configuration ablations. Applying post-block normalization (Sandwich) or element-wise transformations (DynamicTanh) successfully suppresses massive activations. Notably, the model still maintains a significant sink ratio through alternative strategies, demonstrating that sinks can exist independently of massive activations. Highlighted rows denote the baseline result.
| Setup | Perplexity | Sink Ratio | Spike |
| Number of Heads | |||
| 10.4 | 37.1% | 1253 | |
| 10.3 | 41.7% | 1936 | |
| 10.1 | 46.0% | 3818 | |
| Head Dimension | |||
| 11.3 | 4.1% | 291 | |
Table 6: Attention head settings ablations. Head dimension is the primary architectural driver of sink formation; larger dimensions provide the capacity for the attention subspace to isolate the sink keys. Concentrating capacity into fewer, larger heads intensifies sink behavior. Highlighted rows denote the baseline configuration.
| Setup | Perplexity | Sink Ratio | Spike |
| Conditional Gating | |||
| Channel | 10.0 | 4.5% | 202 |
| Head | 10.1 | 6.4% | 186 |
| Single | 10.2 | 31.2% | 316 |
| Unconditional Gating | |||
| Channel | 10.1 | 42.2% | 1922 |
Table 7: Gated attention ablations. Conditional gating—where the gate is a function of current representation—eliminates the need for attention sinks when applied per channel or per head. This suggests that sinks function as a “learned gate” to balance head contributions. Unconditional or static gates (positional or token-based) fail to suppress sinks, as they lack the dynamic, input-dependent routing necessary to substitute for sink behavior.
| Setup | Perplexity | Sink Ratio | Spike |
| Context Length (min/max) | |||
| 1/256 | 12.4 | 42.1% | 5411 |
| 1/1024 | 10.6 | 46.3% | 4442 |
| 1/4096 | 10.1 | 46.0% | 3818 |
| 1024/4096 | 10.1 | 13.0% | 38470 |
| 1024/5120 | 10.1 | 8.0% | 42365 |
Table 8: Context-length ablations. Attention sinks are largely induced to facilitate short-context prediction. When the training distribution is restricted to long sequences, the sink ratio collapses, indicating that sinks are primarily utilized to support short-range dependence. Highlighted rows denote the baseline configuration.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
| Layers | 32 |
| Hidden size | 4096 |
| Attention heads | 32 |
| Head dimension | 128 |
| Intermediate size | 11008 |
| Vocabulary size | 32000 |
Table 9: Baseline training configurations.
| Model Family | Model Name | Layers | Dimensions | Heads | Huggingface Model Id |
| Llama | Llama 2 7B | 32 | 4096 | 32 | meta-llama/Llama-2-7b-hf |
| Llama 2 13B | 40 | 5120 | 40 | meta-llama/Llama-2-13b-hf | |
| Llama 3 8B | 32 | 4096 | 32 | meta-llama/Meta-Llama-3-8B | |
| Qwen | Qwen2.5 7B | 28 | 3584 | 28 | Qwen/Qwen2.5-7B |
| Qwen2.5 14B | 48 | 5120 | 40 | Qwen/Qwen2.5-14B | |
| Qwen3 8B | 36 | 4096 | 32 | Qwen/Qwen3-8B |
Table 10: List of open source models evaluated in Appendix C . We observe consistent massive activations and attention sinks phenomena across the following diverse model families and sizes.
Figure 7: Top-3 coordinate magnitudes after (top panels) and before (bottom panels) the residual branch for 12 open source models. All models exhibit the characteristic step-up and step-down behavior driven by one or few early blocks and then the late blocks.
Figure 8: Frobenius norms for the quadratic forms of Llama models. Spike channels align with matrices that exhibit substantially larger norms than typical channels. These high-norm coordinates appear exclusively in step-up and step-down blocks.