We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.
Figures & tables
Figure 1: Top-3 channel magnitudes across depth in Llama 2 7B and Qwen3 8B (post-residuals vs. block outputs). In both models, early blocks inject massive activations that persist through most of the network before being neutralized by late blocks.
Model
# Blocks
Step-Up
Step-Down
Llama 2 7B
64
4
62
Llama 2 13B
80
8
78, 79
Llama 3 8B
64
4
64
Qwen2.5 7B
56
8, 10
54, 55
Qwen2.5 14B
96
10
90, 92, 94, 95
Qwen3 8B
72
14
70, 72
Table 1: Step-up and step-down block indices across models. Step-up blocks appear near the beginning of the network and step-down blocks near the end, confining massive activations to intermediate layers. Odd and even indices denote attention and feed-forward blocks, respectively.
Figure 2: Input-output characteristics of SiLU in step-up and step-down blocks of Llama 2 7B. Based on 1024 randomly sampled sentences from C4 dataset ( Raffel et al., 2020 ) , we plot the cosine similarity and norm ratio for each token. Points are colored by the maximum magnitude of the block output. For spike tokens (red points), both direction and norm remain largely unchanged, indicating that the SiLU gate operates in a near-identity regime.
Figure 3: Frobenius norms ∥Uk∥F for the quadratic forms in Llama 2 7B. Spike channels align with Uk matrices that have substantially larger norms than typical channels. These high-norm coordinates appear exclusively in step-up and step-down blocks.
Figure 4: Eigenvalue spectra of Sk for spike vs. non-spike channels in Llama 2 7B. Spike channels exhibit a single dominant eigenvalue λ⋆ that is orders of magnitude larger than the remainder of the spectrum; non-spike channels show no such outlier.
Model
# Vocab
# Spike Token
Ratio
Llama 2 7B
32,000
31,887
99.65%
Llama 2 13B
32,000
31,889
99.65%
Llama 3 8B
128,256
127,956
99.77%
Qwen 2.5 7B
152,064
149,587
98.40%
Qwen 2.5 14B
152,064
149,645
98.40%
Qwen 3 8B
151,936
151,830
99.93%
Table 2: Ubiquity of initial spikes across diverse LLMs. For nearly all evaluated models, positional occupancy at the initial position induces massive activations in intermediate layers, independent of the token’s semantic identity.
Figure 5: Cosine similarity among spike tokens before and after step-up block in Llama 2 7B. Pre-step-up representations vary across spike tokens, but post-step-up representations collapse to nearly identical directions, empirically validating the near-constant approximation.
Figure 6: t-SNE visualization of query and key vectors for a representative sink head (left) and non-sink head (right). In the sink head, sink keys k(s) lie closer to q(n) than non-sink keys k(n) , creating large logit gaps. In the non-sink head, k(s) and k(n) are approximately equidistant from q(n) , preventing the formation of a privileged sink position.
Setup
Perplexity
Sink Ratio
Spike
Base Learning Rate
7.5\times10−5
11.8
18.6%
1447
1.5\times10−4
10.7
31.8%
2251
3.0\times10−4
10.1
46.0%
3818
6.0\times10−4
10.0
51.5%
3773
1.2\times10−3
10.2
39.2%
2723
Table 3: Optimization Hyperparameters ablations. We observe that the sink ratio could serve as a proxy for optimization health. Conversely, the magnitude of massive activations varies largely independently of both perplexity and the sink ratio. Highlighted rows denote the baseline result.
Setup
Perplexity
Sink Ratio
Spike
Feed-Forward Block
GeLU
10.1
69.3%
3369
Linear
12.5
58.9%
688
Attention
10.8
73.9%
637
SwiGLU
10.1
46.0%
3818
Table 4: Feed-forward block design ablations. Massive activations and attention sinks emerge across all evaluated designs, including attention-only and single linear configurations. Notably, GeLU and SwiGLU architectures yield significantly higher spike magnitudes by acting as efficient amplifiers. In contrast, linear and attention-only blocks exhibit much lower spikes as they require gradual accumulation across multiple layers. Highlighted rows denote the baseline result.
Setup
Perplexity
Sink Ratio
Spike
Normalization
Sandwich
9.8
44.7%
520
Sandwich (QK)
10.0
42.0%
92
DynamicTanh
10.0
61.0%
153
Pre-Norm
10.1
46.0%
3818
Table 5: Normalization configuration ablations. Applying post-block normalization (Sandwich) or element-wise transformations (DynamicTanh) successfully suppresses massive activations. Notably, the model still maintains a significant sink ratio through alternative strategies, demonstrating that sinks can exist independently of massive activations. Highlighted rows denote the baseline result.
Setup
Perplexity
Sink Ratio
Spike
Number of Heads
8
10.4
37.1%
1253
16
10.3
41.7%
1936
32
10.1
46.0%
3818
Head Dimension
8
11.3
4.1%
291
Table 6: Attention head settings ablations. Head dimension is the primary architectural driver of sink formation; larger dimensions provide the capacity for the attention subspace to isolate the sink keys. Concentrating capacity into fewer, larger heads intensifies sink behavior. Highlighted rows denote the baseline configuration.
Setup
Perplexity
Sink Ratio
Spike
Conditional Gating
Channel
10.0
4.5%
202
Head
10.1
6.4%
186
Single
10.2
31.2%
316
Unconditional Gating
Channel
10.1
42.2%
1922
Table 7: Gated attention ablations. Conditional gating—where the gate is a function of current representation—eliminates the need for attention sinks when applied per channel or per head. This suggests that sinks function as a “learned gate” to balance head contributions. Unconditional or static gates (positional or token-based) fail to suppress sinks, as they lack the dynamic, input-dependent routing necessary to substitute for sink behavior.
Setup
Perplexity
Sink Ratio
Spike
Context Length (min/max)
1/256
12.4
42.1%
5411
1/1024
10.6
46.3%
4442
1/4096
10.1
46.0%
3818
1024/4096
10.1
13.0%
38470
1024/5120
10.1
8.0%
42365
Table 8: Context-length ablations. Attention sinks are largely induced to facilitate short-context prediction. When the training distribution is restricted to long sequences, the sink ratio collapses, indicating that sinks are primarily utilized to support short-range dependence. Highlighted rows denote the baseline configuration.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Layers
32
Hidden size
4096
Attention heads
32
Head dimension
128
Intermediate size
11008
Vocabulary size
32000
Appendix
Table 9: Baseline training configurations.
Model Family
Model Name
Layers
Dimensions
Heads
Huggingface Model Id
Llama
Llama 2 7B
32
4096
32
meta-llama/Llama-2-7b-hf
Llama 2 13B
40
5120
40
meta-llama/Llama-2-13b-hf
Llama 3 8B
32
4096
32
meta-llama/Meta-Llama-3-8B
Qwen
Qwen2.5 7B
28
3584
28
Qwen/Qwen2.5-7B
Qwen2.5 14B
48
5120
40
Qwen/Qwen2.5-14B
Qwen3 8B
36
4096
32
Qwen/Qwen3-8B
Appendix
Table 10: List of open source models evaluated in Appendix C . We observe consistent massive activations and attention sinks phenomena across the following diverse model families and sizes.
Figure 7: Top-3 coordinate magnitudes after (top panels) and before (bottom panels) the residual branch for 12 open source models. All models exhibit the characteristic step-up and step-down behavior driven by one or few early blocks and then the late blocks.
Figure 8: Frobenius norms ∥Uk∥F for the quadratic forms of Llama models. Spike channels align with Uk matrices that exhibit substantially larger norms than typical channels. These high-norm coordinates appear exclusively in step-up and step-down blocks.
Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing explanations have largely focused on the forward pass, yet in pre-norm Transformers, large residual-stream norms play only an indirect forward role because sublayers operate on normalized inputs. We study this relationship from the perspective of backpropagation. Empirically and theoretically, we show that under causal masking, attention sinks can induce pronounced gradient concentration, which we term gradient sinks. Since the RMSNorm Jacobian attenuates gradients roughly in inverse proportion to input norm, massive activations can be understood as adaptive regulators of this localized gradient pressure during training. This interpretation predicts that attenuating sink-induced gradients should weaken massive activations. We test this prediction with V-scale, a modification that adjusts backpropagated gradients on the value path. In V-scale models, attention sinks are preserved, whereas massive activations are suppressed. These results identify gradient sinks as a backward-pass counterpart of attention sinks, and massive activations as an adaptive RMSNorm-mediated response that attenuates the resulting localized training pressure. Our code is available at https://anonymous.4open.science/r/GradientSinkCode-B309.
Transformers often display an attention sink: probability mass concentrates on a fixed, content-agnostic position. Are sinks a byproduct of the optimization/training regime? Or are they sometimes functionally necessary in softmax Transformers? We prove that, in some settings, it is the latter: computing a simple trigger-conditional behavior necessarily induces a sink in softmax self-attention models. Our results formalize a familiar intuition: normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state (e.g., when the model needs to ignore the input). We instantiate this with a concrete task: when a designated trigger token appears, the model must return the average of all preceding token representations, and otherwise output zero, a task which mirrors the functionality of attention heads in the wild (Barbero et al., 2025; Guo et al., 2024). We also prove that non-normalized ReLU attention can solve the same task without any sink, confirming that the normalization constraint is the fundamental driver of sink behavior. Experiments validate our predictions and demonstrate they extend beyond the theoretically analyzed setting: softmax models develop strong sinks while ReLU attention eliminates them in both single-head and multi-head variants.
When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: (i) adaptive nop, where a head suppresses its update by routing to a null token, and (ii) broadcast, where a sink aggregates and redistributes global information. Each mechanism leaves distinct traces (nop-sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs), which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers and concentrate in specialized heads. Causal interventions further connect these signatures to near-null suppression and shared residual contributions. We then use architectural interventions to show how these computations can be reorganized: gating eliminates detected nop-like sinks but increases broadcast-like sinks, registers relocate rather than remove sink computation, and our position-free global pathway provides an explicit route for shared communication that reduces the broadcast-like sinks induced by gating. On dense probes, combining gating with the global pathway gives the strongest results among the tested variants, despite retaining some broadcast-like sinks. Overall, we find that the same attention pattern can reflect two very different computations, and that effective intervention depends not only on identifying the computation, but also on providing architectural alternatives through which the model can reorganize it.
Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2
Kempner Institute Harvard University Cambridge, MA