Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

Yihong Chen; Zhouchen Lin; Quanming Yao

cs.LGMar 18, 2026

Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

Authors: Yihong Chen, Zhouchen Lin, Quanming Yao

Organizations: Tsinghua University · Peking University

Abstract

Attention sinks and massive activations are recurring and closely related phenomena in Transformer models. Existing explanations have largely focused on the forward pass, yet in pre-norm Transformers, large residual-stream norms play only an indirect forward role because sublayers operate on normalized inputs. We study this relationship from the perspective of backpropagation. Empirically and theoretically, we show that under causal masking, attention sinks can induce pronounced gradient concentration, which we term gradient sinks. Since the RMSNorm Jacobian attenuates gradients roughly in inverse proportion to input norm, massive activations can be understood as adaptive regulators of this localized gradient pressure during training. This interpretation predicts that attenuating sink-induced gradients should weaken massive activations. We test this prediction with V-scale, a modification that adjusts backpropagated gradients on the value path. In V-scale models, attention sinks are preserved, whereas massive activations are suppressed. These results identify gradient sinks as a backward-pass counterpart of attention sinks, and massive activations as an adaptive RMSNorm-mediated response that attenuates the resulting localized training pressure. Our code is available at https://anonymous.4open.science/r/GradientSinkCode-B309.

Figures & tables

Panel of Figure 1 — Figure 1: Gradient sinks across models. Each panel plots the token-wise distribution of $\log_{10}\|\nabla_{\widetilde{h}_{t}^{\ell}}\mathcal{L}\|_{2}$ at the post-RMSNorm attention input. Token 0 is highlighted. The reported median and interquartile range (IQR) of $R_{GS}^{\ell}(0)$ summarize how much larger the gradient on token 0 is than the mean over positions 1–31. Across both scratch-trained baselines and pretrained models, the first token consistently exhibits the strongest gradient concentration. Models include 0.1B and 1B baselines together with Qwen2.5-7B, Qwen3-8B, Llama-3.1-8B, and GLM-4-9B.

Figure 2: Token-wise gradient norms of query, key, and value across training checkpoints for the 1B model trained from scratch. Statistics are averaged over layers. Key and especially value gradients exhibit a pronounced spike at token 0, while query gradients remain comparatively flat.

Panel of Figure 3 — Figure 3: Scatter plots relating local gradient reshaping to input activation norms of the attention block. Top row: 0.1B model; bottom row: 1B model. From left to right, the panels show $\mathrm{Bloat}^{\mathrm{attn}}$ , $\mathrm{Change}^{\mathrm{attn}}$ , and $\mathrm{Compress}^{\mathrm{attn}}$ against $\|h_{\mathrm{in}}\|$ on a $\log_{10}$ scale. Each point is colored by token group (token 0, token 1, tokens 2–3, tokens 4–7, and the rest early tokens 8–15). Large-activation points from token 0 populate the high-bloat regime, yet their residual-stream gradient norms change only mildly for most layers. The $\mathrm{Compress}$ ratio indicates stronger attenuation at massive activation sites.

Figure 4: Attention weights as two-way routers.

Figure 5: Schematic of V-scale inside a pre-norm Transformer block.

Panel of Figure 6 — Figure 6: Gradient sinks in the 0.3B baseline and V-scale models. Each panel plots the token-wise distribution of $\log_{10}\|\nabla_{\widetilde{h}_{t}^{\ell}}\mathcal{L}\|_{2}$ at the post-RMSNorm attention input. Token 0 is highlighted. V-scale still leaves a visible gradient sink but reduces the concentration relative to the baseline.

Panel of Figure 7 — Figure 7: Forward phenomena in baseline and V-scale models. Top row: 0.3B models; bottom row: 1B models. From left to right: thresholded sink rate, residual-stream output norm, MLP output norm, and attention output norm. We compare the first token with the mean over the rest early tokens (positions 1–15). Across both scales, V-scale largely preserves attention sinks while reducing the activation norms of token 0, with the larger reduction appearing in MLP outputs.

Model	Wiki	Lambada		ARC-C	BoolQ	COPA	HellaSwag	OBQA	PIQA	WinoGrande
Model	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$
Baseline	22.84	15.27	43.28	24.91	51.38	69.00	51.59	31.40	72.52	54.22
V-scale	22.83	15.91	43.37	27.05	56.27	70.00	51.29	32.80	71.49	53.28

Table 1: Representative standard downstream evaluation of 1B baseline and V-scale models under bfloat16 . Lower perplexity (ppl) is better, while higher accuracy (acc) is better.

	Native				YaRN-extension
Quantization	1024		2048		1024		2048		4096		8192
Quantization	B	V	B	V	B	V	B	V	B	V	B	V
bfloat16	67.96	73.88	65.04	71.64	57.76	59.28	53.88	59.72	48.00	56.16	48.32	62.00
SmoothQuant	64.44	71.64	61.64	69.60	52.44	57.92	49.64	57.92	45.76	52.88	48.96	57.64
BNB	66.16	74.04	61.60	70.56	54.76	58.00	46.28	58.40	42.72	55.88	47.56	62.60
GPTQ	66.20	67.88	63.00	65.56	56.28	56.68	50.24	57.08	45.20	51.16	46.72	58.96

Table 2: Main multi-key Needle-in-a-Haystack result for 1B models. Entries are mean retrieval accuracy percentages over

5

random seeds; higher is better. We directly compare the baseline (B) and V-scale (V) models under native-context evaluation and YaRN extension with a factor of

4

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Item	0.1B runs	0.3B runs	1B runs
Model
total parameters	$\sim$ 101M	$\sim$ 304M	$\sim$ 1.04B
vocab size	50257	50257	50257
hidden layers	16	17	18
hidden size	512	1024	2048
attention heads	8	8	16

Appendix

Table 3: Model and training configuration.

Panel of Figure 8 — Figure 8: Token-wise QKV gradient norms across training checkpoints for smaller scratch-trained models, averaged over layers. Top row: 0.1B model; bottom row: 0.3B model. From left to right: query, key, and value gradient norms as functions of token position. Both models reproduce the pathway asymmetry observed in the 1B model: key and especially value gradients exhibit a strong spike at token 0, while query gradients remain comparatively flat.

Figure 9: Token-wise gradient norms of QKV for pretrained LLMs. Each row corresponds to one model. From left to right: gradient norms of query, key, and value as functions of token position, averaged over layers. Across models, key and especially value gradients consistently exhibit a pronounced spike at token 0.

Panel of Figure 10 — Figure 10: Scatter plots relating gradient reshaping to input activation norms of the MLP block. Top row: 0.1B model; bottom row: 1B model. From left to right: $\log_{10}\mathrm{Bloat}^{\mathrm{mlp}}$ vs. $\log_{10}\|h_{\mathrm{half}}\|$ , $\log_{10}\mathrm{Change}^{\mathrm{mlp}}$ vs. $\log_{10}\|h_{\mathrm{half}}\|$ , and $\log_{10}\mathrm{Compress}^{\mathrm{mlp}}$ vs. $\log_{10}\|h_{\mathrm{half}}\|$ . Each point is colored by token group (token 0, token 1, tokens 2–3, tokens 4–7, and the rest early tokens 8–15). Compared with the attention branch, token 0 is less dominant in MLP bloat, while compression again aligns tightly with activation scale and the $\mathrm{Change}$ ratio remains close to one.

Figure 11: Supplementary gradient-reshaping measurements for the 0.3B baseline and pretrained models. From left to right, each row shows attention-branch

\mathrm{Bloat}

\mathrm{Change}

, and

\mathrm{Compress}

, followed by MLP-branch

\mathrm{Bloat}

\mathrm{Change}

, and

\mathrm{Compress}

Figure 12: Value-state norms in baseline models, averaged over heads and shown across training checkpoints. Left: 0.3B baseline; right: 1B baseline. The solid curve is token 0, the dashed curve is the mean over other early tokens, and the dotted curve is their maximum, where “early” refers to positions 1–15. Token 0 usually has a much smaller value-state norm, placing it in the regime where V-scale can induce stronger value-path gradient attenuation.

Panel of Figure 13 — Figure 13: Final learned V-scale parameters $\theta_{\ell,h}$ . Left: 0.3B V-scale model; right: 1B V-scale model. Each heatmap indexes layers by rows and attention heads by columns, with the color scale centered at the initialization value $\theta=0$ . Positive values correspond to larger $C_{\ell,h}$ and stronger attenuation for fixed small-norm value states, while negative values correspond to smaller $C_{\ell,h}$ .

Panel of Figure 14 — Figure 14: Gradient sinks in additional V-scale models. Left: 0.1B V-scale model; right: 1B V-scale model. Each panel plots the token-wise distribution of $\log_{10}\|\nabla_{\widetilde{h}_{t}^{\ell}}\mathcal{L}\|_{2}$ at the post-RMSNorm attention input, with token 0 highlighted. The sink token remains visible but is more moderate.

Panel A: language modeling and core reasoning
Model	Quantization	Wiki	Lambada		ARC-C	ARC-E	BoolQ	COPA
Model	Quantization	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$
Baseline	bfloat16	22.84	15.27	43.28	24.91	47.98	51.38	69.00
V-scale	bfloat16	22.83	15.91	43.37	27.05	47.47	56.27	70.00
Baseline	SmoothQuant	23.17	16.61	43.02	25.26	47.47	50.92	67.00
V-scale	SmoothQuant	23.01	16.58	43.14	26.11	46.76	55.57	69.00

Appendix

Table 4: Standard downstream evaluation of 1B baseline and V-scale models with unquantized bfloat16 and post-training quantization. bfloat16 denotes unquantized evaluation; SmoothQuant is W8A8 quantization, and BNB, GPTQ, and AWQ are W4A16 quantization methods. Wiki ppl and Lambada ppl are perplexities, where lower is better. Lambada acc, ARC-C/E (AI2 Reasoning Challenge and Easy splits), BoolQ, COPA, HellaSwag, OBQA (OpenBookQA), PIQA, SCIQ, and WinoGrande are zero-shot accuracies in percent, where higher is better. V-scale keeps perplexity nearly unchanged and remains broadly comparable to the baseline across reasoning tasks, with gains on ARC-C, BoolQ, COPA, and OBQA and small drops on several other tasks.

1B Models
Model	Quantization	1024	2048	4096	8192
Baseline	bfloat16	$99.96_{\pm 0.09}$	$98.00_{\pm 0.69}$
V-scale	bfloat16	$100.00_{\pm 0.00}$	$97.80_{\pm 0.42}$
Baseline	SmoothQuant	$99.96_{\pm 0.09}$	$95.92_{\pm 2.09}$
V-scale	SmoothQuant	$100.00_{\pm 0.00}$	$95.64_{\pm 0.99}$
Baseline	BNB	$99.88_{\pm 0.18}$	$98.36_{\pm 0.86}$

Appendix

Table 5: Needle-in-a-Haystack retrieval accuracy on the niah_single_2 single-needle template for 1B models. Columns denote context length in tokens, and entries are mean accuracy percentages with standard deviations over

5

random seeds; higher is better. The first block evaluates native-context 1B models, for which only 1024 and 2048 tokens are tested. Blank cells therefore indicate untested longer contexts rather than failed runs. The second block evaluates the same models after YaRN extension with a factor of

4

, enabling 4096- and 8192-token tests. This template is nearly saturated at shorter lengths, but V-scale gives consistently higher accuracy at 8192 tokens after YaRN extension.

1B Models
Model	Quantization	1024	2048	4096	8192
Baseline	bfloat16	$99.68_{\pm 0.11}$	$85.48_{\pm 1.24}$
V-scale	bfloat16	$98.60_{\pm 0.79}$	$95.60_{\pm 0.95}$
Baseline	SmoothQuant	$99.00_{\pm 0.28}$	$79.28_{\pm 0.93}$
V-scale	SmoothQuant	$96.00_{\pm 0.84}$	$88.48_{\pm 0.46}$
Baseline	BNB	$99.32_{\pm 0.18}$	$79.80_{\pm 1.29}$

Appendix

Table 6: Needle-in-a-Haystack retrieval accuracy on the harder niah_single_3 single-needle template for 1B models. Columns denote context length in tokens, and entries are mean accuracy percentages with standard deviations over

5

random seeds; higher is better. The first block reports native-context evaluation up to 2048 tokens, while the second block uses YaRN extension with a factor of

4

to evaluate up to 8192 tokens. On native 2048-token evaluation, V-scale improves over the baseline for bfloat16 , SmoothQuant, BNB, and GPTQ, with AWQ as the exception. After YaRN extension, the result is mixed, with gains in several 2048-token and BNB settings but degradation under some 4096- and 8192-token quantized settings.

1B Models
Model	Quantization	1024	2048	4096	8192
Baseline	bfloat16	$67.96_{\pm 1.34}$	$65.04_{\pm 1.09}$
V-scale	bfloat16	$73.88_{\pm 1.06}$	$71.64_{\pm 3.46}$
Baseline	SmoothQuant	$64.44_{\pm 1.43}$	$61.64_{\pm 2.91}$
V-scale	SmoothQuant	$71.64_{\pm 1.44}$	$69.60_{\pm 2.19}$
Baseline	BNB	$66.16_{\pm 1.68}$	$61.60_{\pm 1.83}$

Appendix

Table 7: Needle-in-a-Haystack retrieval accuracy on the niah_multikey_1 multi-key template for 1B models. This setting requires retrieving information when multiple key-value associations are present, making it more sensitive to long-context attention allocation than the saturated single-needle cases. Columns denote context length in tokens, and entries are mean accuracy percentages with standard deviations over

5

random seeds; higher is better. Native-context models are evaluated at 1024 and 2048 tokens, and YaRN-extended models with a factor of

4

are evaluated up to 8192 tokens. V-scale improves every listed precision/quantization and context-length combination, with especially large gains at 8192 tokens after YaRN extension.

Explore similar work

Mar 5, 2026

The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks

We study two recurring phenomena in Transformer language models: massive activations, in which a small number of tokens exhibit extreme outliers in a few channels, and attention sinks, in which certain tokens attract disproportionate attention mass regardless of semantic relevance. Prior work observes that these phenomena frequently co-occur and often involve the same tokens, but their functional roles and causal relationship remain unclear. Through systematic experiments, we show that the co-occurrence is largely an architectural artifact of modern Transformer design, and that the two phenomena serve related but distinct functions. Massive activations operate globally: they induce near-constant hidden representations that persist across layers, effectively functioning as implicit parameters of the model. Attention sinks operate locally: they modulate attention outputs across heads and bias individual heads toward short-range dependencies. We identify the pre-norm configuration as the key choice that enables the co-occurrence, and show that ablating it causes the two phenomena to decouple.

Shangwen Sun, Alfredo Canziani, Yann LeCun +1

New York University.

Attention Sinks Transformer Interpretability

Ask PDF

Sep 30, 2026

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transformers, but how their scale is regulated during training remains incompletely understood. Combining training-trajectory analyses and controlled interventions, we trace their emergence, growth, and consolidation. Sink-carrying channels vary across random seeds but stabilize early within each run. Over longer training, surrounding channels erode and the sink concentrates onto a few redundant carriers. Across ablations, gradient attenuation follows the sink token's collective root-mean-square magnitude rather than any single channel, making collective scale central to understanding their effects. Our central result is that weight decay causally controls the turnover of global activation scale. In controlled continuations, removing decay near the peak allows this scale to keep rising, whereas retaining it produces decline even at constant learning rate. We develop a balance model for the rise and peak of massive-activation magnitude, in which AdamW-preconditioned growth opposes weight decay. Sweeping the decay coefficient

λ

shifts peak timing approximately log-linearly and yields peak magnitudes scaling approximately as

λ^{-1/2}

, consistent with this balance. Optimizer measurements further show that preconditioning sustains the large-channel cohort against decay even when raw maintaining forces are too small to do so. Together, these findings connect the observed life cycle to scale-regulating training dynamics and establish weight decay as a training-time lever on activation magnitude.

S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios Saravanos

Aimpoint Digital Labs, Atlanta, GA, USA · New York University, New York, NY, USA

Weight Decay Attention Sinks Neural Network Training Dynamics

Ask PDF

Mar 12, 2026

Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks

Transformers often display an attention sink: probability mass concentrates on a fixed, content-agnostic position. Are sinks a byproduct of the optimization/training regime? Or are they sometimes functionally necessary in softmax Transformers? We prove that, in some settings, it is the latter: computing a simple trigger-conditional behavior necessarily induces a sink in softmax self-attention models. Our results formalize a familiar intuition: normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state (e.g., when the model needs to ignore the input). We instantiate this with a concrete task: when a designated trigger token appears, the model must return the average of all preceding token representations, and otherwise output zero, a task which mirrors the functionality of attention heads in the wild (Barbero et al., 2025; Guo et al., 2024). We also prove that non-normalized ReLU attention can solve the same task without any sink, confirming that the normalization constraint is the fundamental driver of sink behavior. Experiments validate our predictions and demonstrate they extend beyond the theoretically analyzed setting: softmax models develop strong sinks while ReLU attention eliminates them in both single-head and multi-head variants.

Yuval Ran-Milo

Tel Aviv University

Attention Sinks Attention Mechanisms Softmax Attention

Ask PDF