Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention
Organizations: KAIST
Abstract
Transformer attention exhibits attention sinks, where a small subset of source tokens absorbs disproportionate attention mass across layers. Recent work shows that post-SDPA output gating strongly reduces attention-sink behavior, while separate studies link attention sinks to gradient sinks and massive activations through backward training dynamics. Yet how the same gate affects forward sink structure, backward gradient concentration, and training-time representation signatures remains unclear. We address this gap by formulating attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns. Under this view, post-SDPA gating acts as a mode-dependent contraction: it attenuates sink-induced spectral energy in the forward pass and locally reduces value-path gradients routed through sink columns in the backward pass. Empirically, random-matrix diagnostics show that gating reduces outlier mass, weakens sink-subspace alignment, and increases effective rank, while backward analyses show active-gate attenuation of first-token value gradients. Training-time diagnostics further show that gating suppresses the co-emergence of attention concentration, value-gradient amplification, and representation dominance.
Figures & tables
| Metric | High align. | Low align. |
|---|---|---|
| Raw energy ( ) | 0.047 | 0.044 |
| Std. covariance ( ) | 0.740 | 0.945 |
| Scalar gate ( ) | 0.957 | 0.990 |
| Metric | Baseline | Gated |
|---|---|---|
| AS ( ) | 0.084 | 0.013 |
| GS ( ) | 25.40 | 5.67 |
| MA ( ) | 320.0 | 67.4 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Retention metric | High alignment | Low alignment | Difference |
|---|---|---|---|
| Raw activation energy ( ) | 0.047 | 0.044 | |
| Standardized covariance ( ) | 0.740 | 0.945 | |
| Scalar gate ( ) | 0.957 | 0.990 |
| Representation | Original | Permuted | Layers with increase |
|---|---|---|---|
| Baseline | 0.062 | 0.196 | |
| Gated (pre-gate) | 0.220 | 0.510 |
| Model | Cross-token corr. | Norm. feature rank | Align. ( ) | Align. ( ) | Random align. |
|---|---|---|---|---|---|
| Baseline | 4.9 | 0.44 | 0.97 | 0.64 | |
| Gated | 14.4 | 0.33 | 0.79 | 0.32 |
| Metric | Qwen-1B | Pythia-410M |
|---|---|---|
| Align. (pre post) | ||
| Pre/random | ||
| Align. drop | ||
| drop (proj.) |
| Model | Wiki. | LMB. | LMB. | PIQA | Hella | Wino | ARC-e | ARC-c | OBQA | BoolQ | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| ppl | ppl | acc | acc | acc_n | acc | acc | acc_n | acc_n | acc | ||
| Baseline | 29.53 | 55.21 | 29.94 | 65.83 | 36.76 | 51.70 | 56.02 | 26.54 | 30.80 | 62.11 | 44.96 |
| Gated | 27.99 | 40.52 | 32.47 | 66.43 | 37.93 | 52.49 | 57.79 | 27.65 | 32.00 | 61.71 | 46.06 |
| Variant | OutMass | |||||
|---|---|---|---|---|---|---|
| Baseline | 0.084 | 0.125 | 18.42 | 0.679 | 89.14 | 25.40 |
| Gated | 0.013 | 0.013 | 12.49 | 0.545 | 153.20 | 5.67 |
| Model | Diagnostic | Baseline | Gated | Baseline/gated |
|---|---|---|---|---|
| Pythia-410M | (MA) | 379.2 | 108.4 | |
| (MA) | 320.0 | 67.4 | ||
| Qwen-1B | (MA) | 11,854.5 | 146.3 | |
| (MA) | 11,839.5 | 87.2 |
| Model | OutMass | Eff. rank | Sink alignment | |||||
|---|---|---|---|---|---|---|---|---|
| Baseline 2.8B | 0.147 | 0.156 | 28.24 | 27.04 | 0.684 | 24.21 | 111.2 | 0.174 |
| Gated 2.8B | 0.011 | 0.010 | 4.29 | 3.61 | 0.566 | 15.54 | 185.0 | 0.156 |
| 2.8B ratio | 0.077 | 0.067 | 0.152 | 0.133 | 0.827 | 0.642 | 1.664 | 0.895 |
| 410M ratio | 0.129 | 0.101 | 0.195 | 0.160 | 0.796 | 0.658 | 1.717 | 0.657 |
| Train/eval | Model | Sink columns | Norm. eff. rank | Alignment/random | ||||
|---|---|---|---|---|---|---|---|---|
| 4K/4K | Baseline | 0.025 | 0.071 | 2.5 | 44.1 | 0.159 | 180.4 | 58.4 |
| Gated | 0.002 | 0.003 | 6.5 | 29.1 | 0.288 | 58.5 | 6.2 | |
| 8K/4K | Baseline | 0.015 | 0.058 | 3.3 | 53.0 | 0.118 | 203.8 | 54.7 |
| Gated | 0.002 | 0.003 | 5.7 | 35.4 | 0.237 | 50.7 | 7.5 | |
| 8K/8K | Baseline | 0.008 | 0.058 | 3.1 | 59.0 | 0.134 | 349.9 | — |
| Gated | 0.001 | 0.002 | 5.8 | 42.2 | 0.262 | 68.7 | — |
| Convention | OutMass | Eff. rank | Sink alignment | |||||
|---|---|---|---|---|---|---|---|---|
| Plain | 0.135 | 0.108 | 0.206 | 0.175 | 0.797 | 0.649 | 1.723 | 0.660 |
| EOS-separated | 0.183 | 0.226 | 0.253 | 0.320 | 0.786 | 0.644 | 1.735 | 0.385 |
| Input condition | Model | Target attn. mass | Target spectral alignment | Target gradient amplification | |
|---|---|---|---|---|---|
| No EOS; position | Baseline | 0.583 | 0.130 | 0.204 | 31.39 |
| Gated | 0.174 | 0.015 | 0.031 | 6.38 | |
| EOS at | Baseline | 0.930 | 0.200 | 0.229 | 40.97 |
| Gated | 0.706 | 0.037 | 0.056 | 10.77 | |
| EOS at | Baseline | 0.878 | 0.181 | 0.042 | 28.64 |
| Gated | 0.753 | 0.041 | 0.023 | 10.20 |
| Operator Swap (Qwen-1B, layers 8–28) | Trained models (340M, layers 8–24) | ||||||
|---|---|---|---|---|---|---|---|
| Metric | Softmax | Softpick | Effect | Metric | Vanilla | Softpick | Effect |
| 0.0767 | 0.0754 | 0.0786 | 0.0032 | ||||
| 19.86 | 19.69 | 15.39 | 13.74 | ||||
| - | - | - | - | 0.629 | 0.199 | ||
Attention Sinks
Attention sinks are tokens that attract disproportionate Transformer attention, shaping information flow, long-context behavior, and model efficiency.
See all 57 papers