cs.LGNov 16, 2026

Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated Attention

Authors: Seojin Kim, Yehjin Shin, Noseong Park

Organizations: KAIST

Abstract

Transformer attention exhibits attention sinks, where a small subset of source tokens absorbs disproportionate attention mass across layers. Recent work shows that post-SDPA output gating strongly reduces attention-sink behavior, while separate studies link attention sinks to gradient sinks and massive activations through backward training dynamics. Yet how the same gate affects forward sink structure, backward gradient concentration, and training-time representation signatures remains unclear. We address this gap by formulating attention sinks as sink-induced low-rank modes in token-space covariance, with directions aligned to high-mass attention columns. Under this view, post-SDPA gating acts as a mode-dependent contraction: it attenuates sink-induced spectral energy in the forward pass and locally reduces value-path gradients routed through sink columns in the backward pass. Empirically, random-matrix diagnostics show that gating reduces outlier mass, weakens sink-subspace alignment, and increases effective rank, while backward analyses show active-gate attenuation of first-token value gradients. Training-time diagnostics further show that gating suppresses the co-emergence of attention concentration, value-gradient amplification, and representation dominance.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions

    Jun 6, 2026Lukas Fesser*, Mozes Jacobs*, Thomas Fel* +2Transformer InterpretabilityVision Transformer

  2. Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

    Mar 18, 2026Yihong Chen, Zhouchen Lin, Quanming YaoMassive ActivationsAttention Sinks