cs.LGNov 2, 2026

Spectral Degeneration of Self-Attention under Isotropic Score Geometry

Authors: Mengda Li, Jianfeng Yao

Organizations: The Chinese University of Hong Kong, Shenzhen

Abstract

We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix. Motivated by the rotational isotropy of query and key representations, our model replaces the random token-space orientation of the score matrix by its canonical singular-value representative with a logarithmic spike--bulk separation. Empirical diagnostics on pretrained vision and language transformers motivate this model: token-space singular matrices have nearly uniform eigen-angle distributions, and score spectra are dominated by a small number of effective directions. We then prove that the row-wise softmax map transforms this structure into a degenerate attention matrix: dominant modes are preserved, whereas the bulk spectral distributions concentrate at zero. Stable rank remains bounded for bidirectional attention but grows logarithmically for causal attention. Within the canonical model, these results explain how attention can preserve a small number of dominant directions while compressing the remaining bulk modes.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention

    Aug 8, 2026Kasun Dewage, Marianna Pensky, Suranadi De Silva +1Transformer InterpretabilityTransformer Attention

  2. SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers

    Oct 26, 2026Debopam Sanyal, Hongjie Chen, Riya Elizabeth Kuriyan +5Transformer AttentionAttention Head Analysis

  3. Prescriptive SVD-Inspired Attention via Spectral Energy Retention

    Sep 21, 2026Vasileios Arampatzakis, Vasileios Sevetlidis, George PavlidisTransformer InterpretabilityLow-Rank Attention