cs.LGNov 10, 2026

Massive Activations Confound Sparse Autoencoders

Authors: Nathan Delisle, Chenhao Tan

Organizations: University of Chicago

Abstract

Sparse autoencoders (SAEs) are commonly evaluated by variance explained and ablation damage. We show how massive activations undermine both evaluations, and propose a correction. Massive activations cause up to 99.9% of activation variance to lie along the first principal component (PC1). Because the PC1 component has no unique sparse decomposition, each SAE run rebuilds it from a different set of features. These features dominate importance rankings: an average of 47 of the 50 most important features are unmatched in a separate run, compared with about 27 of a random 50. A dictionary can also report 99.96% of variance explained while leaving 33% of the variance outside PC1 unexplained. We bypass PC1 by training the SAE on activations with PC1 removed, then adding PC1 to the reconstruction. In affected models, the top-50 feature sharing rate rises from 4-39.5% to 73-99.5%. At the same TopK sparsity, the resulting SAE degrades the model up to 44% less and reduces content reconstruction error by up to 33.9%.

Figures & tables

Appendix figures & tables63 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Decompose Sparsely Where You Should, Absorb Densely Where You Should No

    Jun 12, 2026Ruixuan Deng, Zehao Jin, Zekun Wang +1Transformer InterpretabilityRepresentation Learning

  2. Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

    Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation SparsityLLM Interpretability

  3. On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders

    May 29, 2026Elana Simon, Etowah Adams, James ZouActivation OutliersSparse Autoencoders