cs.LGOct 31, 2026

Sparse Autoencoder Attribution Graphs Are Decided at the Margin

Authors: Karl Li

Organizations: Columbia University

Abstract

Attribution graphs predict sparse autoencoder (SAE) ablations to first order on the active features, but a TopK SAE passes only its k largest preactivations, so small changes often swap active features. On five public TopK SAEs, zeroing one feature changes the active set of the next layer's SAE in 74% to 89% of cases. Our margin graph adds the first order change to every downstream preactivation, then selects the k largest again. On Pythia-70M the margin graph cuts the median error of predicted activations, relative to their exact change, from the attribution graph's 0.739 to 0.121, 84% lower. Exact recomputation of 48 interventions runs 3.5 times faster than the margin graph on our GPU. Patched into Pythia-70M with the clean reconstruction error, the margin graph's activations predict the change in the subject verb agreement logit difference with 1.95 times the attribution graph's mean error, on interventions above a fixed floor. Between seeds of our SAEs, which share one auxiliary loss coefficient, sparse feature maps recover 85% of a graph's squared norm on active features but 47% with each graph masked to its own, 38 percentage points less. Unmasking the inactive tenth nearest the TopK threshold recovers 73% of the masking loss, against 11% for a random tenth, so the margin to that threshold decides how ablations change activations and what graphs carry between seeds.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features

    May 7, 2026Ruben Fernandez-Boullon, Pablo Magariños-Docampo, Javier Perez-RoblesTransformer InterpretabilitySparse Autoencoders

  2. What Do Sparse Autoencoder Feature Discovery Scores Identify? From Activations to Interventions

    Oct 11, 2026Yifeng Guo, Yanran Li, Xiang ChenSparse AutoencodersMechanistic Interpretability

  3. Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects

    Jul 22, 2026Seonglae Cho, Zekun Wu, Kleyton Da Costa +3LLM InterpretabilitySparse Autoencoders