Sparse Autoencoder Attribution Graphs Are Decided at the Margin
Organizations: Columbia University
Abstract
Attribution graphs predict sparse autoencoder (SAE) ablations to first order on the active features, but a TopK SAE passes only its k largest preactivations, so small changes often swap active features. On five public TopK SAEs, zeroing one feature changes the active set of the next layer's SAE in 74% to 89% of cases. Our margin graph adds the first order change to every downstream preactivation, then selects the k largest again. On Pythia-70M the margin graph cuts the median error of predicted activations, relative to their exact change, from the attribution graph's 0.739 to 0.121, 84% lower. Exact recomputation of 48 interventions runs 3.5 times faster than the margin graph on our GPU. Patched into Pythia-70M with the clean reconstruction error, the margin graph's activations predict the change in the subject verb agreement logit difference with 1.95 times the attribution graph's mean error, on interventions above a fixed floor. Between seeds of our SAEs, which share one auxiliary loss coefficient, sparse feature maps recover 85% of a graph's squared norm on active features but 47% with each graph masked to its own, 38 percentage points less. Unmasking the inactive tenth nearest the TopK threshold recovers 73% of the masking loss, against 11% for a random tenth, so the margin to that threshold decides how ablations change activations and what graphs carry between seeds.
Figures & tables
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| criterion | quantity measured | met when | result | met |
|---|---|---|---|---|
| rate of active set changes at the zero ablation, each public TopK pair | under 0.1 | 0.74 to 0.89 | no | |
| share of squared norm on the entering and leaving coordinates, each public TopK pair | under 0.1 | 0.73 to 0.84 | no | |
| the same on the ReLU pair | under 0.1 | 0.05 | yes | |
| the ReLU pair’s share of squared norm | over 0.5 | 0.05 | no | |
| the margin graph’s error, each public TopK pair | not under 0.75 of the standard graph’s, which is 0.423, 0.186, 0.498, 0.462 and 0.502 on the pairs in the order of Table | 0.013 to 0.033 | no | |
| the margin graph’s error, our SAEs | not under 0.75 of the standard graph’s, which is 0.590 | 0.017 | no |
| check or control | bound | result |
| the patched clean code reproduces the model’s metric on every development prompt | 0.001 | |
| the exact code, patched with the intervened pass’s reconstruction residual, reproduces the intervention’s change | ||
| predicting no change errs by the absolute change | ||
| a scaling factor of one changes nothing | ||
| the errors of Section 4 , reproduced on 20 interventions | ||
| the bootstrap on identical errors returns an interval of zero width | zero width | zero width |
| scaling factor | interventions | standard | margin | no change | random control |
|---|---|---|---|---|---|
| zero ablation | 2,389 | 0.195 | 0.350 | 0.308 | 0.2398 |
| 0.5 | 1,938 | 0.113 | 0.239 | 0.190 | 0.1465 |
| 0.9 | 566 | 0.054 | 0.163 | 0.103 | 0.0832 |
Sparse Autoencoders
Sparse autoencoders decompose neural network activations into sparse features to study, interpret, and control learned representations.
See all 245 papers