Massive Activations Confound Sparse Autoencoders
Organizations: University of Chicago
Abstract
Sparse autoencoders (SAEs) are commonly evaluated by variance explained and ablation damage. We show how massive activations undermine both evaluations, and propose a correction. Massive activations cause up to 99.9% of activation variance to lie along the first principal component (PC1). Because the PC1 component has no unique sparse decomposition, each SAE run rebuilds it from a different set of features. These features dominate importance rankings: an average of 47 of the 50 most important features are unmatched in a separate run, compared with about 27 of a random 50. A dictionary can also report 99.96% of variance explained while leaving 33% of the variance outside PC1 unexplained. We bypass PC1 by training the SAE on activations with PC1 removed, then adding PC1 to the reconstruction. In affected models, the top-50 feature sharing rate rises from 4-39.5% to 73-99.5%. At the same TopK sparsity, the resulting SAE degrades the model up to 44% less and reduces content reconstruction error by up to 33.9%.
Figures & tables
| Model | TopK | BatchTopK | JumpReLU ∗ | Gated | ReLU+ | |
|---|---|---|---|---|---|---|
| Qwen-2.5-7B | .999 | 6.0 | 33.0 | 32.0 | 47.0 † | 97.0 |
| GPT-2 | .970 | 5.5 | 17.5 | 11.5 | 11.0 | 100.0 |
| Llama-3.1-8B | .969 | 56.0 | 62.0 | 61.0 | 99.0 | 97.0 |
| Pythia-1.4B | .964 | 10.0 | 14.0 | 12.0 | 86.0 | 98.5 |
| Pythia-6.9B | .957 | 10.0 | 18.0 | 26.0 | 15.0 ‡ | 97.0 |
| Pythia-160M | .934 | 9.0 | 9.5 | 14.5 | 87.0 | 96.5 |
Appendix figures & tables63 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Layers | Layer used | ||
|---|---|---|---|---|
| GPT-2 | 12 | 6 | 768 | 3,072 |
| Pythia-70M | 6 | 3 | 512 | 2,048 |
| Pythia-160M | 12 | 6 | 768 | 3,072 |
| Pythia-1.4B | 24 | 12 | 2,048 | 8,192 |
| Pythia-6.9B | 32 | 16 | 4,096 | 16,384 |
| Llama-3.1-8B | 32 | 16 | 4,096 | 16,384 |
| Setting | Value |
|---|---|
| Training activations | 5,000,000 |
| Dictionary width | |
| TopK budget | 64 |
| Optimizer | Adam |
| Learning rate | |
| Adam moments |
| Article set | Target tokens | Use |
|---|---|---|
| Calibration | 65,536 | Resampling and fitting controls |
| Ranking | 32,768 | Ranking features and component ablations |
| Validation | 32,768 | Validation |
| PC1 prediction test | 65,536 | Testing predictions of model quantities |
| Explanation | 500,000 | Writing feature descriptions |
| Detection | 500,000 | Testing feature descriptions |
| Model | Dictionary S/B | Top 50 S/B | All-pairs top 50 S/B | Post-hoc |
|---|---|---|---|---|
| GPT-2 | 46.6/44.3 | 5.5/73.0 | 5.4/72.7 | 77.5 |
| Pythia-70M | 47.0/45.5 | 47.5/81.0 | 43.2/77.9 | 81.5 |
| Pythia-160M | 49.0/46.5 | 13.0/83.0 | 13.3/86.1 | 87.0 |
| Pythia-1.4B | 62.9/58.7 | 6.0/92.0 | 6.8/88.4 | 90.0 |
| Pythia-6.9B | 62.4/61.1 | 4.0/95.5 | 4.4/96.1 | 96.0 |
| Llama-3.1-8B | 66.2/68.0 | 39.5/94.5 | 45.7/93.8 | 90.5 |
| Model | Standard | Bypass | Difference |
|---|---|---|---|
| GPT-2 | [5.5, 5.5] | [69.5, 78.5] | [64.0, 73.0] |
| Pythia-70M | [40.5, 52.5] | [74.0, 83.0] | [25.0, 39.5] |
| Pythia-160M | [9.0, 13.0] | [80.5, 86.5] | [68.0, 75.5] |
| Pythia-1.4B | [4.0, 8.0] | [88.5, 93.5] | [82.0, 88.0] |
| Pythia-6.9B | [4.0, 6.0] | [92.0, 99.0] | [87.0, 95.0] |
| Llama-3.1-8B | [36.5, 46.5] | [91.0, 96.5] | [47.0, 58.5] |
| Model | Removed (%) | Standard | Post-hoc | Bypass |
|---|---|---|---|---|
| GPT-2 | 2.26 | 0.0966 | 0.0130 | 0.0141 |
| Pythia-70M | 3.92 | 0.0372 | 0.0248 | 0.0256 |
| Pythia-160M | 2.36 | 0.0690 | 0.0183 | 0.0182 |
| Pythia-1.4B | 1.04 | 0.0536 | 0.0064 | 0.0059 |
| Pythia-6.9B | 0.74 | 0.0401 | 0.0050 | 0.0044 |
| Llama-3.1-8B | 0.74 | 0.0165 | 0.0070 | 0.0064 |
| Model | Dictionary | Top 10 | Top 50 | Top 150 | Top 300 |
|---|---|---|---|---|---|
| GPT-2 | 46.6 | 10.0 | 5.5 | 45.8 | 60.1 |
| Pythia-70M | 47.0 | 30.0 | 47.5 | 58.3 | 62.3 |
| Pythia-160M | 49.0 | 22.5 | 13.0 | 48.5 | 63.8 |
| Pythia-1.4B | 62.9 | 20.0 | 6.0 | 52.8 | 69.4 |
| Pythia-6.9B | 62.4 | 20.0 | 4.0 | 55.5 | 71.6 |
| Llama-3.1-8B | 66.2 | 67.5 | 39.5 | 63.2 | 73.1 |
| Model | Zero | Mean | Gradient activation | Integrated gradients |
|---|---|---|---|---|
| GPT-2 | 5.5/73.0 | 4.0/71.5 | 12.5/83.5 | 7.0/82.0 |
| Pythia-70M | 47.5/81.0 | 46.0/84.5 | 53.5/64.5 | 44.5/71.0 |
| Pythia-160M | 13.0/83.0 | 10.0/84.5 | 17.0/74.0 | 15.0/78.0 |
| Pythia-1.4B | 6.0/92.0 | 2.0/92.0 | 6.0/89.5 | 10.0/90.5 |
| Pythia-6.9B | 4.0/95.5 | 2.0/93.5 | 7.5/93.5 | 9.0/94.5 |
| Llama-3.1-8B | 39.5/94.5 | 39.5/93.0 | 17.0/93.0 | 11.0/96.5 |
| Method | Replacement rule |
|---|---|
| Marginal | Replace every activation with a value sampled from that feature’s calibration activations, including zeros. |
| Support-preserving | Leave zeros unchanged; replace positive activations with sampled positive values for the same feature. |
| Position-stratified | Leave zeros unchanged; sample positive values from the same range of token positions. |
| Model | Marginal | Support | Position | Baseline S/B | ||
|---|---|---|---|---|---|---|
| GPT-2 | 2962/2962 | 3070/3070 | 4.0/77.5 | 83.0/85.0 | 87.0/89.0 | 46.6/44.3 |
| Pythia-70M | 2016/2016 | 2035/2035 | 32.0/80.0 | 51.5/83.0 | 71.0/78.5 | 47.0/45.5 |
| Pythia-160M | 3046/3046 | 3066/3066 | 6.0/76.5 | 50.5/90.0 | 75.0/88.5 | 49.0/46.5 |
| Pythia-1.4B | 1207/6552 | 880/7223 | 0.0/93.5 | 27.5/95.5 | 42.5/97.5 | 64.9/73.3 |
| Pythia-6.9B | 697/9107 | 1138/10209 | 2.0/95.5 | 51.5/94.0 | 74.0/94.0 | 69.4/73.0 |
| Llama-3.1-8B | 1418/9795 | 1001/11996 | 8.0/96.5 | 96.5/97.5 | 97.5/99.5 | 67.0/75.2 |
| Model | Whole | Parallel | Orthogonal | Orthogonal/whole (%) |
|---|---|---|---|---|
| GPT-2 | 4.834 | 0.055 | 4.713 | 97.5 |
| Pythia-70M | 1.862 | 0.028 | 1.961 | 105.3 |
| Pythia-160M | 3.376 | -0.027 | 3.389 | 100.4 |
| Pythia-1.4B | 2.676 | 0.564 | 2.347 | 87.7 |
| Pythia-6.9B | 1.992 | 0.686 | 1.545 | 77.6 |
| Llama-3.1-8B | 0.825 | 0.138 | 0.589 | 71.3 |
| Model | Features | Whole | Parallel | Orthogonal |
|---|---|---|---|---|
| GPT-2 | 455 | 2.385 | 0.825 | 1.318 |
| Pythia-70M | 318 | 1.071 | 0.128 | 1.240 |
| Pythia-160M | 486 | 1.202 | 0.224 | 1.044 |
| Pythia-1.4B | 713 | 1.041 | 0.572 | 0.437 |
| Pythia-6.9B | 114 | 0.588 | 0.421 | 0.120 |
| Llama-3.1-8B | 935 | 2.626 | 2.148 | 0.558 |
| Model | Source rank | Mean | Isotropic null |
|---|---|---|---|
| GPT-2 | 455 | 0.683 | 0.592 |
| Pythia-70M | 318 | 0.761 | 0.686 |
| Pythia-160M | 486 | 0.703 | 0.635 |
| Pythia-1.4B | 713 | 0.591 | 0.350 |
| Pythia-6.9B | 114 | 0.274 | 0.029 |
| Llama-3.1-8B | 935 | 0.601 | 0.311 |
| SAE | sel. | test | Dict. | Total FVU | Content FVU | ||
|---|---|---|---|---|---|---|---|
| Standard | 0.00 | 6.4 | 6.4 | 45.8 | 64.00 | 0.004838 | 0.13941 0.00034 |
| Standard | 0.03 | 17.9 | 14.8 | 47.9 | 63.97 | 0.004824 | 0.13894 0.00028 |
| Standard | 0.10 | 49.9 | 52.1 | 51.2 | 63.80 | 0.004820 | 0.13865 0.00014 |
| Standard | 0.30 | 95.2 | 94.9 | 57.3 | 63.47 | 0.004830 | 0.13869 0.00022 |
| Bypass | 0.00 | 69.1 | 65.8 | 44.3 | 63.84 | 0.004739 | 0.13666 0.00021 |
| Bypass | 0.03 | 82.9 | 82.2 | 47.1 | 63.77 | 0.004734 | 0.13644 0.00011 |
| Int. count | Int. shared | Int. in top 50 | High count | High shared | Low shared | |
|---|---|---|---|---|---|---|
| 0.00 | 65.2 | 2.7 | 93.6 | 1.8 | 95.0 | 46.7 |
| 0.03 | 96.6 | 25.9 | 93.2 | 2.4 | 90.0 | 48.6 |
| 0.10 | 461.4 | 63.3 | 80.0 | 5.0 | 54.4 | 49.0 |
| 0.30 | 1306.6 | 71.7 | 72.8 | 9.8 | 70.9 | 46.3 |
| Dict. | : 42 | : 137 | Total FVU | Content FVU | ||
|---|---|---|---|---|---|---|
| 0 | 10.0 | 62.5 | 63.96 | 63.99 | 0.009167 | 0.22767 |
| 1e-06 | 10.0 | 62.5 | 63.97 | 64.00 | 0.009163 | 0.22756 |
| 1e-05 | 8.0 | 62.6 | 63.98 | 63.99 | 0.009166 | 0.22764 |
| 0.0001 | 8.0 | 63.2 | 63.98 | 63.98 | 0.009163 | 0.22756 |
| 0.001 | 10.0 | 62.1 | 63.98 | 63.98 | 0.009162 | 0.22753 |
| 0.003 | 12.0 | 63.7 | 63.98 | 63.98 | 0.009160 | 0.22748 |
| Model | Full cosine | Perp. cosine | Full shared | Perp. shared | Excluded | |
|---|---|---|---|---|---|---|
| GPT-2 | 0 | 0.178 | 0.165 | 6.0 | 4.0 | 0 |
| GPT-2 | 0.03 | 0.316 | 0.235 | 18.0 | 16.0 | 0 |
| GPT-2 | 0.1 | 0.662 | 0.456 | 50.0 | 50.0 | 0 |
| GPT-2 | 0.3 | 0.901 | 0.831 | 96.0 | 88.0 | 0 |
| Pythia-1.4B | 0 | 0.268 | 0.114 | 10.0 | 10.0 | 0 |
| Pythia-1.4B | 0.001 | 0.281 | 0.115 | 10.0 | 10.0 | 0 |
| Model | Objective | Seeds | S | B | Dict. S | Dict. B | S | B |
|---|---|---|---|---|---|---|---|---|
| GPT-2 | TopK | 5/5 | 5.5 | 79.5 | 46.6 | 44.3 | 64.0 | 63.8 |
| GPT-2 | BatchTopK | 5/5 | 17.5 | 64.0 | 42.6 | 45.1 | 63.0 | 63.8 |
| GPT-2 | JumpReLU (sched.) | 5/5 | 11.5 | 66.5 | 42.8 | 46.4 | 65.1 | 66.0 |
| GPT-2 | ReLU+ | 5/5 | 100.0 | 100.0 | 78.0 | 82.4 | 49.2 | 57.5 |
| GPT-2 | Gated | 3/3 | 11.0 | 92.0 | 41.1 | 51.5 | 56.7 | 59.9 |
| Pythia-160M | TopK | 5/5 | 9.0 | 83.5 | 49.0 | 46.5 | 64.0 | 64.0 |
| Model | Objective | S | B | range S | range B |
|---|---|---|---|---|---|
| GPT-2 | ReLU+ | 5.062 | 4.171 | 48.8–49.4 | 57.0–57.8 |
| GPT-2 | Gated | 5.624 | 5.273 | 54.5–61.0 | 59.7–60.0 |
| Pythia-160M | ReLU+ | 1.155 | 1.138 | 52.4–53.7 | 50.2–51.2 |
| Pythia-160M | Gated | 1.221 | 1.241 | 56.0–56.5 | 56.0–56.8 |
| Pythia-1.4B | ReLU+ | 3.186 | 2.939 | 52.0–52.7 | 46.7–46.9 |
| Pythia-1.4B | Gated | 3.326 | 3.681 | 60.8–65.8 | 54.5–55.9 |
| Parameters updated | Dict. | Content FVU | |||
|---|---|---|---|---|---|
| 0 | Original gate penalty | 11.0 | 41.1 | 56.76 | 0.15699 |
| 0.03 | All coefficient paths | 57.0 | 43.1 | 56.03 | 0.15188 |
| 0.10 | All coefficient paths | 92.0 | 50.7 | 55.08 | 0.14712 |
| 0.30 | All coefficient paths | 98.0 | 54.7 | 59.88 | 0.14577 |
| 1.00 | All coefficient paths | 93.0 | 63.7 | 65.15 | 0.14573 |
| 0.10 | Magnitude bias/scale | 89.0 | 49.7 | 54.92 | 0.14704 |
| Gate coefficient | Dict. | : 42 | : 137 | : 0 | Content FVU | |
|---|---|---|---|---|---|---|
| 7.0 | 15.0 | 60.9 | 71.39 | 69.97 | 69.28 | 0.31909 |
| 8.0 | 77.0 | 56.1 | 54.96 | 58.66 | 59.48 | 0.32546 |
| Model | Threshold/arm | Seeds | Content FVU | Splice CE | |
|---|---|---|---|---|---|
| GPT-2 | learned | 3 | 4.12589 | ||
| GPT-2 | learned bypass | 3 | 4.08565 | ||
| GPT-2 | scheduled | 3 | 4.09868 | ||
| Pythia-1.4B | learned | 3 | 3.20498 |
| Training change | Alive | Tiles | High alignment | Total FVU | Content FVU | |
|---|---|---|---|---|---|---|
| Standard | 64.00 | 2864.0 | 65.0 | 1.0 | 0.00676 | 0.14898 |
| Canonical init. | 63.99 | 2106.0 | 1.0 | 2.0 | 0.00748 | 0.16501 |
| Canonical pinned | 63.98 | 1879.0 | 0.0 | 2.0 | 0.00788 | 0.17362 |
| Half width | 64.00 | 1529.5 | 65.5 | 1.0 | 0.00799 | 0.17624 |
| Double width | 64.00 | 4851.5 | 64.0 | 1.0 | 0.00593 | 0.13083 |
| 32.00 | 2977.0 | 35.0 | 1.0 | 0.00832 | 0.18334 |
| Training | Dict. | CE sharing | CE interval | KL sharing | KL interval | |
|---|---|---|---|---|---|---|
| Standard | 10 | 40.0 | 18.8 | 13.2–21.7 | 12.0 | 9.1–17.0 |
| Standard | 50 | 40.0 | 5.5 | 5.0–8.1 | 4.9 | 4.2–5.6 |
| Standard | 100 | 40.0 | 26.4 | 24.7–28.4 | 30.8 | 29.8–32.4 |
| Bypass | 10 | 37.3 | 81.0 | 72.1–85.6 | 92.0 | 87.0–95.5 |
| Bypass | 50 | 37.3 | 65.4 | 58.8–69.1 | 77.0 | 74.1–81.8 |
| Bypass | 100 | 37.3 | 60.4 | 52.6–61.8 | 78.0 | 75.2–79.8 |
| Training | Ranking tokens | CE | CE 1:1 | KL | KL 1:1 |
|---|---|---|---|---|---|
| Standard | All | 5.5 | 5.5 | 4.9 | 4.9 |
| Standard | After first | 70.0 | 69.6 | 80.0 | 79.3 |
| Bypass | All | 65.4 | 65.3 | 77.0 | 76.8 |
| Bypass | After first | 65.7 | 65.5 | 77.1 | 76.8 |
| Canonical init. | All | 68.0 | 68.0 | 74.4 | 74.3 |
| Canonical init. | After first | 69.3 | 69.2 | 77.9 | 77.8 |
| Model | Objective | Top 50 | Dictionary | |
|---|---|---|---|---|
| GPT-2 | TopK | 5.8/79.1 | 46.5/44.0 | 64.00/63.82 |
| GPT-2 | BatchTopK | 15.9/70.0 | 43.2/45.0 | 63.04/63.77 |
| GPT-2 | ReLU+ | 99.1/99.2 | 75.3/82.1 | 60.02/59.52 |
| Pythia-160M | TopK | 9.3/85.2 | 49.3/46.0 | 63.99/64.00 |
| Pythia-160M | BatchTopK | 14.6/82.9 | 45.2/46.3 | 63.23/63.76 |
| Pythia-160M | ReLU+ | 96.8/96.8 | 77.0/82.7 | 58.76/59.00 |
| Model | Seeds | (%) | Total S | Content S | Total B | Content B | Gain | |
|---|---|---|---|---|---|---|---|---|
| GPT-2 | 5 | 96.850 | 0.004428 | 0.14056 | 31.7 | 0.004330 | 0.13746 | 2.2 |
| Pythia-70M | 5 | 81.918 | 0.020368 | 0.11251 | 5.5 | 0.020168 | 0.11154 | 0.9 |
| Pythia-160M | 5 | 93.096 | 0.010274 | 0.14874 | 14.5 | 0.010123 | 0.14662 | 1.4 |
| Pythia-1.4B | 5 | 96.206 | 0.008595 | 0.22650 | 26.4 | 0.008459 | 0.22294 | 1.6 |
| Pythia-6.9B | 3 | 95.501 | 0.012763 | 0.28363 | 22.2 | 0.012520 | 0.27831 | 1.9 |
| Llama-3.1-8B | 3 | 96.738 | 0.008330 | 0.25510 | 30.6 | 0.007914 | 0.24259 | 4.9 |
| Model | Fixed PCA | Adaptive PCA | Standard SAE | Bypass SAE |
|---|---|---|---|---|
| GPT-2 | 0.54452 | 0.36167 | 0.14137 | 0.13786 |
| Pythia-70M | 0.33588 | 0.21147 | 0.11594 | 0.11471 |
| Pythia-160M | 0.43916 | 0.30388 | 0.15142 | 0.14952 |
| Pythia-1.4B | 0.66039 | 0.52780 | 0.22899 | 0.22582 |
| Pythia-6.9B | 0.74459 | 0.63860 | 0.28156 | 0.27637 |
| Llama-3.1-8B | 0.69284 | 0.58520 | 0.24665 | 0.23346 |
| Model | Method | Seeds | Total | Content FVU | (%) | |
|---|---|---|---|---|---|---|
| GPT-2 | Standard | 5 | 64 | 64.00 | – | |
| GPT-2 | Bypass 63+1 | 5 | 64 | 62.82 | 75.3 | |
| GPT-2 | Bypass 64+1 | 5 | 65 | 63.82 | – | |
| Pythia-1.4B | Standard | 3 | 64 | 63.98 | – | |
| Pythia-1.4B | Bypass 63+1 | 3 | 64 | 63.00 | 80.1 | |
| Pythia-1.4B | Bypass 64+1 | 3 | 65 | 64.00 | – |
| Model | Tokens | Tokens (%) | Active | Error S | Error B | Gain share (%) |
|---|---|---|---|---|---|---|
| GPT-2 | Large norm | 1.40 | 63.11 | 247.22 | 105.95 | 15.9 |
| GPT-2 | Ordinary | 98.60 | 2.85 | 575.84 | 565.08 | 84.1 |
| Pythia-1.4B | Large norm | 2.87 | 62.94 | 158.64 | 61.08 | 47.0 |
| Pythia-1.4B | Ordinary | 97.13 | 1.26 | 385.30 | 382.05 | 53.0 |
| Qwen-2.5-7B | Large norm | 1.88 | 63.17 | 21003.88 | 2506.10 | 66.8 |
| Qwen-2.5-7B | Ordinary | 98.12 | 26.91 | 1166.54 | 988.66 | 33.2 |
| Model | Predictions | Model CE | Standard cost | Bypass cost | Reduction (%) |
|---|---|---|---|---|---|
| GPT-2 | 17,558 | 3.95775 | 0.08054 | 0.07386 | 8.3 |
| Pythia-70M | 17,571 | 4.53997 | 0.17583 | 0.17470 | 0.6 |
| Pythia-160M | 17,571 | 3.95300 | 0.15933 | 0.14691 | 7.8 |
| Pythia-1.4B | 17,571 | 3.19029 | 0.10813 | 0.09693 | 10.4 |
| Pythia-6.9B | 17,571 | 2.96687 | 0.23805 | 0.21291 | 10.6 |
| Llama-3.1-8B | 17,931 | 2.72509 | 0.08931 | 0.08647 | 3.2 |
| Model | Set | Eligible/total | 42 | 137 | 0 | Mean | Control |
|---|---|---|---|---|---|---|---|
| GPT-2 | Standard | 512/512 | 0.00 | 0.00 | 0.00 | 0.00 | 100.00 |
| GPT-2 | Post-hoc | 512/512 | 6.25 | 17.58 | 3.32 | 9.05 | 100.00 |
| GPT-2 | Bypass | 512/512 | 4.10 | 15.23 | 1.95 | 7.10 | 100.00 |
| GPT-2 | Random | 512/512 | 0.00 | 0.00 | 0.00 | 0.00 | 100.00 |
| Pythia-160M | Standard | 264/264 | 0.00 | 0.00 | 0.00 | 0.00 | 100.00 |
| Pythia-160M | Post-hoc | 264/264 | 0.00 | 0.00 | 0.00 | 0.00 | 99.75 |
| Model | Set | ||||||
|---|---|---|---|---|---|---|---|
| GPT-2 | Standard | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| GPT-2 | Post-hoc | 8.66 | 8.98 | 9.05 | 0.00 | 0.00 | 49.41 |
| GPT-2 | Bypass | 7.03 | 7.10 | 7.10 | 0.00 | 0.00 | 48.70 |
| GPT-2 | Random | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Pythia-160M | Standard | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Pythia-160M | Post-hoc | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Model | Set | Original | Top 50 | Full dict. | Full + filter | Cosine | Repeats |
|---|---|---|---|---|---|---|---|
| GPT-2 | Standard | 0.00 | 0.00 | 0.00 | 0.00 | 0.126 | 0.17 |
| GPT-2 | Post-hoc | 9.05 | 8.59 | 8.85 | 8.72 | 0.827 | 0.50 |
| GPT-2 | Bypass | 7.10 | 7.23 | 7.75 | 7.23 | 0.888 | 0.17 |
| GPT-2 | Random | 0.00 | 0.00 | 0.00 | 0.00 | 0.127 | 0.22 |
| Pythia-160M | Standard | 0.00 | 0.00 | 0.00 | 0.00 | 0.297 | 0.00 |
| Pythia-160M | Post-hoc | 0.00 | 0.00 | 0.00 | 0.00 | 0.965 | 0.33 |
| Task | Model | Seeds | Standard | Post-hoc | Bypass |
|---|---|---|---|---|---|
| SVA | GPT-2 | 5 | 0.00 | 34.27 | 26.40 |
| SVA | Pythia-70M | 5 | 74.85 | 74.60 | 76.31 |
| SVA | Pythia-160M | 5 | 0.02 | 15.55 | 17.60 |
| SVA | Pythia-1.4B | 5 | 0.00 | 67.04 | 67.14 |
| SVA | Pythia-6.9B | 1 | 0.00 | 0.00 | 0.00 |
| SVA | Llama-3.1-8B | 1 | 31.85 | 29.82 | 31.97 |
| Task | Model | Examples | Post-hoc standard | Bypass standard |
|---|---|---|---|---|
| SVA | GPT-2 | 4096 | 35.1 [33.6, 36.5] | 20.0 [18.8, 21.2] |
| SVA | Pythia-70M | 3346 | 0.9 [0.1, 1.6] | 5.1 [4.0, 6.2] |
| SVA | Pythia-160M | 3454 | 7.9 [7.0, 8.8] | 20.4 [19.2, 21.8] |
| SVA | Pythia-1.4B | 3456 | 65.2 [63.6, 66.8] | 66.8 [65.2, 68.3] |
| SVA | Pythia-6.9B | 3456 | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] |
| SVA | Llama-3.1-8B | 4094 | -2.0 [-2.5, -1.6] | 0.1 [-0.8, 1.0] |
| Model | Top: random | Control: random | Top: hard | Control: hard |
|---|---|---|---|---|
| GPT-2 | 0.470/0.727 | 0.753/0.757 | 0.411/0.633 | 0.677/0.679 |
| Pythia-160M | 0.566/0.737 | 0.723/0.737 | 0.539/0.618 | 0.641/0.657 |
| Pythia-1.4B | 0.551/0.732 | 0.773/0.755 | 0.475/0.621 | 0.668/0.646 |
| Pythia-6.9B | 0.568/0.757 | 0.812/0.799 | 0.527/0.636 | 0.700/0.694 |
| Qwen-2.5-7B | 0.521/0.722 | 0.789/0.818 | 0.509/0.629 | 0.670/0.705 |
| Llama-3.1-8B | 0.835/0.742 | 0.819/0.805 | 0.780/0.599 | 0.675/0.665 |
| Model | Top: random | Control: random | Top: hard | Control: hard |
|---|---|---|---|---|
| GPT-2 | 0.665/0.715 | 0.714/0.720 | 0.629/0.619 | 0.672/0.648 |
| Pythia-160M | 0.626/0.708 | 0.704/0.704 | 0.589/0.615 | 0.638/0.661 |
| Pythia-1.4B | 0.669/0.743 | 0.734/0.755 | 0.554/0.631 | 0.665/0.663 |
| Pythia-6.9B | 0.645/0.737 | 0.791/0.766 | 0.594/0.608 | 0.713/0.684 |
| Qwen-2.5-7B | 0.586/0.699 | 0.764/0.779 | 0.625/0.621 | 0.673/0.700 |
| Llama-3.1-8B | 0.831/0.733 | 0.783/0.772 | 0.779/0.581 | 0.691/0.689 |
| Model | Qwen explains | Mistral explains |
|---|---|---|
| GPT-2 | 0.257 [0.196, 0.313] | 0.049 [0.008, 0.092] |
| Pythia-160M | 0.171 [0.119, 0.225] | 0.082 [0.037, 0.123] |
| Pythia-1.4B | 0.181 [0.123, 0.235] | 0.074 [0.030, 0.117] |
| Pythia-6.9B | 0.188 [0.139, 0.237] | 0.093 [0.049, 0.138] |
| Llama-3.1-8B | -0.093 [-0.139, -0.048] | -0.097 [-0.147, -0.047] |
| Qwen-2.5-7B | 0.200 [0.136, 0.267] | 0.113 [0.064, 0.158] |
| Model | SST-2 | AG News | Amazon | Emotion | |
|---|---|---|---|---|---|
| GPT-2 | 80.78/79.22 | 80.84/80.61 | 79.58/80.24 | 59.35/59.22 | -0.31 |
| Pythia-160M | 78.12/78.35 | 80.92/79.91 | 79.76/80.62 | 55.84/57.76 | +0.50 |
| Pythia-1.4B | 85.69/85.41 | 80.63/83.76 | 86.60/86.96 | 62.10/62.67 | +0.95 |
| Pythia-6.9B | 87.82/87.64 | 80.00/78.44 | 86.98/88.00 | 61.24/63.24 | +0.32 |
| Qwen-2.5-7B | 87.71/88.19 | 83.17/81.49 | 85.06/86.82 | 59.76/60.70 | +0.38 |
| Llama-3.1-8B | 88.83/90.05 | 77.42/76.97 | 88.80/89.80 | 62.70/60.27 | -0.17 |
| Model | Difference | 95% interval |
|---|---|---|
| GPT-2 | -0.31 | [-0.78, 0.17] |
| Pythia-160M | 0.50 | [0.03, 1.00] |
| Pythia-1.4B | 0.95 | [0.54, 1.36] |
| Pythia-6.9B | 0.32 | [-0.10, 0.75] |
| Llama-3.1-8B | -0.17 | [-0.61, 0.28] |
| Qwen-2.5-7B | 0.38 | [-0.12, 0.89] |
| Model | Mean: | Mean: | Mean: | Last: |
|---|---|---|---|---|
| GPT-2 | 59.10/56.55 | 75.14/74.82 | 78.99/79.13 | 63.49/63.75 |
| Pythia-160M | 55.61/55.52 | 73.66/74.16 | 77.82/78.19 | 55.92/61.78 |
| Pythia-1.4B | 56.43/58.16 | 78.75/79.70 | 82.63/82.98 | 65.11/64.84 |
| Pythia-6.9B | 59.67/59.13 | 79.01/79.33 | 82.96/83.37 | 66.38/66.94 |
| Llama-3.1-8B | 58.20/58.82 | 79.44/79.27 | 83.28/82.89 | 71.18/70.66 |
| Qwen-2.5-7B | 58.19/59.96 | 78.92/79.30 | 81.99/82.89 | 67.72/69.90 |
| Model | Dense | Dense | PC1 | Random S/B | Bypass + PC1 |
|---|---|---|---|---|---|
| GPT-2 | 80.02 | 80.04 | 38.19 | 47.78/48.60 | 74.77 |
| Pythia-160M | 79.19 | 79.24 | 38.57 | 48.03/48.25 | 74.16 |
| Pythia-1.4B | 83.89 | 83.87 | 38.71 | 46.79/47.76 | 79.83 |
| Pythia-6.9B | 85.98 | 85.97 | 39.66 | 45.70/45.85 | 79.49 |
| Llama-3.1-8B | 86.41 | 86.40 | 38.38 | 45.54/46.20 | 79.37 |
| Qwen-2.5-7B | 86.20 | 86.27 | 38.72 | 46.65/46.72 | 79.22 |
| Tokens included | Count | : norm | 95% interval | : frequency | 95% interval |
|---|---|---|---|---|---|
| All | 65,535 | 0.999808 | [0.999786, 0.999826] | 0.543 | [0.532, 0.553] |
| First position | 862 | 0.999994 | [0.999993, 0.999994] | -0.621 | [-0.684, -0.568] |
| Other positions | 64,673 | 0.999616 | [0.999568, 0.999656] | 0.552 | [0.540, 0.562] |
| Model | Target | PC1 | Norm | Random | Random | Raw | |
|---|---|---|---|---|---|---|---|
| GPT-2 | Entropy | 0.057 | 0.052 | 0.062 | 0.013 | 0.597 | 0.649 |
| GPT-2 | Surprisal | 0.019 | 0.017 | 0.023 | 0.005 | 0.241 | 0.258 |
| GPT-2 | Frequency | 0.000 | 0.000 | 0.012 | 0.062 | 0.938 | 0.938 |
| GPT-2 | Position | 0.170 | 0.169 | 0.207 | 0.078 | 0.826 | 0.988 |
| GPT-2 | Sink attention | 0.000 | 0.058 | 0.040 | 0.039 | 0.896 | 0.896 |
| Pythia-160M | Entropy | 0.040 | 0.029 | 0.064 | 0.046 | 0.687 | 0.697 |
| Model | Unedited | Delete PC1 | Position mean | Delete PC2 | Random |
|---|---|---|---|---|---|
| GPT-2 | 3.929 | 4.728 | 3.933 | 4.219 | 3.938 |
| Pythia-160M | 3.975 | 4.198 | 4.054 | 3.997 | 3.986 |
| Pythia-1.4B | 3.149 | 3.694 | 3.292 | 3.174 | 3.162 |
| Pythia-6.9B | 2.968 | 3.379 | 3.271 | 3.046 | 2.969 |
| Llama-3.1-8B | 2.600 | 4.610 | 2.600 | 2.747 | 2.599 |
| Qwen-2.5-7B | 2.578 | 8.443 | 4.437 | 2.575 | 2.585 |
| Model | Layer | Unedited | Remove PC1 | Random | Restore later |
|---|---|---|---|---|---|
| GPT-2 | 2 | 3.929 | 3.913 | 3.930 | 3.927 |
| GPT-2 | 0 | 3.929 | 3.926 | 3.929 | 3.927 |
| Pythia-160M | 3 | 3.975 | 4.746 | 3.976 | 4.520 |
| Pythia-160M | 0 | 3.975 | 4.105 | 3.977 | 4.101 |
| Pythia-1.4B | 3 | 3.149 | 3.806 | 3.149 | 3.808 |
| Pythia-1.4B | 4 | 3.149 | 3.176 | 3.149 | 3.146 |
| Activations | Trained PC1 | Trained PC2 | Untrained PC1 |
|---|---|---|---|
| Embedding | 5.02 | 4.42 | 4.49–4.88 |
| Layer 0 | 9.82 | 4.24 | 4.35–4.64 |
| Layer 1 | 21.02 | 4.66 | 4.23–4.89 |
| Layer 2 | 59.39 | 2.53 | 3.90–5.04 |
| Layer 3 | 92.70 | 0.81 | 3.84–4.93 |
| Layer 4 | 93.28 | 0.74 | 3.81–4.55 |
| Model | Layer | Top 50 | Dictionary | |
|---|---|---|---|---|
| GPT-2 | 0 | 53.7/50.7 | 6.2/6.0 | 63.94/64.00 |
| GPT-2 | 11 | 94.0/95.1 | 58.4/58.5 | 64.00/64.00 |
| Pythia-160M | 1 | 48.6/50.0 | 7.4/7.3 | 64.00/64.00 |
| Pythia-160M | 9 | 18.4/93.1 | 54.7/53.4 | 63.98/64.00 |
| Pythia-1.4B | 2 | 55.8/54.9 | 15.2/14.6 | 63.98/64.00 |
| Pythia-1.4B | 23 | 87.0/90.8 | 59.3/59.2 | 63.90/63.98 |
| Layer | (%) | S | B | Dict. S | Dict. B | CE S | CE B |
|---|---|---|---|---|---|---|---|
| 3 | 93.82 | 14.0 | 68.0 | 30.0 | 23.0 | 3.25249 | 3.24169 |
| 12 | 92.95 | 8.0 | 92.0 | 63.2 | 62.4 | 3.22981 | 3.22407 |
| 23 | 11.09 | 96.0 | 92.0 | 64.6 | 63.7 | 3.67123 | 3.66968 |
| PCs removed | 0 | 1 | 2 | 5 | 10 | 20 | 50 |
|---|---|---|---|---|---|---|---|
| Sharing (%) | 4 | 86 | 90 | 84 | 80 | 74 | 84 |
| Model | Context | Tokenization | Tokens | All | After first | No special |
|---|---|---|---|---|---|---|
| GPT-2 | 128 | Default | 9811 | 96.66 | 3.89 | 3.89 |
| GPT-2 | 512 | Default | 12831 | 95.56 | 3.94 | 3.94 |
| Pythia-160M | 128 | Default | 9821 | 92.41 | 88.16 | 88.16 |
| Pythia-160M | 512 | Default | 13103 | 91.06 | 87.06 | 87.06 |
| Pythia-1.4B | 128 | Default | 9821 | 95.65 | 91.60 | 91.60 |
| Pythia-1.4B | 512 | Default | 13103 | 95.14 | 92.01 | 92.01 |
| Model | Layer | Width | Tag | Total FVU | Content FVU | ||
|---|---|---|---|---|---|---|---|
| Gemma Scope | |||||||
| Gemma-2-2B | 6 | 16k | 36 | 38.3 | 0.22550 | 0.23807 | 1.056 |
| Gemma-2-2B | 6 | 16k | 70 | 76.1 | 0.18626 | 0.19670 | 1.056 |
| Gemma-2-2B | 6 | 64k | 107 | 118.4 | 0.12121 | 0.12802 | 1.056 |
| Gemma-2-2B | 6 | 64k | 56 | 60.9 | 0.15026 | 0.15866 | 1.056 |
| Gemma-2-2B | 12 | 16k | 41 | 42.0 | 0.24474 | 0.30489 | 1.246 |
| Layer | Cutoff | Removed (%) | (%) | Total FVU | Content FVU | ||
|---|---|---|---|---|---|---|---|
| 7 | 50 | All | 0.00 | 99.71 | 1.02107 | 178.42778 | 174.746 |
| 7 | 50 | 5 | 0.80 | 2.91 | 0.24005 | 0.24709 | 1.029 |
| 7 | 50 | 10 | 0.80 | 2.91 | 0.24005 | 0.24709 | 1.029 |
| 7 | 50 | 20 | 0.80 | 2.91 | 0.24005 | 0.24709 | 1.029 |
| 7 | 100 | All | 0.00 | 99.71 | 1.10307 | 242.81489 | 220.125 |
| 7 | 100 | 5 | 0.80 | 2.91 | 0.20789 | 0.21402 | 1.029 |
| Context | Tokens included | Count | (%) | Total FVU | Content FVU | |
|---|---|---|---|---|---|---|
| 128 | After first | 99,211 | 80.04 | 0.04049 | 0.20275 | 5.007 |
| 128 | All | 100,000 | 87.61 | 0.02407 | 0.19416 | 8.067 |
| 512 | After first | 99,795 | 54.99 | 0.08597 | 0.19084 | 2.220 |
| 512 | All | 100,000 | 66.63 | 0.06303 | 0.18873 | 2.994 |
| 2049 | All packed | 100,000 | 33.03 | 0.11928 | 0.17796 | 1.492 |
| 2049 | Exclude EOS | 99,978 | 31.59 | 0.12201 | 0.17819 | 1.460 |
| Model | Method | Coefficients | Total FVU | Content FVU | CE | |
|---|---|---|---|---|---|---|
| GPT-2 | Standard | 64+0 | 64.00 | 0.004348 | 0.13650 | 4.11441 |
| GPT-2 | Bypass 63+1 | 63+1 | 62.77 | 0.004276 | 0.13427 | 4.11617 |
| GPT-2 | Bypass 64+1 | 64+1 | 63.77 | 0.004243 | 0.13323 | 4.11435 |
| GPT-2 | Unit norm | 64+1 | 63.98 | 0.004398 | 0.13799 | 4.12948 |
| GPT-2 | Coord. norm | 64+2 | 64.00 | 0.004405 | 0.13822 | 4.12936 |
| GPT-2 | Coord. norm+bypass | 63+3 | 62.51 | 0.004665 | 0.14650 | 4.16077 |
| Model | Method | Tokens | Total FVU | Content FVU |
|---|---|---|---|---|
| GPT-2 | Standard | 4545 | 0.094406 | 0.13591 |
| GPT-2 | Bypass 63+1 | 4545 | 0.093174 | 0.13416 |
| GPT-2 | Norm-filter train | 4545 | 0.092957 | 0.13384 |
| GPT-2 | Whiten | 4545 | 0.106670 | 0.13429 |
| Pythia-1.4B | Standard | 4461 | 0.128739 | 0.22867 |
| Pythia-1.4B | Bypass 63+1 | 4461 | 0.127931 | 0.22725 |
| Condition | (%) | Max/median | Total FVU | Content FVU | |
|---|---|---|---|---|---|
| Standard | 96.19 | 57.24 | 26.21 | ||
| Random bypass | 96.19 | 57.24 | 26.21 | ||
| PC1 bypass | 96.19 | 57.24 | 26.24 | ||
| Test-time, 50 neurons | 3.96 | 2.32 | 1.03 | ||
| Test-time, 2 neurons | 75.66 | 18.57 | 4.12 | ||
| Pretrained registers | 4.00 | 1.71 | 1.03 |
| Condition | Alive | Median best cosine | Shared (%) | |
|---|---|---|---|---|
| Standard | 32.00 | 6,144 | 0.3954 | 28.84 |
| Random bypass | 32.00 | 6,144 | 0.3924 | 28.50 |
| PC1 bypass | 32.00 | 6,144 | 0.4186 | 16.62 |
| Test-time, 50 neurons | 32.00 | 6,144 | 0.4655 | 13.84 |
| Test-time, 2 neurons | 32.00 | 6,144 | 0.4531 | 25.28 |
| Pretrained registers | 32.00 | 6,144 | 0.4793 | 15.91 |
| Method | Definition |
|---|---|
| X0 | Reconstruct both inputs with one output scale. |
| X1 | Reconstruct both inputs with a separate output scale for each. |
| X2 | Bypass PC1 in each input, saving two coefficients. |
| X3 | Bypass PC1 of the concatenated inputs, saving one coefficient. |
| X4 | Bypass one fixed random direction in each input, saving two coefficients. |
| Activation pair | X0 | X1 | X2 | X3 | X4 |
|---|---|---|---|---|---|
| Identical | 2.7 | 2.0 | 91.3 | 92.3 | 3.3 |
| PC1 doubled | 3.7 | 2.0 | 83.0 | 81.7 | 1.3 |
| Layers 12 and 16 | 2.7 | 3.3 | 93.0 | 92.7 | 3.3 |
| Step 71,000 and final | 2.7 | 2.0 | 93.0 | 92.7 | 1.3 |
| Activation pair | Dict. X0 | Dict. X2 | Content FVU X0 | Content FVU X2 |
|---|---|---|---|---|
| Identical | 66.8 | 67.2 | 0.3661 | 0.2753 |
| PC1 doubled | 60.4 | 66.6 | 0.4453 | 0.3053 |
| Layers 12 and 16 | 72.7 | 71.8 | 0.3495 | 0.2741 |
| Step 71,000 and final | 69.7 | 66.4 | 0.3561 | 0.2828 |
| Activation pair | block-sum | joint | A-only (%) | B-only (%) | |
|---|---|---|---|---|---|
| Layers 12 and 16 | 0.001 | 2707.5 | 3044.8 | 7.27 | 20.75 |
| 0.003 | 619.5 | 1003.9 | 0.06 | 7.11 | |
| 0.010 | 62.9 | 104.1 | 0.02 | 1.07 | |
| Step 71,000 and final | 0.001 | 2433.1 | 2766.4 | 13.43 | 0.64 |
| 0.003 | 756.3 | 1193.1 | 0.15 | 0.08 | |
| 0.010 | 72.6 | 125.7 | 0.10 | 0.02 |
Sparse Autoencoders
Sparse autoencoders decompose neural network activations into sparse features to study, interpret, and control learned representations.
See all 242 papers