| Dataset | Model | Metric | GemmaScope | Trained SAEs |
|---|
| Small | Large | ReLU | TopK | AbsTopK | JumpReLU | BiJumpReLU |
| Wikipedia | Gemma2-2B | Utilization ↑ | 95.62(±5.55) | 76.02(±6.66) | 23.63(±32.29) | 15.62(±5.24) | 16.45(±5.74) | 67.63(±14.58) | 91.47(±20.80) |
|---|
| Retention α1 ↑ | 85.74(±11.90) | 85.17(±12.55) | 96.41(±1.75) | 70.02(±1.31) | 69.72(±1.33) | 99.87(±0.10) | 99.97(±0.02) |
|---|
| Fidelity ↑ | 72.17(±13.21) | 75.52(±13.12) | 75.46(±16.17) | 88.68(±7.05) | 88.11(±6.93) | 94.91(±2.38) | 97.25(±1.44) |
|---|
| Stability α2 ↑ | 67.87(±16.69) | 67.43(±18.43) | 85.58(±7.40) | 39.34(±1.01) | 39.07(±1.02) | 98.41(±1.57) | 99.78(±0.13) |
|---|
| Gemma2-9B | Utilization ↑ | 89.65(±7.05) | 57.61(±7.25) | 25.65(±34.93) | 16.35(±6.04) | 19.28(±5.45) | 54.65(±34.54) | 90.13(±25.28) |
|---|