Towards Isolated Interventions via Almost Orthogonal Features in Language Models
https://github.com/mrtzmllr/sae-icm.Organizations: Center for Human-Compatible AI UC Berkeley
Linear probing is a common technique in mechanistic interpretability despite the fact that the decodability of a given feature does not necessarily imply the network uses that feature. Intervention faithfulness, not merely decodability, is the appropriate criterion for whether a feature serves a defined active role in the model. Meanwhile, probe regularization has been used for feature discovery, but the impact of regularization on feature causality remains unclear. We study probe regularization systematically, starting with unregularized ordinary least squares (OLS) regression and considering increasing amounts of regularization via ridge regression. We show theoretically that OLS probes amplify low-variance directions in activation space that have spurious correlations with the target feature, whereas regularization reduces influence from these directions. Empirically, we find in a toy transformer and Llama-2-7B that intervention faithfulness increases up to a point, and then decreases again, and that the optimal regularization strength leads to features bearing striking similarity (in both direction and variance) to those learned by optimizing for intervention outcomes directly. Together, these results provide mechanistic justification for regularizing regression probes.
| Testbed | Direction | Probe loss (MSE) | Int. loss (MSE) | Variance ( ) | |
|---|---|---|---|---|---|
| Mess3 HMM | OLS probe | ||||
| Ridge probe ( ) | |||||
| DAS | |||||
| Llama-2-7B | OLS probe | ||||
| (birth year) | Ridge probe ( ) | ||||
| DAS |
Supplementary material from the paper’s appendix.
Linear probing uses simple linear readouts to test what information is accessible in learned representations and how it transfers across tasks.
See all 42 papershttps://github.com/mrtzmllr/sae-icm.