Bridging Singular Learning Theory and Mechanistic Interpretability at the Billion-Parameter Scale
Organizations: Department of Computer Science University of California, Los Angeles
Abstract
Mechanistic interpretability (MI) explains a computation in a trained network by identifying the components and variables that implement it and testing them with interventions. Singular learning theory (SLT) describes the same network through the geometry of the loss around its parameters, summarized by the local learning coefficient. In pretrained language models the two have not been compared: SLT's estimates at this scale concern whole models or components whose roles were not independently established, and the estimator, which averages the loss over parameter settings drawn near the trained ones, has had no reference for how far those draws may move the parameters. We show that both are functions of one object, the map from a component's parameters to the model's predictions under inputs and interventions, whose first derivatives determine the Fisher information, the learning coefficient, and intervention effects, and whose higher derivatives add a quartic correction and pairwise terms. We test this correspondence in Pythia-1B on Dyck bracket matching at all 154 training checkpoints, against a five-stage circuit identified by interventions. We measure the local geometry of all 144 components inside the region where predictions are preserved, with the Fisher spectrum at every checkpoint, which gives the quadratic part of the learning-coefficient estimate at every temperature. Changes of the Fisher trace track changes of causal effect (median partial rank correlation 0.51), largely as a trend shared across the model, whereas the gain Fisher of a component's output and the large-temperature count are specific to the component. The count follows causal change across the population but not the growth of a formed circuit component. The quartic part is what a sampler that leaves the region registers beyond curvature. Pairwise interaction curvature tracks joint-ablation interactions.
Figures & tables
| Model | Pythia-1B at its 154 checkpoints (steps 0, 1, 2, 4, …, 512, then every 1,000), in float64; steps 0, 512, 2,000, 8,000, 32,000, and 143,000 served to choose settings, and statistics use the other 148. |
| Inputs | 128 strings of 128 tokens with (opening probability when both actions are legal; seed 0); averages over the 5,475 closer positions. Replication: an independent batch (seed 20261001) for the 33-component panel. |
| Components | All 144 heads and MLPs; the 26 circuit components of Section 5.1 and a panel of 33 (circuit plus seven controls drawn uniformly; Appendix D ) that carries the radius–temperature grid and the replication. |
| Parameters | A head’s slices of ; an MLP’s ; each tensor in units of its RMS at the final checkpoint, so that a displacement means the same change of weights at every checkpoint. |
| Region | At , , and : KL nats, cross-entropy change nats, accuracy change (closer and all positions); no tensor displaced by more than RMS. |
| Draws | 128 directions per component, shared across checkpoints; the draws are the parameter settings , , for (panel) and the first 64 directions at (all components). |
| , | by exact Hessian–vector products; with , which cancels the quadratic term exactly. |
Appendix figures & tables50 assets
Supplementary material from the paper’s appendix.
Appendix
| Original | Batch 1 | Batch 2 | Batch 3 | |
|---|---|---|---|---|
| All pairs: temporal | .534 | .491 | .563 | .482 |
| All pairs: adjusted | .086 | .098 | .097 | .112 |
| Circuit pairs: temporal | .523 | .436 | .557 | .446 |
| Circuit pairs: adjusted | .184 | .161 | .137 | .161 |
| Control pairs: adjusted | .043 | .048 | .053 | .055 |
| Circuit pairs: adjusted changes | .074 | .076 | .090 | .096 |
| fully inside | median inside | max closer KL | (IQR) | ||
|---|---|---|---|---|---|
| 0.00625 | 1.000 | 1.000 | 0.0000 | 0.0000 | 1.001 (0.998, 1.006) |
| 0.0125 | 1.000 | 1.000 | 0.0000 | 0.0001 | 1.001 (0.998, 1.005) |
| 0.025 | 1.000 | 1.000 | 0.0001 | 0.0004 | 1.001 (0.998, 1.005) |
| 0.05 | 0.983 | 1.000 | 0.0003 | 0.0016 | 1.000 (0.999, 1.004) |
| 0.1 | 0.885 | 1.000 | 0.0011 | 0.0065 | 1.000 (1.000, 1.000) |
| 0.15 | 0.692 | 1.000 | 0.0024 | 0.0143 | 0.999 (0.993, 1.002) |
| 0 | 512 | 2,000 | 8,000 | 32,000 | 143,000 | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Component | ||||||||||||
| L0.mlp | 5.894 | -100 | 5.088 | 96.6 | 0.465 | 0.390 | 0.114 | 0.007 | 0.064 | 0.037 | 0.191 | 0.079 |
| L0H0 | 0.167 | -0.039 | 0.084 | 0.118 | 0.038 | 0.021 | 0.026 | -0.004 | 0.036 | 0.003 | 0.054 | 0.038 |
| L0H1 | 0.187 | -0.026 | 0.106 | 0.166 | 0.034 | 0.031 | 0.021 | 0.010 | 0.021 | 0.009 | 0.056 | 0.029 |
| L0H2 | 0.173 | -0.069 | 0.106 | 0.140 | 0.035 | 0.019 | 0.017 | 0.005 | 0.014 | 0.003 | 0.037 | 0.014 |
| L0H3 | 0.251 | -0.077 | 0.169 | 0.335 | 0.045 | 0.048 | 0.025 | 0.016 | 0.026 | 0.013 | 0.066 | 0.042 |
| fully inside | median inside | max closer KL | (IQR) | ||
|---|---|---|---|---|---|
| 0.00625 | 1.000 | 1.000 | 0.0000 | 0.0000 | 1.001 (0.996, 1.012) |
| 0.0125 | 1.000 | 1.000 | 0.0000 | 0.0001 | 1.001 (0.996, 1.011) |
| 0.025 | 1.000 | 1.000 | 0.0001 | 0.0004 | 1.001 (0.996, 1.011) |
| 0.05 | 1.000 | 1.000 | 0.0003 | 0.0016 | 1.001 (0.997, 1.009) |
| 0.1 | 0.894 | 1.000 | 0.0014 | 0.0061 | 1.000 (1.000, 1.000) |
| 0.15 | 0.661 | 1.000 | 0.0031 | 0.0134 | 0.998 (0.987, 1.004) |
| 0 | 512 | 2,000 | 8,000 | 32,000 | 143,000 | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Component | ||||||||||||
| L0.mlp | 0.357 | -0.339 | 0.359 | 0.942 | 0.145 | 0.660 | 0.088 | 0.081 | 0.094 | 0.075 | 0.191 | 0.079 |
| L0H0 | 0.010 | 0.005 | 0.007 | 0.005 | 0.011 | 0.004 | 0.019 | 0.002 | 0.039 | 0.004 | 0.054 | 0.038 |
| L0H2 | 0.010 | 0.005 | 0.008 | 0.005 | 0.008 | 0.003 | 0.010 | 0.004 | 0.016 | 0.004 | 0.037 | 0.014 |
| L2.mlp | 0.125 | 0.024 | 0.080 | 0.221 | 0.053 | 0.190 | 0.123 | 0.249 | 0.345 | 1.265 | 0.506 | 1.215 |
| L14H1 | 0.001 | 0.001 | 0.001 | 0.000 | 0.003 | 0.001 | 0.009 | 0.003 | 0.034 | 0.022 | 0.050 | 0.037 |
| step 8,000, 144 components | 0.934 | 0.744 | 0.437 | 0.206 | 0.082 |
| step 143,000, 144 components | 0.866 | 0.584 | 0.279 | 0.108 | 0.043 |
| step 8,000, panel on independent inputs | 0.878 | 0.682 | 0.402 | 0.202 | 0.082 |
| step 143,000, panel on independent inputs | 0.816 | 0.576 | 0.303 | 0.135 | 0.050 |
| zero-ablation KL | local ablation KL | opener-distance probe | SGLD estimate | ||||||
|---|---|---|---|---|---|---|---|---|---|
| geometry | group | levels | changes | levels | changes | levels | changes | levels | changes |
| all (144) | |||||||||
| circuit (26) | |||||||||
| other (118) | |||||||||
| , | all | ||||||||
| circuit | |||||||||
| (median [IQR]) | (p90) | ||||
|---|---|---|---|---|---|
| 0.0125 | 0.013% [0.008%, 0.037%] | 1.017 [0.965, 1.094] | 1.05 [0.96, 1.16] | 0.017 | |
| 0.0125 | 0.019% [0.012%, 0.050%] | 1.001 [0.973, 1.028] | 1.02 [0.94, 1.15] | 0.162 | |
| 0.0125 | 0.032% [0.018%, 0.079%] | 1.002 [0.986, 1.021] | 0.91 [0.74, 1.01] | 1.367 | |
| 0.025 | 0.041% [0.026%, 0.13%] | 1.022 [0.956, 1.118] | 1.01 [0.91, 1.11] | 0.006 | |
| 0.025 | 0.049% [0.031%, 0.15%] | 1.020 [0.967, 1.086] | 1.01 [0.92, 1.10] | 0.064 | |
| 0.025 | 0.076% [0.048%, 0.20%] | 1.003 [0.972, 1.035] | 1.01 [0.94, 1.12] | 0.614 |
| , | , | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Component | 0 | 512 | 2,000 | 8,000 | 32,000 | 143,000 | 0 | 512 | 2,000 | 8,000 | 32,000 | 143,000 |
| L0.mlp | -2.1 | +5.0 | +8.6 | +2.3 | +1.8 | +0.8 | -12.8 † | +18.5 † | +34.1 | +16.1 † | +9.0 | +3.1 † |
| L2.mlp | +0.4 | +5.9 | +7.6 | +3.9 | +6.9 | +4.1 | +2.0 | +33.3 | +44.1 | +23.9 | +27.4 | +14.8 † |
| L11.mlp | +1.2 | +2.6 | +4.4 | +3.4 | +5.2 | +3.3 | +8.4 | +18.6 | +25.7 | +21.4 | +29.6 | +20.4 |
| L14H1 | +1.1 | +1.1 | +0.5 | +0.8 | +1.2 | +1.6 | +8.0 | +7.6 | +3.7 | +5.2 | +6.5 | +8.5 |
| L0H0 | +1.2 | +1.5 | +1.0 | +0.3 | +0.7 | +0.6 | +8.4 | +10.5 | +6.5 | +3.9 | +3.6 | +2.4 |
| Restriction | Step size | Localization | Burn-in | |
|---|---|---|---|---|
| One head | 1.77 | 0.012 | 100 | |
| One MLP | 1.77 | 0.025 | 100 | |
| One layer | 1.77 | 0.050 | 120 | |
| Full model | 1.77 | 0.100 | 150 |
| Averaging convention | Raw trajectory | Rank-time adjusted | First difference |
|---|---|---|---|
| All optimization steps | 0.713 | 0.589 | 0.053 |
| Retained draws only | 0.728 | 0.615 | 0.062 |
| Setting | displ./RMS | inside | closer KL | grad/noise | loc/noise | full-model CE |
|---|---|---|---|---|---|---|
| , | 0/48 | |||||
| , | 0/48 | |||||
| , | 0/48 | |||||
| , | 0/48 | |||||
| documented sweep ( ) | 0/48 | |||||
| strong localization ( , ) | 48/48 |
| Measurement | Trajectory | Rank-time adjusted | Adjacent change |
|---|---|---|---|
| Finite SGLD response | |||
| Empirical Fisher (raw) | |||
| Model Fisher (raw) | |||
| Gradient activity (raw) | |||
| Zero ablation | |||
| Clean closer loss |
| Measurement | Quantity | View of the circuit |
|---|---|---|
| Causal intervention | Reference KL after zero or mean replacement | Dependence on a component’s activation |
| Empirical Fisher | Mean squared per-token loss-gradient norm | Sensitivity to the observed task labels |
| CE Hessian | Signed trace of task-loss curvature | Local curvature at current performance |
| Model Fisher / KL Hessian | Curvature of reference-model KL | Local change in the task output distribution |
| LLC estimate | Task constraints across nearby parameter settings |
| Inputs | Closer targets | Clean | Remove STACK | Remove ANSWER |
|---|---|---|---|---|
| Synthetic | 5,493 | |||
| Natural context | 177 |
| Component | Inputs | Named | Complement | ANSWER | Random range |
|---|---|---|---|---|---|
| L11.mlp | synthetic | 0.227 | 0.775 | 0.086 | 0.082–0.084 |
| L11.mlp | natural | 0.101 | 0.877 | 0.004 | 0.066–0.070 |
| L14H1 | synthetic | 0.662 | 0.454 | 0.668 | 0.032–0.037 |
| L14H1 | natural | 0.150 | 0.848 | 0.049 | 0.031–0.033 |
| Distribution | Step | Bank | Head sum | Ratio | Gap/ |
|---|---|---|---|---|---|
| natural | 8,000 | 0.995 | -0.81 | ||
| natural | 32,000 | 0.993 | -1.10 | ||
| natural | 143,000 | 0.994 | -0.75 | ||
| synthetic | 8,000 | 0.983 | -1.65 | ||
| synthetic | 32,000 | 0.979 | -1.81 | ||
| synthetic | 143,000 | 0.942 | -5.68 |
| Synthetic | Natural context | |||
|---|---|---|---|---|
| Control | Rank | Median | Rank | Median |
| Half radius | 0.992 | 0.253 | 1.000 | 0.25 |
| Double radius | 0.975 | 3.86 | 1.000 | 4.03 |
| Half inverse temperature | 0.994 | 0.507 | 1.000 | 0.502 |
| Double inverse temperature | 0.990 | 1.95 | 1.000 | 1.99 |
| Proposal 0.5 | 0.998 | 0.996 | 0.996 | 1 |
| Target of (eight layer-0 heads) | Spearman | partial | changes [95%] |
|---|---|---|---|
| Sum of the eight single-head contrast drops | [ , ] | ||
| Sum of their absolute values | [ , ] | ||
| Sum of the eight single-head zero-ablation KLs | [ , ] | ||
| Joint zero-ablation contrast drop of the group | [ , ] |
| Training step | Joint zero drop | Signed marginal sum | Absolute marginal sum |
|---|---|---|---|
| 1,000 | 0.387 | ||
| 8,000 | 1.378 | 0.572 | |
| 27,000 | 2.378 | 0.614 | |
| 32,000 | 2.797 | 0.078 | 0.462 |
| 64,000 | 2.794 | 0.695 | 0.781 |
| 143,000 | 1.560 | 1.332 | 1.332 |
| Symbolic bracket matching | Transformer components and the operations they implement |
|---|---|
| 1. Encode the current token. Read its bracket type and whether it opens or closes. | L0.mlp and L0H0--H7 . is a token lookup, and the eight parallel heads add , mixing token features across the prefix; replacing the MLP by its lookup preserves the contrast. |
| 2. Update the stack. for an opener; for a closer. Depth is . | L1--L5.mlp , L1H1 , L2H1 , L2H5 . These components write into a fitted STACK subspace with additive slot and type terms, ; removing these writes reduces held-out closer accuracy from to . |
| 3. Read the required type. Let be the most recent unmatched opener. Then . | Readout heads, including L7H2 , L7H4 , L7H5 , L9H4 , and L14H1 . , with the normalized residual stream and the value–output map; attention to opener positions moves type information into the ANSWER subspace. |
| 4. Map the type to a closer. . | L11.mlp and distributed answer-writing heads. The MLP writes positive closer evidence already at layer 11; later heads add more. Removing its ANSWER projection costs accuracy points. |
| 5. Emit the next-token prediction. The stack rule specifies the closer that is legal after the observed prefix. | Final readout and L13--L15.mlp . ; the last three MLPs write opposing evidence, and removing any one’s ANSWER projection raises accuracy by – points. |
| Removed directions | Contrast recovery [95% CI] | Accuracy change (pp) |
|---|---|---|
| Early group: STACK | ||
| Answer writers: ANSWER | ||
| L1.mlp: STACK | ||
| L2.mlp: STACK | ||
| L3.mlp: STACK | ||
| L11.mlp: ANSWER |
| Replacement | Seed 42 | Held-out seeds 43–45 |
|---|---|---|
| Narrow named subspaces | 0.439 | |
| Broad named subspaces | 0.762 | |
| Broad named + 64 orthogonal PCs | 0.920 | |
| PCA (128/64) | 0.901 | |
| Linear surrogate | 0.113 |
| Slot | Clean | STACK L1–5 | STACK L1–3 | Random L1–5 |
|---|---|---|---|---|
| 1 | ||||
| 2 | ||||
| 3 | ||||
| 4 |
| Training step | L7H4 | L7H5 | L7H2 |
|---|---|---|---|
| 1,000 | |||
| 3,000 | |||
| 5,000 | |||
| 32,000 | |||
| 143,000 |
Mechanistic Interpretability
Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.
See all 427 papers