Localization Does Not Imply Steerability: A Mechanistic Study of Sign Errors in LLM Mathematics
Organizations: Department of Computer Science & Department of Nuclear Engineering and Radiation Science, Missouri University of Science and Technology · CMPP, NED University of Engineering and Technology, Karachi · Missouri University of Science and Technology
Abstract
When large language models do multi-step mathematics, a characteristic failure recurs: the model writes one wrong sign token mid-computation -- a cofactor sign, the minus of a recursive integration-by-parts step -- then executes the rest of the arithmetic correctly, and the answer comes out wrong. We ask where inside the network that sign is written. Across six open models spanning 14B to 675B parameters, a per-layer readout shows the wrong sign is committed late -- by MLP layers in the last quarter of the network (70-100% per model and task), a pattern present, where tested (Qwen), in the base model before instruction tuning. The mechanism varies: some models carry fixed "minus-stamp" layers that push a minus regardless of the right answer, others carry amplifiers of whichever sign was already chosen, and Qwen adds a downstream rescue layer that opposes the stamp. The circuit is causal: subtracting a single sign direction from the residual stream flips 13 of 14 held-out minus-written integration-by-parts errors on Llama-3.3-70B, with flips replicating on Qwen, Gemma, and Mistral-Small (24-70% per cell), and with representation-level specificity -- the same dose breaks correctly-minus answers while sparing correct-plus answers where the direction is surgical (0/72 on Llama), and matched random-direction and control-layer interventions are null. Core claims were preregistered with pass criteria frozen in git before held-out data was seen: 23 of 29 checks passed, all misses reported. But causality is not uniform: Phi-4 implements an equally well-localized circuit at the same late depth, yet the identical intervention flips almost nothing -- the sign is written redundantly across ~15 layers with margins several times larger (median 17.8 logits). Localization, we conclude, does not imply steerability: two models can share a circuit location yet differ enormously in how overturnable it is.
Figures & tables
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Gemma-3-27B | Llama-3.3-70B | Mistral-Small-24B | Phi-4-14B | Qwen-2.5-72B |
|---|---|---|---|---|---|
| Bank size | 4050 | 2850 | 2050 | 2050 | 675 |
| PASS (correct) | 3555 | 2306 | 438 | 223 | 133 |
| FAIL (wrong / no answer) | 495 | 544 | 1612 | 1827 | 542 |
| Accuracy | 87.8% | 80.9% | 21.4% | 10.9% | 19.7% |
| Sign errors as % of FAILs | 54% | 43% | 16% | 20% | 43% |
| SIGN_ERROR working set | 266 | 234 | 263 | 361 | 233 |
| Model | Discovery (file-verified) | Held-out (file-verified) | Total |
|---|---|---|---|
| Gemma-3-27B | 50 | 138 | 188 |
| Llama-3.3-70B | 80 | 130 | 210 |
| Mistral-Small-24B | 78 | 122 | 200 |
| Phi-4-14B | 50 | 149 | 199 |
| Qwen-2.5-72B | 47 | 141 | 188 |
| Stage | Gemma-3-27B | Llama-3.3-70B | Mistral-Small-24B | Phi-4-14B | Qwen-2.5-72B |
|---|---|---|---|---|---|
| Problem bank (train + validation) | 1,294 | 1,294 | 1,294 | 1,294 | 1,294 |
| PASS (correct inference) | 396 | 395 | 215 | 284 | 435 |
| FAIL (wrong / no answer / N/A) | 898 | 899 | 1,079 | 1,010 | 859 |
| Working set (sign-error candidates) | 133 | 43 | 116 | 120 | 155 |
| Experiment ready | 128 | 43 | 109 | 117 | 132 |
| Model | det / + | IBP / + | Total / + | % minus |
|---|---|---|---|---|
| Gemma-3-27B | 87 / 131 | 64 / 79 | 151 / 210 | 41.8% |
| Llama-3.3-70B | 131 / 77 | 30 / 13 | 161 / 90 | 64.1% |
| Mistral-Small-24B | 199 / 38 | 98 / 11 | 297 / 49 | 85.8% |
| Phi-4-14B | 181 / 30 | 109 / 8 | 290 / 38 | 88.4% |
| Qwen-2.5-72B | 102 / 104 | 36 / 0 | 138 / 104 | 57.0% |
| All five | 700 / 380 | 337 / 111 | 1,037 / 491 | 67.9% |
| Model | Phase | Total | Confirmed (lead 2) | Excluded | Sign errors (conf.) | Corrects (conf.) | Sign errors (raw) | Corrects (raw) | Wrote “+” | Wrote “ ” |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma-3-27B | discovery | 142 | 116 | 26 | 67 | 49 | 92 | 50 | 63 | 79 |
| Gemma-3-27B | held-out | 264 | 169 | 95 | 49 | 120 | 126 | 138 | 130 | 134 |
| Gemma-3-27B | pooled | 406 | 285 | 121 | 116 | 169 | 218 | 188 | 193 | 213 |
| Mistral-Small-24B | discovery | 173 | 107 | 66 | 36 | 71 | 95 | 78 | 41 | 132 |
| Mistral-Small-24B | held-out | 264 | 170 | 94 | 66 | 104 | 142 | 122 | 50 | 214 |
| Mistral-Small-24B | pooled | 437 | 277 | 160 | 102 | 175 | 237 | 200 | 91 | 346 |
| Model | Phenomenon | Errors: total | artifact | genuine | Corrects: total | artifact | genuine |
|---|---|---|---|---|---|---|---|
| Qwen-2.5-72B | the layer-0 lookup (C.4.3) | 28 | 28 | 0 | 20 | 20 | 0 |
| Gemma-3-27B | the early copying band (C.4.4) | 33 | 17 | 16 | 109 | 34 | 75 |
| Phi-4-14B | none — no shared site (C.4.5) | 6 | 2 | 4 | 25 | 2 | 23 |
| Llama-3.3-70B | none — no shared site (C.4.5) | 9 | 5 | 4 | 3 | 0 | 3 |
| Mistral-Small-24B | none — no shared site (C.4.5) | 5 | 3 | 2 | 1 | 0 | 1 |
| Stage | Count |
|---|---|
| Sign-error cases (experiment-ready) | 51 |
| Logit-lens readout run | 51 |
| Direct logit attribution run | 51 |
| Peak in last quarter of depth ( 0.75, i.e. L45) | 39 |
| Causal battery (ablations, patching, steering) | not run |
| Model | Total | Confirmed | Excluded | Late | Late % | Median rel_depth |
|---|---|---|---|---|---|---|
| Gemma-3-27B | 48 | 48 | 0 | 22 | 46% | 0.082 |
| Phi-4-14B | 48 | 48 | 0 | 48 | 100% | 1.00 |
| Mistral-Small-24B | 48 | 48 | 0 | 48 | 100% | 1.00 |
| Llama-3.3-70B | 48 | 48 | 0 | 48 | 100% | 1.00 |
| Qwen-2.5-72B | 48 | 48 | 0 | 48 | 100% | 1.00 |
| Model | Phase | Total | Conf. | Excl. | Err (conf.) | Corr (conf.) | Err | Corr | Wrote “+” | Wrote “ ” |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma | disc. | 120 | 106 | 14 | 56 | 50 | 70 | 50 | 60 | 60 |
| Gemma | held-out | 223 | 192 | 31 | 45 | 147 | 73 | 150 | 44 | 179 |
| Gemma | pooled | 343 | 298 | 45 | 101 | 197 | 143 | 200 | 104 | 239 |
| Mistral | disc. | 82 | 69 | 13 | 29 | 40 | 39 | 43 | 20 | 62 |
| Mistral | held-out | 226 | 181 | 45 | 31 | 150 | 70 | 156 | 76 | 150 |
| Mistral | pooled | 308 | 250 | 58 | 60 | 190 | 109 | 199 | 96 | 212 |
| Model | Layer | Cases | Det role | IBP role | Det wrote + | Det wrote | IBP wrote + | IBP wrote |
|---|---|---|---|---|---|---|---|---|
| Gemma | L58 | errors | stamper | amplifier | -0.98 [-1.42, -0.51] n = 131 | +2.18 [+1.77, +2.60] n = 87 | +0.66 [+0.08, +1.21] n = 79 | +3.46 [+2.99, +3.93] n = 64 |
| corrects | amplifier | amplifier | +2.30 [+1.63, +2.96] n = 62 | +4.56 [+4.27, +4.83] n = 126 | +2.88 [+2.34, +3.49] n = 25 | +4.90 [+4.73, +5.07] n = 175 | ||
| Qwen | L75 | errors | stamper/ judge | — (a) | -0.72 [-0.88, -0.55] n = 104 | +1.74 [+1.60, +1.89] n = 102 | — (a) | +2.09 [+1.94, +2.23] n = 36 |
| corrects | stamper | — (a) | -1.02 [-1.25, -0.76] n = 29 | +1.72 [+1.66, +1.79] n = 159 | — (a) | +2.09 [+2.07, +2.11] n = 75 | ||
| Phi | L39 | errors | amplifier | amplifier | +1.56 [+0.97, +2.13] n = 30 | +0.97 [+0.68, +1.29] n = 181 | +0.62 [+0.30, +0.91] n = 8 * | +2.67 [+2.38, +2.95] n = 109 |
| corrects | amplifier | amplifier | +2.12 [+1.32, +3.04] n = 23 | +0.96 [+0.66, +1.27] n = 176 | +3.89 [+3.39, +4.38] n = 66 | +3.77 [+3.37, +4.16] n = 130 |
| Model | n | Top head (DLA) | Top MLP layer (DLA) |
|---|---|---|---|
| Determinant | |||
| Gemma | 92 | L60H4 0.00 | L55 +2.89 |
| Llama | 16 | L78H32 0.53 | L79 +3.06 |
| Mistral | 95 | L38H7 +0.43 | L37 +0.53 |
| Phi | 61 | L36H33 +2.89 | L37 +1.72 |
| Qwen | 59 | L77H44 +0.36 | L77 +1.12 |
| Model (split) | Method | n | Heads | Strongest head | Most positive | Strict flips |
|---|---|---|---|---|---|---|
| Determinant | ||||||
| Gemma (held-out) | zero | 126 | 5 | L60H5 0.16 | L60H4 +0.13 | 0 |
| Llama (held-out) | mean | 192 | 12 | H47 0.03 | H41 +0.09 | 0 |
| Mistral (held-out) | zero | 142 | 8 | L38H29 0.15 | L38H30 +0.06 | 0 |
| Phi (disc) | zero | 61 | 8 | L36H33 0.46 | L35H37 +0.03 | 0 |
| Qwen (held-out) | mean | 147 | 12 | H44 0.17 | L75H62 +0.04 | 0 |
| Model | n | Top identified layer | Runner-up | Controls (range) |
|---|---|---|---|---|
| Determinant | ||||
| Gemma | 126 | L55 0.52 | L59 0.38 | +0.04 to +0.15 |
| Llama | 192 | L79 1.79 | L78 1.39 | +0.05 to +0.30 |
| Mistral | 142 | L37 0.15 | L38 0.08 | 0.20 to +0.18 |
| Phi | 150 | L37 0.21 | L39 0.17 | 0.10 to +0.12 |
| Qwen | 147 | L78 0.46 | L75 0.42 | 0.03 to +0.02 |
| Model | Disc (n) | Disc null mean [range] | Held (n) | Held , baseline +2 (n) | Held beats null | Held null mean [range] | Held flips (rand.) |
|---|---|---|---|---|---|---|---|
| Determinant | |||||||
| Gemma | 3.62 (92) | 0.24 [ 1.45, +0.76] | 1.24 (126) | 3.15 (49) | 33/126 | +0.49 [ 0.36, +1.31] | 22 (5.5) |
| Llama | 4.87 (16) | 0.62 [ 2.36, +0.43] | 4.25 (192) | 5.61 (138) | 91/192 | 0.67 [ 1.42, +0.16] | 41 (11.5) |
| Mistral | 0.69 (93) | 0.36 [ 0.81, +0.11] | 0.32 (142) | 2.73 (66) | 44/142 | 0.23 [ 0.57, +0.16] | 11 (4.0) |
| Phi | +0.43 (61) | 0.90 [ 2.12, 0.12] | 0.19 (150) | 0.27 (143) | 7/150 | 2.70 [ 4.47, 1.27] | 2 (3.4) |
| Qwen | 2.08 (59) | 0.46 [ 1.55, +0.13] | 2.52 (147) | 3.53 (94) | 82/147 | 0.42 [ 1.47, +0.17] | 24 (2.1) |
| Model | Disc (n) | Disc null mean [range] | Held (n) | Held , baseline +2 (n) | Held beats null | Held null mean [range] | Held flips (rand.) |
|---|---|---|---|---|---|---|---|
| Determinant | |||||||
| Gemma | 4.82 (50) | 0.11 [ 0.60, +0.42] | 2.63 (138) | 2.64 (120) | 57/138 | 1.03 [ 1.78, +0.29] | 15 (7.0) |
| Llama | 6.91 (80) | 0.72 [ 2.29, +0.10] | — | — | — | — | — |
| Mistral | 3.40 (78) | 1.08 [ 1.61, 0.43] | 3.02 (122) | 3.97 (104) | 63/122 | 1.21 [ 1.65, 0.48] | 16 (2.7) |
| Phi | 0.74 (50) | 1.61 [ 3.44, 0.32] | +0.99 (149) | +0.93 (139) | 3/149 | 0.72 [ 2.02, 0.08] | 1 (2.6) |
| Qwen | — | — | — | — | — | — | — |
| Model / task (split) | n | H1 pushers | H2 +corrector | H3 correctors | H4 control |
|---|---|---|---|---|---|
| Gemma det (disc) | 92 | 0.64 (1) | 0.24 (2) | +0.75 (1) | 0.00 (0) |
| Gemma det (held-out) | 126 | 0.16 (3) | 0.03 (2) | +0.20 (2) | +0.01 (0) |
| Gemma IBP (disc) | 70 | 1.64 (3) | 0.54 (3) | +1.85 (1) | 0.01 (0) |
| Gemma IBP (held-out) | 73 | 0.80 (5) | 0.41 (4) | +0.83 (2) | 0.02 (0) |
| Mistral det (disc) | 93 | 0.29 (1) | 0.22 (0) | +0.11 (0) | +0.01 (1) |
| Model | det lead (n) | IBP lead (n) |
|---|---|---|
| Gemma-3-27B | 0.12 (126) | 0.52 (72) |
| Mistral-Small-24B | 0.19 (132) | 0.96 (67) |
| Phi-4-14B | 6.70 (146) | 7.11 (76) |
| Llama-3.3-70B | 3.09 (189) | 8.16 (20) |
| Qwen-2.5-72B | 2.84 (99) | 4.94 (25) |
| Model | Phases | n discovery | n held-out | n used | Doses |
|---|---|---|---|---|---|
| Llama-3.3-70B | pooled | 15 | 189 | 204 | 0.5, 1, 2 |
| Qwen-2.5-72B | pooled | 44 | 99 | 143 | 0.5, 1, 2 |
| Gemma-3-27B | pooled | 89 | 126 | 215 | 0.5, 1, 2 |
| Phi-4-14B | discovery only | 57 | — | 57 | 5, 8, 12 |
| Mistral-Small-24B | pooled | 91 | 132 | 223 | 0.5, 1, 2 |
| Model | Layer | at = 0.5 | at = 1 | at = 2 | Flips ( = 0.5 / 1 / 2) |
|---|---|---|---|---|---|
| Llama-3.3-70B (n = 204) | L79 (writer) | 0.46 | 0.91 | 1.83 | 5 / 13 / 32 |
| L77 (writer) | 0.21 | 0.44 | 0.90 | 2 / 2 / 6 | |
| L30 (control) | +0.02 | +0.00 | +0.05 | 1 / 1 / 1 | |
| L50 (control) | 0.01 | 0.01 | 0.02 | 1 / 2 / 2 | |
| random dir. | — | +0.000 | — | 0 | |
| Qwen-2.5-72B (n = 143) | L75 (writer) | 0.04 | 0.07 | 0.13 | 0 / 0 / 0 |
| Model | Layer | at low | at mid | at high | Flips (low / mid / high) |
|---|---|---|---|---|---|
| Gemma-3-27B (n = 215; = 0.5/1/2) | L58 (writer) | 0.031 | 0.059 | 0.145 | 0 / 0 / 3 |
| L60 (writer) | 0.143 | 0.281 | 0.560 | 2 / 6 / 18 | |
| L23 (control) | +0.002 | 0.001 | 0.001 | 2 / 1 / 3 | |
| L39 (control) | 0.013 | 0.013 | +0.015 | 1 / 0 / 2 | |
| Phi-4-14B (n = 57, discovery; = 5/8/12) | L37 (writer) | 5.604 | 9.501 | 15.008 | 0 / 3 / 12 |
| L39 (writer) | 4.543 | 7.350 | 10.991 | 0 / 8 / 10 |
| Model | Phase | Error — minus-written: flipped (mean lead) | Error — plus-written: flipped | Correct — minus: broken | Correct — plus: broken |
|---|---|---|---|---|---|
| Gemma-3-27B | discovery | 16/43 ( 6.94) | 1/46 | 4/33 | 0/17 |
| held-out | 17/41 ( 5.37) | 1/85 | 70/93 | 6/45 | |
| pooled | 33/84 ( 6.17) | 2/131 | 74/126 | 6/62 | |
| Qwen-2.5-72B | discovery | 5/14 ( 7.30) | 0/30 | 17/36 | 0/11 |
| held-out | 6/25 ( 8.14) | 0/74 | 49/123 | 0/18 | |
| pooled | 11/39 ( 7.84) | 0/104 | 66/159 | 0/29 |
| Model | Phase | Error — minus-written: flipped (mean lead) | Error — plus-written: flipped | Correct — minus: broken | Correct — plus: broken |
|---|---|---|---|---|---|
| Gemma-3-27B | discovery | 18/33 ( 8.19) | 0/35 | 1/25 | 0/25 |
| held-out | 15/28 ( 3.96) | 0/44 | 63/150 | 0 cases | |
| pooled | 33/61 ( 6.25) | 0/79 | 64/175 | 0/25 | |
| Qwen-2.5-72B | discovery | 3/10 ( 7.95) | 0 cases | — | — |
| held-out | 10/25 ( 7.20) | 0 cases | 0/50 | 0 cases | |
| pooled | 13/35 ( 7.41) | 0 cases | 0/50 | 0 cases |
| Model | Freeze date (commit) | What else to know |
|---|---|---|
| Llama-3.3-70B | det: 2026-08-31 (03e5549); IBP: 2026-09-01 (fb98035) | The separate IBP document (12 bars) was adjudicated on its own: 11 of 12 met, CONFIRMED (c418503). Table I.5 and Figure 6 score the det document; its P6 steering arm is labeled exploratory †. |
| Qwen-2.5-72B | 2026-08-31 (64580f2) | Held-out results committed 2026-09-03 (412976b) and 2026-09-05 (3bb48f8). |
| Gemma-3-27B | 2026-09-13 (df73a1a; amended 8d56bb3) | Same-day amendment, before any measurement: one pass bar was withdrawn (the correct-answer battery became descriptive), so P5 carries no verdict. |
| Phi-4-14B | 2026-09-15 (67d0d73) | The correct-answer battery is descriptive by design (no pass bar). |
| Mistral-Small-24B | 2026-09-14 (04c2cd2) | The file header still reads “DRAFT”; the Sep 14 commit is the freeze. Held-out run 2026-09-19 (7e7ea87). |
| Pred. | Preregistered bar (held-out) | Measured value | Verdict |
|---|---|---|---|
| P1 | Decided late: lens peak L55 in 50% of cases, per domain | det 54/126 = 42.9%; IBP 29/73 = 39.7% | NEAR-MISS |
| P2 | Mean MLP DLA > 0 at L55 and L58, both domains | L55 +0.913 (pass); L58 0.353 (fail) | NEAR-MISS |
| P3 | Minus-written mean projection < plus-written at the capture layers | ordering inverted at L58/L60; only L56 in the registered direction | NEAR-MISS |
| P4 | Joint ablation dose–response: pooled mean lead 1.5 and 30% of cases beat the null | mean 1.235; 33/126 = 26.2% beat the null | NEAR-MISS |
| P5 | No pass bar (descriptive by pre-measurement amendment) | reported descriptively only | NOT SCORED |
| P6 | Steering flips errors, control flat: mean lead negative and monotone; strict flips at = 3 5%; |control| 0.5 | 18/66 = 27.3% flips at = 3; mean 3.33; control 0 ¶ | CONFIRMED |
| Pred. | Preregistered bar (held-out) | Measured value | Verdict |
|---|---|---|---|
| P1 | Decided late: lens peak L30 in 40% det and 60% IBP | det 74/142 = 52.1% (pass); IBP 39/70 = 55.7% vs the 60% bar | NEAR-MISS (IBP) |
| P2 | DLA > 0 at L37 and < 0 at L36 (det); > 0 at L37 and L38 (IBP) | det L37 +0.893, L36 0.236; IBP L37 +0.993, L38 +0.611 | CONFIRMED |
| P3 § | Minus-written mean projection > plus-written at L37 (minus>plus convention §) | minus +0.673, plus 0.467 | CONFIRMED |
| P4 | Joint ablation: pooled mean 0.30; +2 stratum 1.5; 20% beat the null | mean 0.317; +2 stratum (n = 66) 2.734; 44/142 = 31.0% beat the null | CONFIRMED |
| P5 | Corrects broken at = 3 in 40% of baseline>0 cases; mean lead negative, monotone | 79/107 = 73.8% broken; mean 11.18 | CONFIRMED |
| P6 | Error flips at = 3 20% (det) / 40% (IBP); |control| 0.1 | det 46/80 = 57.5% flips; mean 5.54; control 0 ¶ | CONFIRMED |
| Pred. | Preregistered bar (held-out) | Measured value | Verdict |
|---|---|---|---|
| P1 ‡ | Decided late: lens peak L30 in 80% det and 95% IBP | det 142/150 = 94.7% (pass); IBP 71/77 = 92.2% ‡ | CONFIRMED ‡ |
| P2 | DLA > 0 at L37 and L39, both above controls L15/L25 | L37 +1.665, L39 +1.156; controls +0.038, +0.105 | CONFIRMED |
| P3 | Single-site ablation at L37/L39 more negative than at the controls | L37 0.208, L39 0.175 vs L15 +0.117, L25 0.103 | CONFIRMED |
| P4 | Joint ablation null (the unsteerable prediction): mean 1.0; < 50% beat the null | mean 0.192; 7/150 = 4.7% beat the null | CONFIRMED |
| P5 | Direction subtraction near-zero flips: 10% at = 3; random control 0 flips | L37 0/146 = 0.0%; L39 4/146 = 2.7% | CONFIRMED |
| P6 | Steering flips 10%; |control| 0.2 with 0 strict flips | 6/145 = 4.1% flips at = 3; mean 8.039; control 0 | CONFIRMED |
| Pred. | Preregistered bar (held-out) | Measured value | Verdict |
|---|---|---|---|
| P1 | Decided late: lens peak L60 in 60% of det cases | 152/192 = 79.2% | CONFIRMED |
| P2 | Pooled mean MLP DLA > 0 at L77 and L79 | L77 +1.067, L79 +1.717 | CONFIRMED |
| P3 | Pairwise separation 70% at L79 | 80.1% of case pairs (6,753/8,432); plus-written projection < 0 in 61/68 = 89.7%; means +0.019 (minus) vs 0.791 (plus) | CONFIRMED |
| P4 | Direction subtraction: mean lead negative at L77 and L79 at every , monotone | L77 0.437 / 0.891 and L79 0.860 / 1.723 at = 1/2; controls L30/L50 0 | CONFIRMED |
| P5 | Reverse patching on corrects: mean lead negative at both layers; broken 5% | L77 1.530, L79 3.093; broken 0/130 | CONFIRMED |
| P6 † | Exploratory †: mean lead negative, monotone; control 0 | flips 14/32/71 at = 1/2/3 (71/163 = 43.6%); mean 7.15; control 0 ¶ | CONFIRMED † |
| Pred. | Preregistered bar (held-out) | Measured value | Verdict |
|---|---|---|---|
| P1 | Decided late: lens peak L60 in 60% of det cases (all held-out errors) | 90/147 = 61.2% | CONFIRMED |
| P2 | Mean MLP DLA > 0 at L75 and L78 | L75 +0.472, L78 +0.568 | CONFIRMED |
| P3 | Minus-written mean projection < plus-written at L75 and L78 | ordering reversed at L75/L78; the separation holds only at L79 (77.9% of case pairs), not at the preregistered layers | MISS |
| P4 | Direction subtraction: L78 mean lead negative and monotone at every | minus-written: 0.773 / 1.605 at = 0.5/1; monotone; random control 0 | CONFIRMED |
| P5 | Judge gap at L79: minus-correct > 0 and plus-correct < 0 in the majority; broken 3% | minus-correct 123/123 positive (mean +1.606); plus-correct 18/18 negative (mean 1.135); broken 1/123 | CONFIRMED |
| P6 | Steering: mean lead negative at every , monotone; control 0 | minus-written baseline>0 (n = 25): 6/25 = 24.0% flips at = 3; mean 8.143; control 0 ¶ | CONFIRMED |
| Model | Registered instrument | Frozen pass bar (verbatim) | Held-out result (frozen) | Verdict |
|---|---|---|---|---|
| Mistral | C2 subtraction on corrects at [L38, L39], grid | Corrects broken at = 3 in 40% of baseline>0 cases; mean lead negative, monotone | 79/107 = 73.8% broken; mean 11.18 | CONFIRMED |
| Llama | s07 error-mean patch into corrects at habit layers | Reverse patching on corrects: mean lead negative at both layers; broken 5% | L77 1.530, L79 3.093; broken 0/130 | CONFIRMED |
| Qwen | Judge arm: error-mean vs self-mean patch at check layer L79 | Judge gap at L79: minus-correct > 0 and plus-correct < 0 in the majority; broken 3% | minus-correct 123/123 positive (mean +1.606); plus-correct 18/18 negative (mean 1.135); broken 1/123 | CONFIRMED |
| Phi | Direction subtraction, single layers L37 / L39 (inverse bet) | Direction subtraction near-zero flips: 10% at = 3; random control 0 flips | L37 0/146 = 0.0%; L39 4/146 = 2.7% | CONFIRMED |
| Gemma | — | No pass bar (descriptive by pre-measurement amendment) | reported descriptively only | NOT SCORED |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 833 papers