Prior work established that a probe can decode an in-context binding on model errors and that probe-guided steering can repair some of them. We follow probe accuracy, model output, and steering response across public pretraining and post-training checkpoints. Probe accuracy rises during Pythia pretraining, while probe-guided steering moves from negligible all-trial benefit to a larger benefit at two model sizes. Saved scores distinguish probe-correct errors with low and above-uniform model probability for the correct candidate. Oracle-target steering already repairs many early errors, but saved aggregates cannot separate target quality from intervention sensitivity. A held-out comparison of decoders trained on the final state or candidate logits finds no detected final-state advantage on late-checkpoint model errors. An information-theoretic counterexample explains why decodability on errors alone cannot establish discarded output information. The connection to downstream omissions remains open.
Figures & tables
Figure 1: A downstream motivation and a controlled binding result. (a) Failing repository-fix attempts find gold files nearly as well as passing ones (recall 0.68 vs 0.81 ) but edit far fewer ( 0.55 vs 0.74 ); 27% of failures open a required file and never edit it (DeepSWE-v11, 11,583 attempts). These corpus statistics come from a transcribed summary; the raw source is not in the release. (b) Probed on controlled binding, a separate probe often decodes the correct obligation on trials with an incorrect model output, following the protocol of Ravulapalli et al. (2026) ; gpt-oss-20B has pAcc∣wrong=0.73 (model accuracy on these trials is 0 by construction). The repository episodes and binding trials are separate experiments; this figure does not establish a causal link between them.
Pythia-1.4B step
Wrong trials
Δ log loss
Bootstrap 95% interval
Accuracy: state / logits
1,000
232
−1.811
[−2.053,−1.564]
0.047/0.026
143,000
193
−0.007
[−0.179,0.162]
0.140/0.181
Table 1: Held-out final-state versus candidate-logit decoding on native model errors. Δ is candidate-logit-decoder log loss minus final-state-decoder log loss. The interval resamples test episodes and does not cover training seeds or prompt templates. One model seed and one held-out test template were evaluated.
Step
Errors
P+S+
P+S−
P−S+
P−S−
1,000
330
11
64
34
221
8,000
321
10
64
37
210
32,000
306
22
134
24
126
143,000
272
78
147
7
40
Table 2: Operational error groups in saved Pythia-1.4B posterior batteries. Each row contains 400 trials with K=6 present candidates. P+ means probe top-one correct; S+ means model probability of the correct candidate at least 1/K . Superscript minus denotes the complementary test. The four groups partition native model errors. Probe layer is selected separately for each checkpoint; prompt identities are not saved.
Figure 2: Fixed-middle-layer probe geometry (Pythia-1.4B). (a) Full-state binding probe accuracy rises ( ρ=+1.000 , exact p=4.0×10−4 ). (b) Alignment of the gradient-derived subspace with binding class means has no detected trend ( ρ=+0.429 , p=0.35 ). (c) Gradient-subspace probe accuracy minus class-mean-subspace probe accuracy falls ( ρ=−0.893 , exact p=0.01 ). The gradient-derived subspace comes from a gold-label output margin through the remaining network. Neither projection isolates the model’s natural readout.
Scale
n
Full-state ρ
Full − gradient ρ
Gradient − class mean ρ
160M
9
−0.24
+0.98†
−0.86†
410M
10
+0.47
+0.83†
−0.81†
1B
14
+0.89†
+0.86†
−0.90†
1.4B
7
+1.000†
+0.964†
−0.893†
6.9B
7
+0.96†
+0.95†
+0.36
Table 3: Pythia fixed-middle-layer probe geometry. Per-scale Spearman ρ vs pretraining step for full-state probe accuracy, full-state minus gradient-subspace probe accuracy, and gradient-subspace minus class-mean-subspace probe accuracy; † marks exact permutation p<0.05 . Full-state accuracy clears 1/K=0.167 at 1B/1.4B/6.9B. The third contrast has an opposite point-estimate trend at 6.9B and is not significant there. These probe correlations do not test the final output map. Pythia- 2.8B is censored (duplicate weights). n counts weight-distinct checkpoints; the 6.9B column’s small n limits its power.
Figure 3: Probe accuracy by model size. Pythia fixed-middle-layer and OLMo recorded probe accuracy for the queried binding against pretraining step (log axis), all scales of a family overlaid; the dashed line is the 1/K present-set baseline. Scales at and above 1B separate from the near-chance 160M/410M curves, whose paths end at or barely above the baseline; within the ≥1 B group, final probe accuracy is similar (0.65–0.71). On OLMo-2 the 1B and 13B sweeps clear the baseline; the 7B sweep is truncated at 34B tokens. The OLMo probe generator is unavailable in the release. Line identity is carried by direct labels and dash patterns.
Model
Step
Failures
Oracle target
Probe target
Random
Pythia-1.4B
1,000
324
9
3
5
8,000
298
239
56
12
143,000
271
241
174
21
Pythia-6.9B
1,000
221
29
18
5
8,000
214
98
35
7
143,000
182
159
100
15
Table 4: Repair of native failures at matched intervention strength. Counts are repaired failures over the common failure set within each saved steering file. Oracle and probe arms use the same probe-weight direction construction with the gold or decoded target, respectively; the random direction is norm matched. Each arm uses α=0.5 . Layers differ across checkpoints.
Figure 4: Output, steering, and probe compatibility during code-reward training. Output accuracy falls across the sampled steps; probe-guided steering changes output accuracy. A probe trained at the base checkpoint loses more accuracy than probes refitted at each step. The comparison shows probe staleness, not a change in the model’s natural readout.
Figure 5: Post-training checkpoint comparisons. (a) OLMo-2-7B across post-training: failures fall and probe decodability jumps at SFT, then plateau. (b) per-step Olmo-3-7B-RL-Zero-Code : recall at K=10 falls across 2,400 RL steps; a probe on a held-out fold shows no trend (Figure 6 ). One family/seed per panel; n=400 in (a) and n=300 in (b).
Substrate
Probe decode
Verdict
Executed MBPP/HumanEval
2.8 – 3.7×1/∣V∣ ; 0.9 – 1.2×1/K
clears vocab, straddles 1/K ( 4/4 )
Boxes entity-tracking
declaration 0.54 – 0.87 vs. tracked 0.00 – 0.29
declaration-confounded
CRUXEval-O outputs
input 12.7 – 16.0× vs. output 2.7 – 5.1×
pre-registered kill ( 0/4 )
Table 5: Computed-state substrates: what the probe reads, and why the protocol is power-limited there. Executed code straddles the present-set 1/K ; entity-tracking decodes the declaration, not the tracked state; the pre-registered CRUXEval-O test is censored by model accuracy. A surface-literal control names the shared confound. Decodes are in units of the stated chance baseline unless noted.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 6: RL supplement (companion to Figure 4 ). Left : across sampled GRPO steps, the refitted probe shows no detected trend. Right : probe-guided steering recovers a portion of the measured output-accuracy loss. This intervention does not identify the cause of that loss.
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
A wrong answer does not show whether the model lacked the needed information or held it and failed to use it. On an entity-obligation binding task, a language model can emit an incorrect prompt-supplied binding while a linear probe can recover the correct one from its frozen hidden state. We measure how often this occurs across 16 public checkpoints, each evaluated with three seeds. We fit a probe on a training fold, select its layer on a validation fold, and report results on a disjoint test fold. On the trials each model gets wrong, probe accuracy exceeds the strict present-obligation baseline, 1/K = 0.125, by +0.196 (95% CI [+0.101, +0.296], bootstrapped over models). A query-entity counterfactual rules out token presence and recency. A score built from the sign of probe-output disagreement improves failure detection over the model's own confidence by +0.079 AUROC (95% CI [+0.036, +0.126]). Raw probe confidence gives no measurable improvement over model confidence. Steering the residual stream toward the probe-decoded binding, with no gold label, raises accuracy on all eight models tested by a mean of +0.168 (95% CI [+0.066, +0.280]). Where recent studies report that probe-detected errors are resistant to interventions, we find that in-context binding is a setting in which probes are actionable.
Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. Hari
Can linearly decodable failure signals in LLM hidden states be leveraged to correct those failures? We investigate this classification-correction gap via Overthinking (OT)--a stable behavioral regime (Jaccard >= 0.81, 94% inter-annotator agreement) in medical QA where models answer correctly under resampling yet fail in extended chain-of-thought. OT is linearly decodable at 71.6% balanced accuracy (p < 10^{-16}). Yet five families of fixed linear steering (29 configurations, n=1,273) all yield Delta ~= 0, with identical null results cross-architecture (Qwen2.5-7B) and cross-domain (MMLU-STEM). Three convergent lines of evidence suggest representational entanglement: the OT direction has 85-88% overlap with task-critical computation (specificity ratio <= 0.152); non-targeted shared-direction steering damages accuracy (-12.1pp); and LEACE concept erasure damages accuracy (-3.6pp, p=0.01), while 10 random erasures produce Delta=+0.3pp. The per-instance probe-steering correlation is r=-0.002 (p=0.97). Positively, the same probe enables selective abstention (held-out AUROC=0.610, exceeding all five uncertainty baselines, p=0.009): decodable failure structure supports post-generation reliability estimation even when the fixed linear steering family cannot exploit it for correction.
Probing is widely used to study which features can be decoded from language model representations. However, the common decoding probe approach has two limitations that we aim to solve with our new encoding probe approach: contributions of different features to model representations cannot be directly compared, and feature correlations can affect probing results. We present an Encoding Probe that reverses this direction and reconstructs internal representations of models using interpretable features. We evaluate this method on text and speech transformer models, using feature sets spanning acoustics, phonetics, syntax, lexicon, and speaker identity. Our results suggest that speaker-related effects vary strongly across different training objectives and datasets, while syntactic and lexical features contribute independently to reconstruction. These results show that the Encoding Probe provides a complementary perspective on interpreting model representations beyond decodability.