Decodable In-Context State and Model Output Across Training

Manas Venkata Sai Ravulapalli; Samrath Singh Chadha

cs.LGSep 25, 2026

Decodable In-Context State and Model Output Across Training

Authors: Manas Venkata Sai Ravulapalli, Samrath Singh Chadha

Organizations: Efficient Computation Inc.

Abstract

Prior work established that a probe can decode an in-context binding on model errors and that probe-guided steering can repair some of them. We follow probe accuracy, model output, and steering response across public pretraining and post-training checkpoints. Probe accuracy rises during Pythia pretraining, while probe-guided steering moves from negligible all-trial benefit to a larger benefit at two model sizes. Saved scores distinguish probe-correct errors with low and above-uniform model probability for the correct candidate. Oracle-target steering already repairs many early errors, but saved aggregates cannot separate target quality from intervention sensitivity. A held-out comparison of decoders trained on the final state or candidate logits finds no detected final-state advantage on late-checkpoint model errors. An information-theoretic counterexample explains why decodability on errors alone cannot establish discarded output information. The connection to downstream omissions remains open.

Figures & tables

Figure 1: A downstream motivation and a controlled binding result. (a) Failing repository-fix attempts find gold files nearly as well as passing ones (recall

0.68

0.81

) but edit far fewer (

0.55

0.74

);

27\%

of failures open a required file and never edit it (DeepSWE-v11,

11,583

attempts). These corpus statistics come from a transcribed summary; the raw source is not in the release. (b) Probed on controlled binding, a separate probe often decodes the correct obligation on trials with an incorrect model output, following the protocol of Ravulapalli et al. (2026) ; gpt-oss-20B has

\mathrm{pAcc}\!\mid\!\mathrm{wrong}=0.73

(model accuracy on these trials is

0

by construction). The repository episodes and binding trials are separate experiments; this figure does not establish a causal link between them.

Pythia-1.4B step	Wrong trials	$\Delta$ log loss	Bootstrap 95% interval	Accuracy: state / logits
1,000	232	$-1.811$	$[-2.053,-1.564]$	$0.047/0.026$
143,000	193	$-0.007$	$[-0.179,0.162]$	$0.140/0.181$

Table 1: Held-out final-state versus candidate-logit decoding on native model errors.

\Delta

is candidate-logit-decoder log loss minus final-state-decoder log loss. The interval resamples test episodes and does not cover training seeds or prompt templates. One model seed and one held-out test template were evaluated.

Step	Errors	$P^{+}S^{+}$	$P^{+}S^{-}$	$P^{-}S^{+}$	$P^{-}S^{-}$
1,000	330	11	64	34	221
8,000	321	10	64	37	210
32,000	306	22	134	24	126
143,000	272	78	147	7	40

Table 2: Operational error groups in saved Pythia-1.4B posterior batteries. Each row contains

400

trials with

K=6

present candidates.

P^{+}

means probe top-one correct;

S^{+}

means model probability of the correct candidate at least

1/K

. Superscript minus denotes the complementary test. The four groups partition native model errors. Probe layer is selected separately for each checkpoint; prompt identities are not saved.

Figure 2: Fixed-middle-layer probe geometry (Pythia-1.4B). (a) Full-state binding probe accuracy rises (

\rho=+1.000

, exact

p=4.0\times 10^{-4}

). (b) Alignment of the gradient-derived subspace with binding class means has no detected trend (

\rho=+0.429

p=0.35

). (c) Gradient-subspace probe accuracy minus class-mean-subspace probe accuracy falls (

\rho=-0.893

, exact

p=0.01

). The gradient-derived subspace comes from a gold-label output margin through the remaining network. Neither projection isolates the model’s natural readout.

Scale	$n$	Full-state $\rho$	Full $-$ gradient $\rho$	Gradient $-$ class mean $\rho$
160M	$9$	$-0.24$	$+0.98\,\dagger$	$-0.86\,\dagger$
410M	$10$	$+0.47$	$+0.83\,\dagger$	$-0.81\,\dagger$
1B	$14$	$+0.89\,\dagger$	$+0.86\,\dagger$	$-0.90\,\dagger$
1.4B	$7$	$+1.000\,\dagger$	$+0.964\,\dagger$	$-0.893\,\dagger$
6.9B	$7$	$+0.96\,\dagger$	$+0.95\,\dagger$	$+0.36$

Table 3: Pythia fixed-middle-layer probe geometry. Per-scale Spearman

\rho

vs pretraining step for full-state probe accuracy, full-state minus gradient-subspace probe accuracy, and gradient-subspace minus class-mean-subspace probe accuracy;

\dagger

marks exact permutation

p<0.05

. Full-state accuracy clears

1/K=0.167

at 1B/1.4B/6.9B. The third contrast has an opposite point-estimate trend at 6.9B and is not significant there. These probe correlations do not test the final output map. Pythia-

2.8B

is censored (duplicate weights).

n

counts weight-distinct checkpoints; the 6.9B column’s small

n

limits its power.

Figure 3: Probe accuracy by model size. Pythia fixed-middle-layer and OLMo recorded probe accuracy for the queried binding against pretraining step (log axis), all scales of a family overlaid; the dashed line is the

1/K

present-set baseline. Scales at and above 1B separate from the near-chance 160M/410M curves, whose paths end at or barely above the baseline; within the

{\geq}1

B group, final probe accuracy is similar (0.65–0.71). On OLMo-2 the 1B and 13B sweeps clear the baseline; the 7B sweep is truncated at 34B tokens. The OLMo probe generator is unavailable in the release. Line identity is carried by direct labels and dash patterns.

Model	Step	Failures	Oracle target	Probe target	Random
Pythia-1.4B	1,000	324	9	3	5
	8,000	298	239	56	12
	143,000	271	241	174	21
Pythia-6.9B	1,000	221	29	18	5
	8,000	214	98	35	7
	143,000	182	159	100	15

Table 4: Repair of native failures at matched intervention strength. Counts are repaired failures over the common failure set within each saved steering file. Oracle and probe arms use the same probe-weight direction construction with the gold or decoded target, respectively; the random direction is norm matched. Each arm uses

\alpha=0.5

. Layers differ across checkpoints.

Figure 4: Output, steering, and probe compatibility during code-reward training. Output accuracy falls across the sampled steps; probe-guided steering changes output accuracy. A probe trained at the base checkpoint loses more accuracy than probes refitted at each step. The comparison shows probe staleness, not a change in the model’s natural readout.

Figure 5: Post-training checkpoint comparisons. (a) OLMo-2-7B across post-training: failures fall and probe decodability jumps at SFT, then plateau. (b) per-step Olmo-3-7B-RL-Zero-Code : recall at

K{=}10

falls across

2,400

RL steps; a probe on a held-out fold shows no trend (Figure 6 ). One family/seed per panel;

n=400

in (a) and

n=300

in (b).

Substrate	Probe decode	Verdict
Executed MBPP/HumanEval	$2.8$ – $3.7\times\,1/\|V\|$ ; $0.9$ – $1.2\times\,1/K$	clears vocab, straddles $1/K$ ( $4/4$ )
Boxes entity-tracking	declaration $0.54$ – $0.87$ vs. tracked $0.00$ – $0.29$	declaration-confounded
CRUXEval-O outputs	input $12.7$ – $16.0\times$ vs. output $2.7$ – $5.1\times$	pre-registered kill ( $0/4$ )

Table 5: Computed-state substrates: what the probe reads, and why the protocol is power-limited there. Executed code straddles the present-set

1/K

; entity-tracking decodes the declaration, not the tracked state; the pre-registered CRUXEval-O test is censored by model accuracy. A surface-literal control names the shared confound. Decodes are in units of the stated chance baseline unless noted.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: RL supplement (companion to Figure 4 ). Left : across sampled GRPO steps, the refitted probe shows no detected trend. Right : probe-guided steering recovers a portion of the measured output-accuracy loss. This intervention does not identify the cause of that loss.

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

Legible Failures: Detecting and Repairing In-Context Binding Errors
Sep 10, 2026Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. HariLanguage Model Error Detection LLM Reliability
Efficient Computation Inc.
Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes
May 7, 2026Ming LiuLanguage Model Steering Medical LLMs
Amazon
Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe
May 1, 2026Gaofei Shen, Martijn Bentum, Tomas O. Lentz +2Transformer Interpretability Representation Probing
Tilburg University · Radboud University

Sep 10, 2026J/K move · Enter open · S save

Legible Failures: Detecting and Repairing In-Context Binding Errors

Manas Venkata Sai Ravulapalli, Samrath Singh Chadha, Abhinav M. Hari

Efficient Computation Inc.

Language Model Error Detection LLM Reliability In-Context Learning Language Model Probing

Open paper AskPDF arXiv

A wrong answer does not show whether the model lacked the needed information or held it and failed to use it. On an entity-obligation binding task, a language model can emit an incorrect prompt-supplied binding while a linear probe can recover the correct one from its frozen hidden state. We measure how often this occurs across 16 public checkpoints, each evaluated with three seeds. We fit a probe on a training fold, select its layer on a validation fold, and report results on a disjoint test fold. On the trials each model gets wrong, probe accuracy exceeds the strict present-obligation baseline, 1/K = 0.125, by +0.196 (95% CI [+0.101, +0.296], bootstrapped over models). A query-entity counterfactual rules out token presence and recency. A score built from the sign of probe-output disagreement improves failure detection over the model's own confidence by +0.079 AUROC (95% CI [+0.036, +0.126]). Raw probe confidence gives no measurable improvement over model confidence. Steering the residual stream toward the probe-decoded binding, with no gold label, raises accuracy on all eight models tested by a mean of +0.168 (95% CI [+0.066, +0.280]). Where recent studies report that probe-detected errors are resistant to interventions, we find that in-context binding is a setting in which probes are actionable.

Abstract

LLM Interpretability

Explore similar work

Legible Failures: Detecting and Repairing In-Context Binding Errors

Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes

Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe