One way to study language-model introspection is to test whether models can identify where their activations have been perturbed. We train a simple linear classifier to identify the perturbed sentence from the model's activations, then compare its predictions with the model's own answers across pretraining and injection strengths. This tests whether the model is being asked to report information that is already easy to recover. If so, successful localization may be less demanding than it first appears and may not extend to the broader uses that motivate introspection research, such as faithfully explaining a model's reasoning or helping researchers assess the reliability of its answers. With an injection four times the typical activation norm at layer 0 and probe readout at layer 24 in OLMo 3 32B, the supervised probe reaches 98.3 % accuracy after 8.4 billion training tokens, while the model's zero-shot answer remains at chance (11.1 %). At the released base checkpoint, these accuracies reach 99.1 % and 89.6 %, respectively. Across 7B and 32B, location is already decodable in randomly initialized networks, showing that the architecture can support these traces without pretraining. In these experiments, location information lies close enough to the surface for a simple linear probe to recover it, so successful reporting need not imply broader access to the model's internal computation.
Figures & tables
Figure 1: The localization test. (a) A sweep prompt with sentence 4 injected. The shaded trailing space is the final input token; its logits score the next digit. (b) The residual stream, one column per token and one row per block; Q marks the question tokens and T the last token (blue, corresponding to the trailing space in (a)). Dark orange: the injected tokens of Sk at block L . Light orange: cells that can be affected through later attention to the injected tokens. Gray: unchanged. The blue box is the probe’s readout, at T in block ℓ .
Figure 2: OLMo 3 32B across pretraining, with injection at layer 0, strength α=4 and probe readout at layer 24 throughout. This fixed-setting analysis was chosen after examining the sweep. Lines show means across three data seeds; bars show one standard deviation. Concept and random directions are pooled. The dashed line marks one-in-nine chance. B and T denote billions and trillions of training tokens; positions are categorical, and the released base includes subsequent training.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Probe and raw answer across stage-1 pretraining and the released base. Each measure independently selects injection layer and strength on two seeds and is scored on the third; the probe also selects readout layer. Lines average the three held-out scores; bars show one standard deviation across seeds, not confidence intervals. Concept and random directions are pooled. B and T denote billions and trillions of training tokens. Positions are categorical; the released base includes subsequent training. Chance is one in nine.
Figure 4: Probe accuracy by injection strength at three checkpoints, with injection at layer 0. The readout layer is chosen on two seeds and scored on the third. Lines are means over three held-out seeds; bars show one standard deviation across seeds, not confidence intervals. Concept and random directions are pooled. Doses double between equally spaced ticks. Chance is one in nine.
Location (%)
Strength
Gap 0
Gap 16
Gap 64
Gap 256
0
11.5
11.3
11.9
10.9
1
12.2
13.1
13.3
11.5
2
16.9
18.1
15.9
13.5
4
27.4
26.9
21.9
19.1
8
31.5
29.8
23.5
21.5
Appendix
Table 1: No-prompt localization in randomly initialized networks at layer 24, averaged over three seeds (chance: one in nine). Gap counts neutral suffix tokens after the list. Each cell uses a separately fitted probe tested on 180 held-out trials per seed. Strength zero is an uninjected control with assigned target labels.
Strength
Gap 0
Gap 256
0
14.3
14.1
1
24.3
13.5
2
50.7
14.1
4
53.0
14.8
8
48.1
14.3
Appendix
Table 2: Pretrained no-prompt localization (%). Strength-zero accuracy is elevated and remains unexplained.
Large language models (LLMs) can sometimes report perturbations to their internal activations---even when the input provides no evidence that an intervention occurred. How do models detect and localize such internal changes? We study this question using a controlled task that keeps the input text fixed. We either inject a concept vector into the hidden state at one of ten token positions or apply no intervention. The model is asked to identify the perturbed position or report that no intervention occurred. Across three model families, we identify two small groups of attention heads with distinct roles in introspective reporting. Middle-layer gate heads influence whether the model reports a change, while router heads in a later layer help select the position to report. Interventions on gate heads can suppress position reports even when router heads supply location information. We further examine why reporting accuracy varies across concepts. Concept vectors that are localized more accurately produce stronger attention-score and output responses in gate heads, which is associated with better alignment of the induced key and value changes in their QK and OV computations. Together, these findings identify attention-head mechanisms supporting introspective detection and localization.
Jiahong Zou, Xiangkun Sun, Lingkai Kong +1
Shandong University · Tsinghua University · Northeastern University +1
Are frontier models able to introspect about their internal states? Recent work suggests that under certain conditions a complex enough model can audit its own internals, call out what changed, and report back confidently about it. We tested that claim on eight open-weight models from seven families and found no such ability: asked whether their own computation had been altered, none answered better than chance. To test it we built Open-Weight Masked Introspection (OWMI), a framework that intervenes on residual-stream sites, attention heads and sparse-autoencoder features, then interrogates the model about the change against the null conditions an answer has to beat: sham runs where nothing was altered, impact-matched random perturbations, and a text-only observer that sees only the visible output. Over 78,000 measurements, no model's report discriminates a real intervention from a sham beyond chance (AUROC ~0.5007), and an equivalence test bounds the effect below 0.15 percentage points of AUROC. Surprisingly, all the information needed is in the models. A model fine-tuned to report this class of intervention reaches near-perfect recovery on held-out directions, and a linear probe recovers intervention presence from the same activations at 75% to 95.8% accuracy, sharpening to no held-out error at the last layer before the model speaks. In one model the signal surfaces in the confidence rather than the words: its yes-or-no report never varies, while the confidence attached to it separates intervention from sham at AUROC 0.647. The failure sits in the path from internal state to verbal report, so oversight that reads a model's own testimony needs validating against an internal reference. While our results show the inability of current open-weight models to introspect, the debate is not settled for future models.
We uncover a latent capacity for introspection in a Qwen 32B model, demonstrating that the model can detect when concepts have been injected into its earlier context and identify which concept was injected. While the model denies injection in sampled outputs, logit lens analysis reveals clear detection signals in the residual stream, which are attenuated in the final layers. Furthermore, prompting the model with accurate information about AI introspection mechanisms can dramatically strengthen this effect: the sensitivity to injection increases massively (0.3% -> 39.9%) with only a 0.6% increase in false positives. Also, mutual information between nine injected and recovered concepts rises from 0.61 bits to 1.05 bits, ruling out generic noise explanations. Our results demonstrate models can have a surprising capacity for introspection and steering awareness that is easy to overlook, with consequences for latent reasoning and safety.
Theia Pearson-Vogel, Martin Vanek, Raymond Douglas +1