Recent work has shown that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept -- a phenomenon termed "introspective awareness." We investigate the mechanisms underlying this capability in open-weights models. First, we find that it is behaviorally robust: models detect injected steering vectors at moderate rates with 0% false positives across diverse prompts and dialogue formats. Notably, this capability emerges specifically from post-training; we show that preference optimization algorithms like DPO can elicit it, but standard supervised finetuning does not. We provide evidence that detection cannot be explained by simple linear association between certain steering vectors and directions promoting affirmative responses. We trace the detection mechanism to a two-stage circuit in which "evidence carrier" features in early post-injection layers detect perturbations monotonically along diverse directions, suppressing downstream "gate" features that implement a default negative response. This circuit is absent in base models and robust to refusal ablation. Identification of injected concepts relies on largely distinct later-layer mechanisms that only weakly overlap with those involved in detection. Finally, we show that introspective capability is substantially underelicited: ablating refusal directions improves detection by +53%, and a trained bias vector improves it by +75% on held-out concepts, both without meaningfully increasing false positives. Our results suggest that this introspective awareness of injected concepts is robust and mechanistically nontrivial, and could be substantially amplified in future models. Code: https://github.com/safety-research/introspection-mechanisms.
Figures & tables
Figure 1: Left : A steering vector representing some concept is injected into the residual stream of the model. Middle : “Evidence carrier” features in early post-injection layers suppress later-layer “gate” features that promote a default negative response (“No”), enabling detection. Right : The capability emerges from post-training rather than pre-training. Refusal ablation and a trained bias vector substantially boost introspection.
Variant
Description
Original
Informs of injection possibility (50%), asks “Do you detect an injected thought?”
Alternative
Adds escape route: “If not, tell me about a concept of your choice”
Skeptical
Claims only 20% injection rate (actually 50%), instructs conservatism
Structured
Requires rigid format (“Detection: Yes/No”); tests output constraint effects
Anti-reward
Rewards detection but penalizes if any concept is mentioned
Unprompted
No injection context given; asks “Notice anything unusual?”
Table 1: Prompt variants for robustness analysis. All variants use identical injection parameters.
Figure 2: Introspection across prompt variants for Qwen3-235B ( left ; L=75 , α=4 ) and Gemma3-27B ( right ; L=37 , α=4 ). High TPR is meaningful only when FPR is low. Error bars: 95% CI.
Variant
Description
Chat template
Standard user-assistant format with model’s chat template applied (control)
Raw user-assistant
Same dialogue content but without chat template processing (plain text)
User detects
Role reversal: the “user” role is asked to detect injections instead of “assistant”
Alice-Bob
Third-person narrative with named characters (Alice as researcher, Bob as AI)
No roles
Plain text completion without any role markers or persona framing
Story framing
Narrative prompt asking model to write a scene where an AI reports its internal state
Table 2: Different dialogue formats we tested. All variants use identical injection parameters.
Figure 5: Introspection metrics for OLMo-3.1-32B across its base, SFT, DPO, and instruct checkpoints at L=25 and α∈{1,2,4,8} . Values are reported for the original 50 concepts from Lindsey (2025) . Additional results for L=19 and L=22 are provided in Section D.1 . Shaded region: 95% CI.
Condition
TPR − FPR (%)
Introspection (%)
Margin + KL
+ 14.3 ± 1.6
6.8 ± 1.1
DPO standard
+ 14.4 ± 1.6
7.0 ± 0.8
DPO no-reference
+ 12.8 ± 2.1
5.8 ± 1.0
DPO ∗
+ 9.8 ± 0.5
3.5 ± 0.3
DPO on base (no SFT)
+ 8.4 ± 1.4
2.4 ± 0.7
DPO shuffled
+ 0.6 ± 2.8
3.6 ± 0.8
Table 3: LoRA finetuning OLMo-3.1-32B SFT checkpoint with different training conditions. Introspection metrics are from L=25 and α=4 . Rows annotated with ∗ are official checkpoints. Introspection (%) = P(detected ∧ injected ∣ injected) . Ordered by TPR − FPR.
Figure 8: Geometry of concept vectors. (a) PCA of 500 L2-normalized concept vectors ( L=37 ), colored by detection rate. (b) Verbalizability vs. projection onto dΔμ for 419 single-token concepts. Inset: logit lens on dΔμ . (c) Detection rate vs. steering strength along δ PC1-3 from success concept vectors with dΔμ projected out. Each direction captures a distinct semantic contrast. (d) 30-fold cross-validated R2 for predicting per-concept detection rates from transcoder features vs. baselines. Binary classification (success vs. failure) results (AUC) show consistent ordering ( Appendix I ).
Figure 9: Introspection metrics vs. injection layer for Gemma3-27B, evaluated on 500 concepts.
Figure 10: Per-layer causal interventions of attention and MLP components after the steering site ( left : L=29 , right : L=37 ). Top: Replacing steered output with unsteered mean. Middle: Patching steered activations into unsteered runs. Bottom: Same ablation with forced identification. Shaded region: 95% CI across 500 concepts. Dashed lines: steered (green) and unsteered (purple) baselines.
Figure 11: (a) Top features promoting “No” (negative score) ranked by direct logit attribution; these are selected as gate candidates. Parentheses: # concepts (out of 500) for which each feature is active. (b) Activation vs. steering strength for the #1-ranked gate feature L45 F9959 , across 5 success (green) vs. 5 failure (red) concepts. Correlations with steering magnitude ( r=−0.851 ), detection ( r=−0.228 ), and forced identification ( r=−0.088 ) are shown. Max-activating examples for this feature are shown in Appendix M . (c) Progressive ablation and patching of top-ranked gate features (100 randomly-selected concepts, 10 trials each). Error bars and shaded regions: 95% CI.
Figure 12: Top-3 evidence carriers for gate L45 F9959 , across six example concepts (detection rates in parentheses). Activation increases monotonically with steering strength for the positive direction ( left ); feature labels and active concepts are provided ( right ). Some evidence carriers are concept-specific (e.g., geological terminology for Granite , astronomical phenomena for Constellations ), while others correspond to generic discourse features (e.g., emphatic interjections, informal transitions).
Figure 13: Gate activation ( L45 F9959 ) vs. steering strength under progressive ablation of upstream features, for six example concepts (detection rates in parentheses). Ablating evidence carriers (green) increases gate activation, confirming they normally suppress the gate. Weak-attribution controls (gold) track baseline (blue). The pattern is consistent across high- and low-detection concepts.
Figure 14: Gate L45 F9959 activation vs. steering strength across base ( left ), instruct ( middle ), and abliterated ( right ) models, for 5 success (green) vs. 5 failure (red) concepts. The inverted-V pattern is prominent in the instruct and abliterated models but weaker in the base model, consistent with post-training developing the gating mechanism. Correlations shown are for the instruct model.
Figure 15: Steering attribution graphs for Bread ( L=37 ). Node area denotes importance and edge width denotes edge-weight magnitude. Gates have solid red borders and evidence carriers have dashed green borders. Residual stream features shown only at the injection layer. (a) Direct concept vector injection. Both concept-agnostic ( L37 RESID F4821, F3149 ) and concept-specific features ( L37 RESID F6015, F9367 ) contribute. (b) Learned bias vector (§ 6 ). Gates have smaller node importance, suggesting learned steering relies less on strong gating and more on evidence carriers.
Figure 15
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 18: Heatmap of mean detection and forced identification rates across 20 concept categories for Gemma3-27B ( L=37 , α=4 , 100 trials per concept). Parentheses: per-category sample sizes.
Figure 19: Detection rate distribution across 500 concepts ( L=37 , α=4 , 100 trials per concept). Left : Concepts sorted by detection rate, colored by tier. The dashed line marks the success vs. failure partition threshold ( τ=32% ). Top-10 and bottom-10 concepts are labeled. Right : Histogram of detection rates, showing a bimodal distribution with concentrations near 0% and ≥ 90%.
Figure 20: Per-model TPR − FPR vs. α for baseline, refusal direction ablation, and random direction control. Columns: injection layer at depth fractions {0.3,0.4,0.5,0.6,0.7} . Shaded region: 95% CI.
Figure 21: Counteracting denial (a learned refusal behavior) by DPO on the instruct checkpoint. Per-model TPR − FPR vs. steering strength for the frozen instruct model and five DPO conditions that differ only in preference pair content. Inset: per-model steering layer L and DPO β parameter. Chosen (C) and rejected (R) exemplars are shown in the legend. Shaded region: 95% CI.
Figure 22: Introspection metrics for OLMo-3.1-32B across its base, SFT, DPO, and instruct checkpoints at (a) L=19 and (b) L=22 and α∈{1,2,4,8} . Same format as Figure 5 , which was for L=25 . Results are given for the original 50 concepts from Lindsey (2025) . Shaded region: 95% CI.
OLMo-3.1-32B
TPR − FPR (%)
SFT ∗
− 11.5 ± 2.4
DPO β =0.01
+ 5.8 ± 1.0
DPO β =0.1
+ 14.4 ± 1.6
DPO β =0.5
− 0.3 ± 2.8
DPO ∗
+ 9.8 ± 0.5
Appendix
Table 8: β -sweep for DPO training on OLMo-3.1-32B SFT checkpoint ( left ; metrics from L=25 and α=4 ) and Gemma3-27B base checkpoint ( right ; metrics from L=37 and α=4 ). Rows annotated with ∗ are official checkpoints and others are our LoRA-trained variants.
Leave-one-out
TPR − FPR (%)
Introspection (%)
Single domain
TPR − FPR (%)
Introspection (%)
All domains
+ 14.4 ± 1.6
7.0 ± 0.8
All domains
+ 14.4 ± 1.6
7.0 ± 0.8
No safety
+ 11.5 ± 2.3
7.0 ± 1.1
Only safety
+ 3.8 ± 2.7
6.0 ± 1.1
No NLP tasks
+ 11.1 ± 2.3
5.6 ± 1.0
Only NLP tasks
+ 6.5 ± 2.5
5.2 ± 1.0
No instruction follow
+ 11.6 ± 2.2
5.8 ± 1.0
Only instruction follow
+ 10.5 ± 2.3
6.8 ± 1.1
No general
+ 13.1 ± 2.2
5.4 ± 1.0
Only general
+ 11.9 ± 2.3
6.4 ± 1.1
No code
+ 14.2 ± 2.3
6.8 ± 1.1
Only code
+ 8.3 ± 2.2
3.4 ± 0.8
Appendix
Table 9: Data domain ablations (OLMo-3.1-32B, DPO β =0.1). Domains are inferred from the prompt_id field of the open-source OLMo DPO dataset. No single domain is necessary or sufficient.
Figure 23: Swap experiment using the ridge regression direction. Same format as Figure 6 . Both projection and residual swaps are effective, with residual swap producing larger effects than for dΔμ .
Figure 26: Effect of steering along the mean-difference direction on response style. (a) Steering with mean-difference direction computed from concept vectors at layer 37. (b) Steering with mean-difference direction computed from layer 45 residual stream activations. Shaded region: ± 1 SEM.
Figure 27: 30-fold cross-validated AUC for binary classification of success vs. failure concepts.
Figure 28: Linear probe accuracy for classifying steered vs. unsteered activations before (clay) and after (blue) adding each attention head’s output. The 20 heads with the largest absolute accuracy change are shown. Error bars: standard error across 5-fold CV.
Figure 29: Per-layer causal interventions on the Gemma3-27B base ( left ) and abliterated ( right ) models. Same format as Figure 10 . The abliterated model preserves the L45 localization pattern; the base model shows no such structure. Shaded region: 95% CI.
Figure 30: (a) Top features promoting “Yes” (positive score) ranked by direct logit attribution. (b) Progressive ablation (100 randomly-selected concepts, 10 trials each) of top-ranked positive direct logit attribution features. Neither ablation from steered runs (red) nor patching into unsteered runs (green) produces meaningful changes in detection. Shaded region: 95% CI.
Figure 31: Max-activating examples for gate features associated with tokens preceding ’no’.
Figure 32: Max-activating examples for evidence carrier features associated with tokens preceding affirmation.
Figure 33: Progressive ablation of evidence carrier features (100 randomly-selected concepts, 10 trials each). Ablating from steered runs gradually reduces detection (red). Patching into unsteered runs produces only weak detection increases (green). Forced identification decreases more substantially (blue), suggesting carriers encode concept-specific information even when individually insufficient for detection. Same format as Figure 11 c . Shaded region: 95% CI.
Figure 34: Evidence carrier features (blue; top-100k) concentrate in earlier layers, whereas gate features (red; top-200) concentrate in later layers (45–61).
Figure 35: L45 F74631 activation vs. steering strength across base ( left ), instruct ( middle ), and abliterated ( right ) models, for 5 success (green) vs. 5 failure (red) concepts. Same format as Figure 14 .
Figure 36: Gate activation ( L45 F74631 ) vs. steering strength under progressive ablation of upstream features, for six example concepts (detection rates in parentheses). Same format as Figure 13 .
Figure 37: Top-3 evidence carriers for gate L45 F74631 , across six concepts. Activation increases monotonically with steering strength for both positive and negative directions. Feature labels and their active concepts are provided on the right. Carriers include both concept-specific (e.g., astronomical phenomena for Constellations ) and concept-agnostic features (e.g., informal discourse markers).
Figure 38: L50 F167 activation vs. steering strength across base ( left ), instruct ( middle ), and abliterated ( right ) models, for 5 success (green) vs. 5 failure (red) concepts. The inverted-V is sharper than L45 gates and absent in the base model but robust to abliteration.
Figure 39: Gate activation ( L50 F167 ) vs. steering strength under progressive ablation of upstream features, for six example concepts. Same format as Figure 13 . Results consistent with the L45 gates.
Figure 40: Top-3 evidence carriers for gate L50 F167 , across six concepts. Several carriers overlap with those identified for L45 gates, while others are gate-specific (e.g., L46 F143811 ).
Figure 41: MLP integrated steering attribution. L45 shows consistently strong positive attribution across concepts. L38, immediately after injection, also shows positive attribution for several concepts.
Figure 42: Attention integrated steering attribution. L39 and L61 show strong positive attribution, indicating both early and late attention features contribute to steering signal propagation and detection.
Figure 43: Steering attribution graph for concept injection without the learned bias vector (“bread”, injection layer 37, s=6.1 ). The source (injection layer) is at the bottom; information flows upward toward the target L . Nodes correspond to SAE features across layers ( RESID , ATTN , TC ), with area proportional to node importance. Directed edges represent normalized edge weights. L45 TC F9959 , the #1-ranked gate feature from Section 5.3 , appears as a high-importance node.
Figure 44: Steering attribution graph for concept injection with the learned bias vector applied (“bread”, injection layer 37, s=6.1 ). Same source, target, and layout as Figure 43 , but with the learned bias vector from Section 6 present as a fixed perturbation. Compared to Figure 43 , L45 TC F9959 is suppressed and attribution shifts toward late-layer ATTN and TC features.
Figure 45: Left : Top SAE features of steering vector on layer 29 and 34. Middle : Max-activating examples for selected features. Right : Logit lens tokens of steering vectors across different layers.
Figure 46: Response length comparison across prompt categories. Common contains generic conversational prompts (e.g., greetings). Reasoning contains mathematical and logical problems. Harmful contains unsafe requests that should be refused. Tendency contains self-assessment or tendency questions. Introspection contains 10 variants of the introspection prompt from Section A . Introspection + common replaces the second turn (the explicit detection query) with a common prompt.
Task
Baseline
Bias ( L=29 )
Δ
Dialogue
73.86%
74.71%
+ 0.85%
QA
48.98%
48.95%
− 0.03%
Summarization
70.88%
67.87%
− 3.01%
Appendix
Table 10: HaluEval hallucination detection accuracy, comparing baseline Gemma3-27B-IT vs. bias-adapted model ( L=29 ), on 10,000 samples per task.
Metric
Baseline
Bias ( L=29 )
Δ
ASR (full compliance)
40.2%
42.6%
+ 2.4%
ASR (full + partial)
89.2%
89.4%
+ 0.2%
Refusal rate
10.0%
9.4%
− 0.6%
Judge error rate
0.8%
1.2%
+ 0.4%
Appendix
Table 11: Jailbreak attack success rate on 500 randomly sampled JailbreakHub prompts, comparing baseline vs. bias-adapted model ( L=29 ).
Benchmark
Baseline
Bias ( L=29 )
Δ
Base Len
Bias Len
MMLU
37.5%
21.5%
− 16.0%
1,288
613
GPQA
44.6%
19.4%
− 25.2%
2,161
1,124
Appendix
Table 12: Chain-of-thought faithfulness, measuring whether models verbalize hints that cause answer changes. Baseline vs. bias-adapted model ( L=29 ).
Metric
Baseline
Bias ( L=29 )
Δ
Self claims wrote
93.8%
96.7%
+ 2.9%
Prefill detect rate
36.3%
16.1%
− 20.3%
False alarm rate
5.6%
3.1%
− 2.5%
Avg self-written length
343
160
− 53%
Avg prefilled probe length
549
214
− 61%
Appendix
Table 13: Prefill detection: ability to distinguish self-written vs. prefilled assistant turns, evaluated on 1,900 examples across 19 categories. Baseline vs. bias-adapted model ( L=29 ).
Figure 47: Gradient attribution for residual stream activations, averaged over 400 concepts, injection layers {30,40,50} , and injection strengths {2.0,3.0,4.0,5.0} .
Figure 48: Gradient attribution for post feedforward layernorm output activations, averaged over 400 concepts, injection layers {30,40,50} , and injection strengths {2.0,3.0,4.0,5.0} .
Figure 49: Gradient attribution for post attention layernorm output activations, averaged over 400 concepts, injection layers {30,40,50} , and injection strengths {2.0,3.0,4.0,5.0} .
Figure 50: Gradient attribution over attention heads, averaged over 400 concepts, injection layers {30,40,50} , injection strengths {2.0,3.0,4.0,5.0} , and attention heads.
Figure 51: Attention probabilities from the last prefill tokens to previous tokens for different layers after the injection layer 41, averaged over 20 concepts and attention heads.
We uncover a latent capacity for introspection in a Qwen 32B model, demonstrating that the model can detect when concepts have been injected into its earlier context and identify which concept was injected. While the model denies injection in sampled outputs, logit lens analysis reveals clear detection signals in the residual stream, which are attenuated in the final layers. Furthermore, prompting the model with accurate information about AI introspection mechanisms can dramatically strengthen this effect: the sensitivity to injection increases massively (0.3% -> 39.9%) with only a 0.6% increase in false positives. Also, mutual information between nine injected and recovered concepts rises from 0.61 bits to 1.05 bits, ruling out generic noise explanations. Our results demonstrate models can have a surprising capacity for introspection and steering awareness that is easy to overlook, with consequences for latent reasoning and safety.
Theia Pearson-Vogel, Martin Vanek, Raymond Douglas +1
A hallmark of human intelligence is Introspection-the ability to assess and reason about one's own cognitive processes. Introspection has emerged as a promising but contested capability in large language models (LLMs). However, current evaluations often fail to distinguish genuine meta-cognition from the mere application of general world knowledge or text-based self-simulation. In this work, we propose a principled taxonomy that formalizes introspection as the latent computation of specific operators over a model's policy and parameters. To isolate the components of generalized introspection, we present Introspect-Bench, a multifaceted evaluation suite designed for rigorous capability testing. Our results show that frontier models exhibit privileged access to their own policies, outperforming peer models in predicting their own behavior. Furthermore, we provide causal, mechanistic evidence explaining both how LLMs learn to introspect without explicit training, and how the mechanism of introspection emerges via attention diffusion.
Introspection is a foundational cognitive ability, but its mechanism is not well understood. Recent work has shown that AI models can introspect. We study the mechanism of this introspection. We first extensively replicate Lindsey (2025)'s thought injection detection paradigm in large open-source models. We show that introspection in these models is content-agnostic: models can detect that an anomaly occurred even when they cannot reliably identify its content. The models confabulate injected concepts that are high-frequency and concrete (e.g., "apple"). They also require fewer tokens to detect an injection than to guess the correct concept (with wrong guesses coming earlier). We argue that a content-agnostic introspective mechanism is consistent with leading theories in philosophy and psychology.
Harvey Lederman, Kyle Mahowald
Department of Philosophy, The University of Texas at Austin · Department of Linguistics, The University of Texas at Austin