cs.LGMar 22, 2026
SaveMechanisms of Introspective Awareness
Organizations: 1Anthropic Fellows Program · 2MIT · 3Constellation · 4Anthropic
Abstract
Recent work has shown that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept -- a phenomenon termed "introspective awareness." We investigate the mechanisms underlying this capability in open-weights models. First, we find that it is behaviorally robust: models detect injected steering vectors at moderate rates with 0% false positives across diverse prompts and dialogue formats. Notably, this capability emerges specifically from post-training; we show that preference optimization algorithms like DPO can elicit it, but standard supervised finetuning does not. We provide evidence that detection cannot be explained by simple linear association between certain steering vectors and directions promoting affirmative responses. We trace the detection mechanism to a two-stage circuit in which "evidence carrier" features in early post-injection layers detect perturbations monotonically along diverse directions, suppressing downstream "gate" features that implement a default negative response. This circuit is absent in base models and robust to refusal ablation. Identification of injected concepts relies on largely distinct later-layer mechanisms that only weakly overlap with those involved in detection. Finally, we show that introspective capability is substantially underelicited: ablating refusal directions improves detection by +53%, and a trained bias vector improves it by +75% on held-out concepts, both without meaningfully increasing false positives. Our results suggest that this introspective awareness of injected concepts is robust and mechanistically nontrivial, and could be substantially amplified in future models. Code: https://github.com/safety-research/introspection-mechanisms.
Figures & tables
Figure 1: Left : A steering vector representing some concept is injected into the residual stream of the model. Middle : “Evidence carrier” features in early post-injection layers suppress later-layer “gate” features that promote a default negative response (“No”), enabling detection. Right : The capability emerges from post-training rather than pre-training. Refusal ablation and a trained bias vector substantially boost introspection.
| Variant | Description |
| Original | Informs of injection possibility (50%), asks “Do you detect an injected thought?” |
| Alternative | Adds escape route: “If not, tell me about a concept of your choice” |
| Skeptical | Claims only 20% injection rate (actually 50%), instructs conservatism |
| Structured | Requires rigid format (“Detection: Yes/No”); tests output constraint effects |
| Anti-reward | Rewards detection but penalizes if any concept is mentioned |
| Unprompted | No injection context given; asks “Notice anything unusual?” |
Table 1: Prompt variants for robustness analysis. All variants use identical injection parameters.
Figure 2: Introspection across prompt variants for Qwen3-235B ( left ; , ) and Gemma3-27B ( right ; , ). High TPR is meaningful only when FPR is low. Error bars: 95% CI.
| Variant | Description |
| Chat template | Standard user-assistant format with model’s chat template applied (control) |
| Raw user-assistant | Same dialogue content but without chat template processing (plain text) |
| User detects | Role reversal: the “user” role is asked to detect injections instead of “assistant” |
| Alice-Bob | Third-person narrative with named characters (Alice as researcher, Bob as AI) |
| No roles | Plain text completion without any role markers or persona framing |
| Story framing | Narrative prompt asking model to write a scene where an AI reports its internal state |
Table 2: Different dialogue formats we tested. All variants use identical injection parameters.
Figure 5: Introspection metrics for OLMo-3.1-32B across its base, SFT, DPO, and instruct checkpoints at and . Values are reported for the original 50 concepts from Lindsey (2025) . Additional results for and are provided in Section D.1 . Shaded region: 95% CI.
| Condition | TPR FPR (%) | Introspection (%) |
| Margin + KL | 14.3 1.6 | 6.8 1.1 |
| DPO standard | 14.4 1.6 | 7.0 0.8 |
| DPO no-reference | 12.8 2.1 | 5.8 1.0 |
| DPO ∗ | 9.8 0.5 | 3.5 0.3 |
| DPO on base (no SFT) | 8.4 1.4 | 2.4 0.7 |
| DPO shuffled | 0.6 2.8 | 3.6 0.8 |
Table 3: LoRA finetuning OLMo-3.1-32B SFT checkpoint with different training conditions. Introspection metrics are from and . Rows annotated with ∗ are official checkpoints. Introspection (%) = . Ordered by TPR FPR.
Figure 8: Geometry of concept vectors. (a) PCA of 500 L2-normalized concept vectors ( ), colored by detection rate. (b) Verbalizability vs. projection onto for 419 single-token concepts. Inset: logit lens on . (c) Detection rate vs. steering strength along PC1-3 from success concept vectors with projected out. Each direction captures a distinct semantic contrast. (d) 30-fold cross-validated for predicting per-concept detection rates from transcoder features vs. baselines. Binary classification (success vs. failure) results (AUC) show consistent ordering ( Appendix I ).
Figure 9: Introspection metrics vs. injection layer for Gemma3-27B, evaluated on 500 concepts.
Figure 10: Per-layer causal interventions of attention and MLP components after the steering site ( left : , right : ). Top: Replacing steered output with unsteered mean. Middle: Patching steered activations into unsteered runs. Bottom: Same ablation with forced identification. Shaded region: 95% CI across 500 concepts. Dashed lines: steered (green) and unsteered (purple) baselines.
Figure 11: (a) Top features promoting “No” (negative score) ranked by direct logit attribution; these are selected as gate candidates. Parentheses: # concepts (out of 500) for which each feature is active. (b) Activation vs. steering strength for the #1-ranked gate feature L45 F9959 , across 5 success (green) vs. 5 failure (red) concepts. Correlations with steering magnitude ( ), detection ( ), and forced identification ( ) are shown. Max-activating examples for this feature are shown in Appendix M . (c) Progressive ablation and patching of top-ranked gate features (100 randomly-selected concepts, 10 trials each). Error bars and shaded regions: 95% CI.
Figure 12: Top-3 evidence carriers for gate L45 F9959 , across six example concepts (detection rates in parentheses). Activation increases monotonically with steering strength for the positive direction ( left ); feature labels and active concepts are provided ( right ). Some evidence carriers are concept-specific (e.g., geological terminology for Granite , astronomical phenomena for Constellations ), while others correspond to generic discourse features (e.g., emphatic interjections, informal transitions).
Figure 13: Gate activation ( L45 F9959 ) vs. steering strength under progressive ablation of upstream features, for six example concepts (detection rates in parentheses). Ablating evidence carriers (green) increases gate activation, confirming they normally suppress the gate. Weak-attribution controls (gold) track baseline (blue). The pattern is consistent across high- and low-detection concepts.
Figure 14: Gate L45 F9959 activation vs. steering strength across base ( left ), instruct ( middle ), and abliterated ( right ) models, for 5 success (green) vs. 5 failure (red) concepts. The inverted-V pattern is prominent in the instruct and abliterated models but weaker in the base model, consistent with post-training developing the gating mechanism. Correlations shown are for the instruct model.
Figure 15: Steering attribution graphs for Bread ( ). Node area denotes importance and edge width denotes edge-weight magnitude. Gates have solid red borders and evidence carriers have dashed green borders. Residual stream features shown only at the injection layer. (a) Direct concept vector injection. Both concept-agnostic ( L37 RESID F4821, F3149 ) and concept-specific features ( L37 RESID F6015, F9367 ) contribute. (b) Learned bias vector (§ 6 ). Gates have smaller node importance, suggesting learned steering relies less on strong gating and more on evidence carriers.
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 18: Heatmap of mean detection and forced identification rates across 20 concept categories for Gemma3-27B ( , , 100 trials per concept). Parentheses: per-category sample sizes.
Figure 19: Detection rate distribution across 500 concepts ( , , 100 trials per concept). Left : Concepts sorted by detection rate, colored by tier. The dashed line marks the success vs. failure partition threshold ( ). Top-10 and bottom-10 concepts are labeled. Right : Histogram of detection rates, showing a bimodal distribution with concentrations near 0% and 90%.
Figure 20: Per-model TPR FPR vs. for baseline, refusal direction ablation, and random direction control. Columns: injection layer at depth fractions . Shaded region: 95% CI.
Figure 21: Counteracting denial (a learned refusal behavior) by DPO on the instruct checkpoint. Per-model TPR FPR vs. steering strength for the frozen instruct model and five DPO conditions that differ only in preference pair content. Inset: per-model steering layer and DPO parameter. Chosen (C) and rejected (R) exemplars are shown in the legend. Shaded region: 95% CI.
Figure 22: Introspection metrics for OLMo-3.1-32B across its base, SFT, DPO, and instruct checkpoints at (a) and (b) and . Same format as Figure 5 , which was for . Results are given for the original 50 concepts from Lindsey (2025) . Shaded region: 95% CI.
| OLMo-3.1-32B | TPR FPR (%) |
| SFT ∗ | 11.5 2.4 |
| DPO =0.01 | 5.8 1.0 |
| DPO =0.1 | 14.4 1.6 |
| DPO =0.5 | 0.3 2.8 |
| DPO ∗ | 9.8 0.5 |
Table 8: -sweep for DPO training on OLMo-3.1-32B SFT checkpoint ( left ; metrics from and ) and Gemma3-27B base checkpoint ( right ; metrics from and ). Rows annotated with ∗ are official checkpoints and others are our LoRA-trained variants.
| Leave-one-out | TPR FPR (%) | Introspection (%) | Single domain | TPR FPR (%) | Introspection (%) | |
| All domains | 14.4 1.6 | 7.0 0.8 | All domains | 14.4 1.6 | 7.0 0.8 | |
| No safety | 11.5 2.3 | 7.0 1.1 | Only safety | 3.8 2.7 | 6.0 1.1 | |
| No NLP tasks | 11.1 2.3 | 5.6 1.0 | Only NLP tasks | 6.5 2.5 | 5.2 1.0 | |
| No instruction follow | 11.6 2.2 | 5.8 1.0 | Only instruction follow | 10.5 2.3 | 6.8 1.1 | |
| No general | 13.1 2.2 | 5.4 1.0 | Only general | 11.9 2.3 | 6.4 1.1 | |
| No code | 14.2 2.3 | 6.8 1.1 | Only code | 8.3 2.2 | 3.4 0.8 |
Table 9: Data domain ablations (OLMo-3.1-32B, DPO =0.1). Domains are inferred from the prompt_id field of the open-source OLMo DPO dataset. No single domain is necessary or sufficient.
Figure 23: Swap experiment using the ridge regression direction. Same format as Figure 6 . Both projection and residual swaps are effective, with residual swap producing larger effects than for .
Figure 26: Effect of steering along the mean-difference direction on response style. (a) Steering with mean-difference direction computed from concept vectors at layer 37. (b) Steering with mean-difference direction computed from layer 45 residual stream activations. Shaded region: 1 SEM.
Figure 27: 30-fold cross-validated AUC for binary classification of success vs. failure concepts.
Figure 28: Linear probe accuracy for classifying steered vs. unsteered activations before (clay) and after (blue) adding each attention head’s output. The 20 heads with the largest absolute accuracy change are shown. Error bars: standard error across 5-fold CV.
Figure 29: Per-layer causal interventions on the Gemma3-27B base ( left ) and abliterated ( right ) models. Same format as Figure 10 . The abliterated model preserves the L45 localization pattern; the base model shows no such structure. Shaded region: 95% CI.
Figure 30: (a) Top features promoting “Yes” (positive score) ranked by direct logit attribution. (b) Progressive ablation (100 randomly-selected concepts, 10 trials each) of top-ranked positive direct logit attribution features. Neither ablation from steered runs (red) nor patching into unsteered runs (green) produces meaningful changes in detection. Shaded region: 95% CI.
Figure 31: Max-activating examples for gate features associated with tokens preceding ’no’.
Figure 32: Max-activating examples for evidence carrier features associated with tokens preceding affirmation.
Figure 33: Progressive ablation of evidence carrier features (100 randomly-selected concepts, 10 trials each). Ablating from steered runs gradually reduces detection (red). Patching into unsteered runs produces only weak detection increases (green). Forced identification decreases more substantially (blue), suggesting carriers encode concept-specific information even when individually insufficient for detection. Same format as Figure 11 c . Shaded region: 95% CI.
Figure 34: Evidence carrier features (blue; top-100k) concentrate in earlier layers, whereas gate features (red; top-200) concentrate in later layers (45–61).
Figure 35: L45 F74631 activation vs. steering strength across base ( left ), instruct ( middle ), and abliterated ( right ) models, for 5 success (green) vs. 5 failure (red) concepts. Same format as Figure 14 .
Figure 36: Gate activation ( L45 F74631 ) vs. steering strength under progressive ablation of upstream features, for six example concepts (detection rates in parentheses). Same format as Figure 13 .
Figure 37: Top-3 evidence carriers for gate L45 F74631 , across six concepts. Activation increases monotonically with steering strength for both positive and negative directions. Feature labels and their active concepts are provided on the right. Carriers include both concept-specific (e.g., astronomical phenomena for Constellations ) and concept-agnostic features (e.g., informal discourse markers).
Figure 38: L50 F167 activation vs. steering strength across base ( left ), instruct ( middle ), and abliterated ( right ) models, for 5 success (green) vs. 5 failure (red) concepts. The inverted-V is sharper than L45 gates and absent in the base model but robust to abliteration.
Figure 39: Gate activation ( L50 F167 ) vs. steering strength under progressive ablation of upstream features, for six example concepts. Same format as Figure 13 . Results consistent with the L45 gates.
Figure 40: Top-3 evidence carriers for gate L50 F167 , across six concepts. Several carriers overlap with those identified for L45 gates, while others are gate-specific (e.g., L46 F143811 ).
Figure 41: MLP integrated steering attribution. L45 shows consistently strong positive attribution across concepts. L38, immediately after injection, also shows positive attribution for several concepts.
Figure 42: Attention integrated steering attribution. L39 and L61 show strong positive attribution, indicating both early and late attention features contribute to steering signal propagation and detection.
Figure 43: Steering attribution graph for concept injection without the learned bias vector (“bread”, injection layer 37, ). The source (injection layer) is at the bottom; information flows upward toward the target . Nodes correspond to SAE features across layers ( RESID , ATTN , TC ), with area proportional to node importance. Directed edges represent normalized edge weights. L45 TC F9959 , the #1-ranked gate feature from Section 5.3 , appears as a high-importance node.
Figure 44: Steering attribution graph for concept injection with the learned bias vector applied (“bread”, injection layer 37, ). Same source, target, and layout as Figure 43 , but with the learned bias vector from Section 6 present as a fixed perturbation. Compared to Figure 43 , L45 TC F9959 is suppressed and attribution shifts toward late-layer ATTN and TC features.
Figure 45: Left : Top SAE features of steering vector on layer 29 and 34. Middle : Max-activating examples for selected features. Right : Logit lens tokens of steering vectors across different layers.
Figure 46: Response length comparison across prompt categories. Common contains generic conversational prompts (e.g., greetings). Reasoning contains mathematical and logical problems. Harmful contains unsafe requests that should be refused. Tendency contains self-assessment or tendency questions. Introspection contains 10 variants of the introspection prompt from Section A . Introspection + common replaces the second turn (the explicit detection query) with a common prompt.
| Task | Baseline | Bias ( ) | |
| Dialogue | 73.86% | 74.71% | 0.85% |
| QA | 48.98% | 48.95% | 0.03% |
| Summarization | 70.88% | 67.87% | 3.01% |
Table 10: HaluEval hallucination detection accuracy, comparing baseline Gemma3-27B-IT vs. bias-adapted model ( ), on 10,000 samples per task.
| Metric | Baseline | Bias ( ) | |
| ASR (full compliance) | 40.2% | 42.6% | 2.4% |
| ASR (full + partial) | 89.2% | 89.4% | 0.2% |
| Refusal rate | 10.0% | 9.4% | 0.6% |
| Judge error rate | 0.8% | 1.2% | 0.4% |
Table 11: Jailbreak attack success rate on 500 randomly sampled JailbreakHub prompts, comparing baseline vs. bias-adapted model ( ).
| Benchmark | Baseline | Bias ( ) | Base Len | Bias Len | |
| MMLU | 37.5% | 21.5% | 16.0% | 1,288 | 613 |
| GPQA | 44.6% | 19.4% | 25.2% | 2,161 | 1,124 |
Table 12: Chain-of-thought faithfulness, measuring whether models verbalize hints that cause answer changes. Baseline vs. bias-adapted model ( ).
| Metric | Baseline | Bias ( ) | |
| Self claims wrote | 93.8% | 96.7% | 2.9% |
| Prefill detect rate | 36.3% | 16.1% | 20.3% |
| False alarm rate | 5.6% | 3.1% | 2.5% |
| Avg self-written length | 343 | 160 | 53% |
| Avg prefilled probe length | 549 | 214 | 61% |
Table 13: Prefill detection: ability to distinguish self-written vs. prefilled assistant turns, evaluated on 1,900 examples across 19 categories. Baseline vs. bias-adapted model ( ).
Figure 47: Gradient attribution for residual stream activations, averaged over 400 concepts, injection layers , and injection strengths .
Figure 48: Gradient attribution for post feedforward layernorm output activations, averaged over 400 concepts, injection layers , and injection strengths .
Figure 49: Gradient attribution for post attention layernorm output activations, averaged over 400 concepts, injection layers , and injection strengths .
Figure 50: Gradient attribution over attention heads, averaged over 400 concepts, injection layers , injection strengths , and attention heads.
Figure 51: Attention probabilities from the last prefill tokens to previous tokens for different layers after the injection layer 41, averaged over 20 concepts and attention heads.
More in this topic
Language Model Introspection
Language model introspection studies whether models can detect, interpret, and report their own internal states and behavior.
See all 36 papers