Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations

Sanjay Basu; Sadiq Y. Patel; Parth Sheth; Bhairavi Muralidharan; Namrata Elamaran; Aakriti Kinra; John Morgan; Rajaie Batniji

cs.AIMar 18, 2026

Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations

Authors: Sanjay Basu, Sadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji

Organizations: University of California San Francisco, San Francisco, CA, USA · Waymark, San Francisco, CA, USA · University of Pennsylvania, Philadelphia, PA, USA · Virginia Commonwealth University, Richmond, VA, USA ·

Abstract

Language models encode task-relevant knowledge in internal representations that far exceeds their output performance, but whether mechanistic interpretability methods can bridge this knowledge-action gap has not been systematically tested. We compared four mechanistic interpretability methods -- concept bottleneck steering (Steerling-8B), sparse autoencoder feature steering, logit lens with activation patching, and linear probing with truthfulness separator vector steering (Qwen 2.5 7B Instruct) -- for correcting false-negative triage errors using 400 physician-adjudicated clinical vignettes (144 hazards, 256 benign). Linear probes discriminated hazardous from benign cases with 98.2% AUROC, yet the model's output sensitivity was only 45.1%, a 53-percentage-point knowledge-action gap. Concept bottleneck steering corrected 20% of missed hazards but disrupted 53% of correct detections, indistinguishable from random perturbation (p=0.84). SAE feature steering produced zero effect despite 3,695 significant features. TSV steering at high strength corrected 24% of missed hazards while disrupting 6% of correct detections, but left 76% of errors uncorrected. Current mechanistic interpretability methods cannot reliably translate internal knowledge into corrected outputs, with implications for AI safety frameworks that assume interpretability enables effective error correction.

Figures & tables

Characteristic	Physician-created ( $n=200$ )	Real-world ( $n=200$ )	Total ( $n=400$ )
Hazard cases	132 (66.0%)	12 (6.0%)	144 (36.0%)
Benign cases	68 (34.0%)	188 (94.0%)	256 (64.0%)
Hazard categories
Prescription adherence	16	—	16
Obstetric emergency	13	—	13
Privacy concerns	11	—	11

Table 1: Study population characteristics. The physician-created subset contains vignettes authored by a board-certified physician with pre-specified ground-truth labels. The real-world subset contains de-identified Medicaid patient encounter messages.

Model	Dataset	TP	FN	FP	TN	Sensitivity (95% CI)	Specificity (95% CI)	MCC (95% CI)
Steerling-8B	Physician	47	85	16	52	0.356 (0.264–0.459)	0.765 (0.650–0.851)	0.123 ( $-$ 0.035–0.233)
Steerling-8B	Real-world	4	8	62	126	0.333 (0.121–0.630)	0.670 (0.601–0.733)	0.002 ( $-$ 0.086–0.105)
Steerling-8B	Overall	51	93	78	178	0.354 (0.280–0.435)	0.695 (0.638–0.747)	0.044 ( $-$ 0.046–0.131)
Qwen 2.5 7B	Physician	62	70	22	46	0.470 (0.386–0.555)	0.676 (0.558–0.778)	0.134 (0.007–0.252)
Qwen 2.5 7B	Real-world	3	9	18	170	0.250 (0.070–0.591)	0.904 (0.855–0.938)	0.100 ( $-$ 0.058–0.299)
Qwen 2.5 7B	Overall	65	79	40	216	0.451 (0.371–0.534)	0.844 (0.795–0.883)	0.280 (0.175–0.378)

Table 2: Baseline triage performance. Sensitivity, specificity, and Matthews correlation coefficient (MCC) for each model on each dataset subset, with 95% confidence intervals. Wilson score 95% CIs are reported for sensitivity and specificity. BCa bootstrap 95% CIs (1,000 resamples) are reported for MCC.

Arm	Method	FN correction rate (95% CI)	TP disruption rate (95% CI)	Key metric	$p$ vs. control
1	Concept (best: $\alpha$ =0.00)	20.0% (12.8–29.8)	53.2% (38.7–67.1)	Net: $-$ 33.2 pp	$p$ =0.84
1	In-distribution (TP-P95)	22.4% (14.6–32.5)	51.1% (36.8–65.1)	Net: $-$ 28.8 pp	—
1	Random ( $\alpha$ =0.00)	30.6% (21.6–41.2)	61.7% (47.0–74.6)	Net: $-$ 31.1 pp	—
2	SAE feature steering	0.0% (0.0–4.6)	0.0% (0.0–5.6)	3,695 features; no effect	—
3	Logit lens	—	—	Cohen’s $d$ =1.39 (layer 22)	—
3	Act. patching ( $\alpha$ =5.0)	20.3% (12.9–30.4)	7.7% (3.3–16.8)	net +11	—

Table 3: Head-to-head comparison of mechanistic interpretability methods. Comparison of all four arms for their ability to correct false-negative triage errors without disrupting true-positive detections. Wilson score 95% CIs are reported for correction and disruption rates. AUROC 95% CIs are from 1,000 bootstrap resamples.

Figure 1: Dose-response relationship between concept steering strength and triage outcomes in Steerling-8B. False-negative correction rate (solid blue) and true-positive disruption rate (dashed red) as a function of concept activation alpha level for hazard-associated concept steering in the physician-created dataset (85 false negatives, 47 true positives at baseline). Error bars indicate 95% Wilson score confidence intervals. At every alpha level, the true-positive disruption rate exceeds the false-negative correction rate. The random-concept control (triangles) shows comparable correction and disruption rates, indicating that hazard-concept steering is no more effective than random perturbation.

Figure 2: Per-layer probe AUROC and TSV discrimination in Qwen 2.5 7B. Cross-validated AUROC (5-fold stratified) for L2-regularised logistic regression probes trained at each of the 28 layers of Qwen 2.5 7B to discriminate physician-adjudicated hazardous from benign cases (

n=400

). Probes achieve AUROC above 0.95 at all layers, peaking at 0.982 (95% CI 0.968 to 0.993) at layer 23 (arrow). The dashed horizontal line indicates the model’s actual output sensitivity of 0.451. At layer 23, the truthfulness separator vector discriminates true-positive from false-negative cases with AUROC 0.814 (95% CI 0.738 to 0.887). The gap between what the model knows internally and what it does quantifies the knowledge-action gap that TSV steering only partially bridged.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hazard Category	$N$	Baseline TP	Baseline FN	Sensitivity
Rx Adherence	16	2	14	0.125
Obstetric Emergency	13	8	5	0.615
Privacy	11	3	8	0.273
Med Reconciliation	10	0	10	0.000
Privacy Proxy	10	3	7	0.300
Misuse Escalation	10	7	3	0.700

Appendix

Table 4: Hazard category detection rates (Steerling-8B, physician-created dataset).

Statistic	Value
Total concept activations ( $400\times 33{,}732$ )	13,492,800
Global mean activation	0.000126
Global median activation	0.000009
Global maximum activation	0.628
Fraction of activations $<0.01$	99.92%
99th percentile activation	0.002

Appendix

Table 5: Concept activation distribution.

Layer	Accuracy	AUROC	Best $C$
0	0.9550	0.9635	10.0
1	0.9475	0.9680	10.0
2	0.9550	0.9707	10.0
3	0.9425	0.9538	10.0
4	0.9425	0.9547	10.0
5	0.9450	0.9551	10.0

Appendix

Table 6: Per-layer probe results for Qwen 2.5 7B Instruct. Bold indicates best layer by AUROC.

Layer	TP Mean Rank ( $n$ =65)	FN Mean Rank ( $n$ =79)	Cohen’s $d$
0	118,576	118,768	0.28
7	106,544	106,619	0.17
14	105,586	105,832	0.66
21	103,247	103,951	1.09
22	103,081	103,984	1.39
27	13,380	16,370	1.37

Appendix

Table 7: Logit lens hazard token ranks by layer (Qwen 2.5 7B Instruct). Bold indicates critical layer (maximum Cohen’s

d

Parser	TP	FN	FP	TN	Sensitivity (95% CI)	Specificity (95% CI)	MCC
Original	65	79	40	216	0.451 (0.372–0.533)	0.844 (0.794–0.883)	0.322
Refined	105	39	54	202	0.729 (0.651–0.795)	0.789 (0.735–0.835)	0.508

Appendix

Table 8: Response parser sensitivity analysis. Panel A: Qwen 2.5 7B. Panel B: Steerling-8B.

Group	$N$	TP	FN	TN	FP	Sensitivity	Specificity	Accuracy
White	200	59	73	47	21	0.447	0.691	0.530
Black	200	64	68	44	24	0.485	0.647	0.540
Hispanic	200	49	83	44	24	0.371	0.647	0.465

Appendix

Table 9: Demographic variation in detection performance (Steerling-8B, physician-created vignettes).

Figure 3: Concept activation heatmap by hazard category. Mean concept activation (sigmoid space) for the top 10 concepts across 18 hazard categories and benign cases. Darker cells indicate higher mean activation. The sparsity of the heatmap reflects the overall sparsity of the concept activation space (99.92% of activations $<0.01$ ).

Figure 4: False-negative correction and true-positive disruption rates for concept-level interventions. (A) False-negative correction rate for hazard-concept amplification (

\alpha=1.0

), random-concept amplification (

\alpha=1.0

), and prompt engineering. (B) True-positive disruption rate for concept suppression (

\alpha=0.0

): hazard-concept suppression disrupted 53.2% of correct detections, comparable to random-concept suppression (61.7%). Error bars indicate 95% Wilson score CIs.

Figure 5: Triage sensitivity and specificity by demographic group (Steerling-8B, 200 physician-created vignettes with racial/ethnic descriptors). Sensitivity ranged from 0.371 (Hispanic) to 0.485 (Black); differences were not statistically significant (

\chi^{2}

test,

p=0.24

). Specificity was similar across groups (0.647 to 0.691).

Item	Section	Page
Title	Title page	1
Abstract	Abstract	1
Background/rationale	Introduction	2
Objectives	Introduction, final paragraph	3
Source of data	Methods, Dataset	4
Participants	Methods, Dataset	4

Appendix

Table 15

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 391 papers

Scaling Inherently Interpretable Language Models
Aug 6, 2026Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail +7Language Model Steering LLM Interpretability
Guide Labs Team
The Misery of Mechanistic Interpretability: A Formal Perspective
Sep 14, 2026Tobias Ladner, Matthias AlthoffNeural Network Verification LLM Interpretability
Technical University of Munich, Germany
When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models
Sep 4, 2026Gnaneswar Villuri, Hashmath Shaik, Alex DoboliLLM Auditing Language Model Calibration
Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA

Aug 6, 2026J/K move · Enter open · S save

Scaling Inherently Interpretable Language Models

Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail, Giang Nguyen+6

Guide Labs Team

Language Model Steering LLM Interpretability Diffusion Language Models Representation Geometry in Language Models LLM Training

Open paper AskPDF arXiv

Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.

Abstract

Mechanistic Interpretability

Explore similar work

Scaling Inherently Interpretable Language Models

The Misery of Mechanistic Interpretability: A Formal Perspective

When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models