Causality is Key for Interpretability Claims to Generalise

Shruti Joshi; Aaron Mueller; David Klindt; Wieland Brendel; Patrik Reizinger; Dhanya Sridhar

cs.LGFeb 18, 2026

Causality is Key for Interpretability Claims to Generalise

Authors: Shruti Joshi, Aaron Mueller, David Klindt, Wieland Brendel, Patrik Reizinger, Dhanya Sridhar

Organizations: Mila - Québec AI Institute & Université de Montréal · Boston University · Cold Spring Harbor Laboratory · Max-Planck-Institute for Intelligent Systems, ELLIS Institute Tübingen, University of Tübingen

Abstract

Interpretability research on large language models (LLMs) has yielded important insights into model behaviour, yet recurring pitfalls persist: findings that do not generalise, and causal interpretations that outrun the evidence. Our position is that causal inference specifies what constitutes a valid mapping from model activations to invariant high-level structures, the data or assumptions needed to achieve it, and the inferences it can support. Specifically, Pearl's causal hierarchy clarifies what an interpretability study can justify. Observations establish associations between model behaviour and internal components. Interventions (e.g., ablations or activation patching) support claims how these edits affect a behavioural metric (e.g., average change in token probabilities) over a set of prompts. However, counterfactual claims -- i.e., asking what the model output would have been for the same prompt under an unobserved intervention -- remain largely unverifiable without controlled supervision. We show how causal representation learning (CRL) operationalises this hierarchy, specifying which variables are recoverable from activations and under what assumptions. Together, these motivate a diagnostic framework that helps practitioners select methods and evaluations matching claims to evidence such that findings generalise.

Figures & tables

Method	Aim	What the Evidence Supports
Sparse autoencoders	L3 The learned features correspond to a unique set of concepts. Identifiability claim.	L1 A sparse basis that minimises reconstruction error on the training data. Describes a basis, without yet establishing uniqueness.
Auto-interpretability Explanations	L3 This feature corresponds to the underlying concept named by the description. Semantic assignment.	L1 The description predicts when the feature activates on held-out text. Distinguishes activating from non-activating contexts, without confirming the feature’s causal role.
Circuit discovery	L3 This circuit is a key mediator of the behaviour. Causal attribution.	L2 Ablating this circuit changes model behaviour on evaluated prompts. An intervention effect for the chosen ablation, not yet a unique localisation.

Table 1: Interpretability methods typically produce associational or interventional evidence (L1–L2), yet the interpretations we’d like to draw often implicitly require counterfactual reasoning (L3) . Recognising this rung gap clarifies what additional evidence is needed to justify stronger claims.

Inferential Gap	Implied Claim	Actual Scope
Existence $\to$ Uniqueness	This is the circuit or factor causing a behaviour.	A circuit that produces output aligned with the behaviour has been found, but the solution is generically non-unique. Other circuits can implement equivalent input–output behaviour ( Wang et al., 2023 ; McGrath et al., 2023 ) .
Correlation $\to$ Causation	Feature ${\mathbf{h}}^{(l)}_{S}$ causally mediates behaviour related to concept $c$ .	Without targeted interventions (eg. interchange interventions Geiger et al. 2021 or causal scrubbing Chan et al. 2022 ), we can only determine that ${\mathbf{h}}_{S}^{(l)}$ and $c$ are correlated, but the correlation may reflect confounding rather than a causal pathway.
Decodability $\to$ Model Use	The model represents and computes with concept $c$ .	A probe can decode $c$ from ${\mathbf{h}}_{S}^{(l)}$ , which does not imply the model’s computations depend on $c$ ( Belinkov, 2022 ; Elazar et al., 2021 ; Hewitt and Liang, 2019 ; Ravichander et al., 2021 ) .
Local Sensitivity $\to$ Global Causal Role	The mechanism generalises beyond tested prompts.	Sensitivity to $c$ holds for specific inputs since local attributions can be input dependent and may not generalise to held-out distributions ( Adebayo et al., 2020 ; Bilodeau et al., 2024 ) .
Subspace $\to$ Direction	Concept $c$ is encoded along a single direction in activation space.	A linear probe or PCA component recovers the single most predictive direction for $c$ , but it may be best represented through a subspace. Probes trained on different datasets may hence discover different directions depending on which component is most prevalent, each direction a valid but lossy projection of the same higher-dimensional representation ( Pan et al., 2025 ; Engels et al., 2025 ) .
Sufficiency $\to$ Necessity	This component or mechanism is necessary for observed behaviour.	A pathway under specific interventions may be sufficient to produce the desired behaviour, but sufficiency does not entail necessity since alternative pathways for the same behaviour may exist. ( Wang et al., 2023 ; Heimersheim and Nanda, 2024 ) .

Table 2: Common estimand-evidence gaps in mechanistic interpretability. Each row contrasts an Implied Claim with its Actual Scope , as supported by the reported evidence, citing work that documents the gap or demonstrates ways to address it. Our aim is to offer shared terminology that helps unify these efforts and make progress comparable.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Intervenable object	Representative methods
Inputs
prompts, context	Minimal pairs [ Marvin and Linzen, 2018 ] $\cdot$ Prompt perturbations [ Ribeiro et al., 2020 ] $\cdot$ Jailbreak suffixes [ Zou et al., 2023 ] $\cdot$ Persona modulation [ Zheng et al., 2024 ]
Activations
layer states	Activation patching [ Wang et al., 2023 ] $\cdot$ Causal mediation analysis [ Vig et al., 2020 ] $\cdot$ Ablations [ Ghorbani and Zou, 2020 ]
Representations
learned features	SAE feature steering [ Templeton et al., 2024 ] $\cdot$ Interchange interventions [ Geiger et al., 2022 ]

Appendix

Table 3: Examples of intervention classes in LLMs and representative methods.

Rung of the Ladder	Query Distribution	Practical Query (with Examples)
1: Association	$p({\mathbf{y}}\mid{\mathbf{x}})$	Prediction: Given an input prompt ${\mathbf{x}}$ , what output tokens ${\mathbf{y}}$ are likely? (Standard next-token prediction, logit inspection)
	$p({\mathbf{h}}\mid{\mathbf{x}})$	Encoding: Given an input ${\mathbf{x}}$ , what internal representations or features ${\mathbf{h}}$ are typically activated? (Activation logging, SAE feature attribution, probing)
	$p({\mathbf{y}}\mid{\mathbf{h}})$	Decoding: Given an observed internal trace ${\mathbf{h}}$ , what outputs ${\mathbf{y}}$ are likely? (Linear probes, sparse decoding, feature-to-token analyses)
2: Intervention	$p({\mathbf{y}}\mid\mathrm{do}({\mathbf{x}}={\mathbf{x}}^{\prime}))$	Input intervention: If I change the prompt from ${\mathbf{x}}$ to ${\mathbf{x}}^{\prime}$ , how does the output change? (Minimal pairs, prompt perturbations, jailbreak tests)
	$p({\mathbf{y}}\mid\mathrm{do}({\mathbf{a}}^{(l)}={\mathbf{a}}^{\prime}))$	Activation intervention: If I overwrite or patch activations at layer $l$ , how does the output change? (Activation patching, causal tracing, ablations)
	$p({\mathbf{y}}\mid\mathrm{do}({\mathbf{h}}={\mathbf{h}}^{\prime}))$	Representation intervention: If I manipulate a feature or direction in representation space, what behavior changes? (SAE feature steering/ablation, DAS interchange interventions)

Appendix

Table 4: Interpretability queries across Pearl’s causal ladder, expressed in terms of association, intervention, and counterfactuals.

Method family	Representative methods	Implicit estimand	Evidence type
Feature attribution saliency / scores	Saliency $\cdot$ Grad $\times$ Input $\cdot$ Integrated Gradients $\cdot$ SHAP $\cdot$ LIME	Local input–output sensitivity (at a given input and parameterization); marginal contribution relative to an analyst-defined baseline or perturbation distribution	Gradients / synthetic perturbations
Diagnostics / probing readouts	Linear probes $\cdot$ CAVs $\cdot$ Logit lens	Existence of a decision function in a restricted probe class that predicts a concept from internal states; correlational alignment with external labels	Supervised labels / correlations
Causal / circuit-based interventions	Activation patching $\cdot$ Causal tracing $\cdot$ Circuit analysis	Interventional effect of manipulating internal states (e.g. replacing $\mathbf{a}^{(l)}$ ) on downstream outputs	Controlled interventions
Structured decomposition basis learning	SAEs $\cdot$ PCA $\cdot$ Dictionary learning	Factors induced by an optimisation objective (e.g. reconstruction with sparsity or low-rank constraints)	Reconstruction objective
Theory-grounded identifiable structure	DAS $\cdot$ SSAEs	Recovery up to an equivalence class under stated identifiability assumptions	Interventions / Sufficient variations / Inductive biases

Appendix

Table 5: Interpretability method families and the implicit estimands they target.

Model (version)	Provider	Claims	Est. cost
Claude Opus 4.5 ( claude-opus-4-5 )	Anthropic	186	$88
Claude Sonnet 4 ( claude-sonnet-4-20250514 )	Anthropic	186	$18
GPT-5.2 ( gpt-5.2 )	OpenAI	186	$14
Gemini 3 Flash ( gemini-3-flash-preview )	OpenRouter	186	$1
Mistral Large ( mistral-large-2512 )	OpenRouter	186	$11
DeepSeek V3 ( deepseek-v3.2 )	OpenRouter	186	$2

Appendix

Table 6: Annotator models and estimated API costs (February 2026 pricing). Each call sends

{\sim}30\text{K}

input tokens (codebook + paper text + claim) and receives

{\sim}500

output tokens.

Variable	$N$	Primary	95% CI	Secondary	Unanimous
method_rung	186	$\alpha_{ord}$ = 0.66	[0.53, 0.77]	Light’s $\kappa_{w}$ = 0.68	69.9%
claim_rung	186	$\alpha_{ord}$ = 0.56	[0.48, 0.62]	Light’s $\kappa_{w}$ = 0.52	34.4%
gap_score	186	$\alpha_{ord}$ = 0.56	[0.50, 0.61]	Light’s $\kappa_{w}$ = 0.56	41.4%
gap_binary	186	Fleiss’ $\kappa$ = 0.55	[0.49, 0.61]	$\alpha_{nom}$ = 0.55	47.8%
confidence	186	$\alpha_{ord}$ = 0.11	[0.06, 0.16]	Light’s $\kappa_{w}$ = 0.17	5.9%

Appendix

Table 7: Multi-rater agreement across 7 LLM annotators. Primary metric: Krippendorff’s

\alpha

(ordinal) for rung variables, Fleiss’

\kappa

for binary overclaim. Bootstrap CIs use paper-level cluster resampling (percentile method,

n=10\,000

	Opus 4.5	GPT-5.2	DS-V3	Gemini 3F	Mistral L	Qwen 3
GPT-5.2	0.79
DS-V3	0.73	0.66
Gemini 3F	0.72	0.59	0.70
Mistral L	0.64	0.60	0.67	0.68
Qwen 3	0.65	0.65	0.68	0.64	0.66
Sonnet 4	0.83	0.80	0.70	0.62	0.68	0.67

Appendix

Table 8: Pairwise weighted

\kappa

for method_rung across 7 annotators.

	Opus 4.5	GPT-5.2	DS-V3	Gemini 3F	Mistral L	Qwen 3
GPT-5.2	0.52
DS-V3	0.59	0.48
Gemini 3F	0.49	0.47	0.41
Mistral L	0.43	0.44	0.37	0.68
Qwen 3	0.53	0.55	0.47	0.57	0.63
Sonnet 4	0.71	0.56	0.55	0.52	0.44	0.60

Appendix

Table 9: Pairwise weighted

\kappa

for claim_rung across 7 annotators.

	Opus 4.5	GPT-5.2	DS-V3	Gemini 3F	Mistral L	Qwen 3
GPT-5.2	0.61
DS-V3	0.57	0.53
Gemini 3F	0.59	0.57	0.42
Mistral L	0.53	0.55	0.39	0.66
Qwen 3	0.57	0.58	0.45	0.59	0.59
Sonnet 4	0.76	0.60	0.50	0.61	0.50	0.60

Appendix

Table 10: Pairwise weighted

\kappa

for gap_score across 7 annotators.

Paper type	$N$	$\alpha_{ord}$
Applied/production	26	0.58
Circuit discovery	59	0.55
Evaluation/benchmark	16	0.61
Knowledge localization	10	0.64
Other	75	0.52

Appendix

Table 11: Krippendorff’s

\alpha

(ordinal) for gap_score stratified by Paper type.

Model	$N$	$\bar{R}_{m}$	$\bar{R}_{c}$	$\bar{g}$	OC %	$\bar{conf}$
Opus 4.5	186	1.69	2.22	0.54	41.9%	4.28
GPT-5.2	186	1.62	2.12	0.63	50.5%	4.19
DS-R1	166	1.73	2.16	0.51	43.4%	4.28
DS-V3	186	1.75	2.17	0.42	34.4%	4.09
Gemini 3 Flash	186	1.78	2.54	0.79	63.4%	4.99
Mistral Large	186	1.77	2.56	0.82	65.6%	4.55

Appendix

Table 12: Per-model annotation tendencies across all rated claims.

\bar{R}_{m}

\bar{R}_{c}

: mean method/claim rung;

\bar{g}

: mean gap score; OC: overclaim rate (gap

>0

	Primary (7)		+R1 (8)
Variable	$N$	$\alpha$	$N$	$\alpha$
method_rung	186	0.66	166	0.67
claim_rung	186	0.56	166	0.55
gap_score	186	0.56	166	0.54
gap_binary	186	0.55	166	0.55
confidence	186	0.11	166	0.13

Appendix

Table 13: Krippendorff’s

\alpha

(ordinal): 7 primary annotators vs. 8 annotators (including DeepSeek R1 supplementary on 166 claims).

Explore similar work

Sep 13, 2026

Calibrating Interpretability Instruments Before Trusting Their Verdicts

Causal claims about large language model (LLM) internals rest on measurements. Those might include a projection, a cosine, an ablation delta, or an interchange patch among others. These measurements fail in specific, diagnosable ways that return a plausible number instead of an error, so a broken instrument can easily read as a finding. A covariance-matched null can saturate until every direction looks typical, a per-head attribution can overshoot the true residual write threefold on reordered-normalization architectures, an interchange patch can go sign-chaotic because its outcome is pinned at a ceiling, or a read-from verdict can be an artifact of measuring past the layer where the model already decided. This note documents six such failures from a causal interpretability program on refusal and moral representation, spanning several papers and a four-model open-weight panel; each mode is established on one or two of the four. For each we give the tell that catches it and a protocol keyed to a detectable trigger (reordered normalization, massive activations, a low-dimensional decision channel), so we and readers can check whether a given setup is exposed. The discipline reduces to four moves: calibrate against a positive-control ladder, certify with an orthogonal cell, compute power before spending compute, and state every read-from verdict at a depth referenced to the model's commitment. The evidence is four architectures across three families within a single program; external replication across programs is future work.

Orion Reblitz-Richardson

Distiller Labs

LLM Interpretability Causal Interpretability Causal Interventions in Language Models Explainability Evaluation

Ask PDF

May 26, 2026

Why LLMs Fail at Causal Discovery and How Interventional Agents Escape

Causal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it reliably remains an open question. Recent benchmarks show that even fine-tuned models plateau on simple causal graphs and degrade as complexity grows, but why they fail has not been established. We prove the failure is fundamental: supervised fine-tuning, direct preference optimization, and in-context learning all produce predictors that cannot distinguish between causal graphs generating similar observational data, and any attempt to do so requires the model's internal representations to grow unboundedly, violating the very conditions under which these methods work. We formalize this as a kernel obstruction theorem, establishing that the limitation is intrinsic to the learning paradigm, not any particular model or dataset. We propose Agentic Causal Bayesian Optimization (A-CBO), wherein a frozen language model serves as an interventional oracle answering targeted queries about intervention effects, while an external Bayesian loop concentrates beliefs over candidate graphs in logarithmically many rounds. Because the decision operates outside the space where the obstruction applies, A-CBO provably converges while the underlying model remains unchanged. On Corr2Cause, A-CBO matches fine-tuned baselines without any training. On Extended Corr2Cause, a new benchmark scaling to 24 variables with 18K test samples, A-CBO significantly outperforms both fine-tuning and preference optimization, with the advantage growing

Amartya Roy, Sonali Parbhoo

SIRE, IIT Delhi and Robert Bosch GmbH, India · Imperial College London

Causal Discovery Causal Reasoning in Language Models Causal Identifiability Causal Interventions in Language Models

Ask PDF

Oct 2, 2026

Slaying the Hydra: Interaction-Aware Circuit Discovery in Language Models

Localizing behavior to individual components of a language model is a central goal of mechanistic interpretability. However, scoring components one at a time misses context-dependent effects: a primary component can inhibit the activation of a backup, leading to issues with ranking components. Actual causality studies the structure of such interactions via witnesses: variables that provide contextual information to resolve interaction terms. However, estimation with witnesses typically requires combinatorial enumeration and is infeasible in practice. We introduce the witness-integrated set effect (WISE), a family of causal estimands that build on the witness mechanism while taking expectations over sets of causes and witnesses to remain computationally feasible. Building on this approach, we introduce JuntaLearner, a gradient-based circuit discovery method that learns to rank components by their causal impact across varying-sized sets of components and witnesses. Alongside faithfulness metrics, we introduce measures of necessity and task specificity, and the circuit recognition score (CRS) to summarize each metric across circuit sizes while emphasizing effects achieved by small circuits. Across tasks and models of increasing size, JuntaLearner achieves higher mean CRS compared to attribution baselines on all metrics. Since its cost does not grow with the number of candidate components, JuntaLearner scales to large models while accounting for set-level interactions and avoiding first-order approximations.

Sankaran Vaidyanathan, Rafal Urbaniak, Emily Bunnapradist +2

Basis Research Institute · University of Massachusetts Amherst · MIT

Circuits in Language Models Mechanistic Interpretability Causal Interpretability Causal Interventions in Language Models Actual Causality

Ask PDF