Organizations: Mila - Québec AI Institute & Université de Montréal · Boston University · Cold Spring Harbor Laboratory · Max-Planck-Institute for Intelligent Systems, ELLIS Institute Tübingen, University of Tübingen
Interpretability research on large language models (LLMs) has yielded important insights into model behaviour, yet recurring pitfalls persist: findings that do not generalise, and causal interpretations that outrun the evidence. Our position is that causal inference specifies what constitutes a valid mapping from model activations to invariant high-level structures, the data or assumptions needed to achieve it, and the inferences it can support. Specifically, Pearl's causal hierarchy clarifies what an interpretability study can justify. Observations establish associations between model behaviour and internal components. Interventions (e.g., ablations or activation patching) support claims how these edits affect a behavioural metric (e.g., average change in token probabilities) over a set of prompts. However, counterfactual claims -- i.e., asking what the model output would have been for the same prompt under an unobserved intervention -- remain largely unverifiable without controlled supervision. We show how causal representation learning (CRL) operationalises this hierarchy, specifying which variables are recoverable from activations and under what assumptions. Together, these motivate a diagnostic framework that helps practitioners select methods and evaluations matching claims to evidence such that findings generalise.
Figures & tables
Method
Aim
What the Evidence Supports
Sparse autoencoders
L3 The learned features correspond to a unique set of concepts. Identifiability claim.
L1 A sparse basis that minimises reconstruction error on the training data. Describes a basis, without yet establishing uniqueness.
Auto-interpretability Explanations
L3 This feature corresponds to the underlying concept named by the description. Semantic assignment.
L1 The description predicts when the feature activates on held-out text. Distinguishes activating from non-activating contexts, without confirming the feature’s causal role.
Circuit discovery
L3 This circuit is a key mediator of the behaviour. Causal attribution.
L2 Ablating this circuit changes model behaviour on evaluated prompts. An intervention effect for the chosen ablation, not yet a unique localisation.
Table 1: Interpretability methods typically produce associational or interventional evidence (L1–L2), yet the interpretations we’d like to draw often implicitly require counterfactual reasoning (L3) . Recognising this rung gap clarifies what additional evidence is needed to justify stronger claims.
Inferential Gap
Implied Claim
Actual Scope
Existence → Uniqueness
This is the circuit or factor causing a behaviour.
A circuit that produces output aligned with the behaviour has been found, but the solution is generically non-unique. Other circuits can implement equivalent input–output behaviour ( Wang et al., 2023 ; McGrath et al., 2023 ) .
Correlation → Causation
Feature hS(l) causally mediates behaviour related to concept c .
Without targeted interventions (eg. interchange interventions Geiger et al. 2021 or causal scrubbing Chan et al. 2022 ), we can only determine that hS(l) and c are correlated, but the correlation may reflect confounding rather than a causal pathway.
Decodability → Model Use
The model represents and computes with concept c .
A probe can decode c from hS(l) , which does not imply the model’s computations depend on c ( Belinkov, 2022 ; Elazar et al., 2021 ; Hewitt and Liang, 2019 ; Ravichander et al., 2021 ) .
Local Sensitivity → Global Causal Role
The mechanism generalises beyond tested prompts.
Sensitivity to c holds for specific inputs since local attributions can be input dependent and may not generalise to held-out distributions ( Adebayo et al., 2020 ; Bilodeau et al., 2024 ) .
Subspace → Direction
Concept c is encoded along a single direction in activation space.
A linear probe or PCA component recovers the single most predictive direction for c , but it may be best represented through a subspace. Probes trained on different datasets may hence discover different directions depending on which component is most prevalent, each direction a valid but lossy projection of the same higher-dimensional representation ( Pan et al., 2025 ; Engels et al., 2025 ) .
Sufficiency → Necessity
This component or mechanism is necessary for observed behaviour.
A pathway under specific interventions may be sufficient to produce the desired behaviour, but sufficiency does not entail necessity since alternative pathways for the same behaviour may exist. ( Wang et al., 2023 ; Heimersheim and Nanda, 2024 ) .
Table 2: Common estimand-evidence gaps in mechanistic interpretability. Each row contrasts an Implied Claim with its Actual Scope , as supported by the reported evidence, citing work that documents the gap or demonstrates ways to address it. Our aim is to offer shared terminology that helps unify these efforts and make progress comparable.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Intervenable object
Representative methods
Inputs
prompts, context
Minimal pairs [ Marvin and Linzen, 2018 ] ⋅ Prompt perturbations [ Ribeiro et al., 2020 ] ⋅ Jailbreak suffixes [ Zou et al., 2023 ] ⋅ Persona modulation [ Zheng et al., 2024 ]
Activations
layer states
Activation patching [ Wang et al., 2023 ] ⋅ Causal mediation analysis [ Vig et al., 2020 ] ⋅ Ablations [ Ghorbani and Zou, 2020 ]
Representations
learned features
SAE feature steering [ Templeton et al., 2024 ] ⋅ Interchange interventions [ Geiger et al., 2022 ]
Appendix
Table 3: Examples of intervention classes in LLMs and representative methods.
Rung of the Ladder
Query Distribution
Practical Query (with Examples)
1: Association
p(y∣x)
Prediction: Given an input prompt x , what output tokens y are likely? (Standard next-token prediction, logit inspection)
p(h∣x)
Encoding: Given an input x , what internal representations or features h are typically activated? (Activation logging, SAE feature attribution, probing)
p(y∣h)
Decoding: Given an observed internal trace h , what outputs y are likely? (Linear probes, sparse decoding, feature-to-token analyses)
2: Intervention
p(y∣do(x=x′))
Input intervention: If I change the prompt from x to x′ , how does the output change? (Minimal pairs, prompt perturbations, jailbreak tests)
p(y∣do(a(l)=a′))
Activation intervention: If I overwrite or patch activations at layer l , how does the output change? (Activation patching, causal tracing, ablations)
p(y∣do(h=h′))
Representation intervention: If I manipulate a feature or direction in representation space, what behavior changes? (SAE feature steering/ablation, DAS interchange interventions)
Appendix
Table 4: Interpretability queries across Pearl’s causal ladder, expressed in terms of association, intervention, and counterfactuals.
Local input–output sensitivity (at a given input and parameterization); marginal contribution relative to an analyst-defined baseline or perturbation distribution
Gradients / synthetic perturbations
Diagnostics / probing readouts
Linear probes ⋅ CAVs ⋅ Logit lens
Existence of a decision function in a restricted probe class that predicts a concept from internal states; correlational alignment with external labels
Table 5: Interpretability method families and the implicit estimands they target.
Model (version)
Provider
Claims
Est. cost
Claude Opus 4.5 ( claude-opus-4-5 )
Anthropic
186
$88
Claude Sonnet 4 ( claude-sonnet-4-20250514 )
Anthropic
186
$18
GPT-5.2 ( gpt-5.2 )
OpenAI
186
$14
Gemini 3 Flash ( gemini-3-flash-preview )
OpenRouter
186
$1
Mistral Large ( mistral-large-2512 )
OpenRouter
186
$11
DeepSeek V3 ( deepseek-v3.2 )
OpenRouter
186
$2
Appendix
Table 6: Annotator models and estimated API costs (February 2026 pricing). Each call sends ∼30K input tokens (codebook + paper text + claim) and receives ∼500 output tokens.
Variable
N
Primary
95% CI
Secondary
Unanimous
method_rung
186
αord = 0.66
[0.53, 0.77]
Light’s κw = 0.68
69.9%
claim_rung
186
αord = 0.56
[0.48, 0.62]
Light’s κw = 0.52
34.4%
gap_score
186
αord = 0.56
[0.50, 0.61]
Light’s κw = 0.56
41.4%
gap_binary
186
Fleiss’ κ = 0.55
[0.49, 0.61]
αnom = 0.55
47.8%
confidence
186
αord = 0.11
[0.06, 0.16]
Light’s κw = 0.17
5.9%
Appendix
Table 7: Multi-rater agreement across 7 LLM annotators. Primary metric: Krippendorff’s α (ordinal) for rung variables, Fleiss’ κ for binary overclaim. Bootstrap CIs use paper-level cluster resampling (percentile method, n=10000 ).
Opus 4.5
GPT-5.2
DS-V3
Gemini 3F
Mistral L
Qwen 3
GPT-5.2
0.79
DS-V3
0.73
0.66
Gemini 3F
0.72
0.59
0.70
Mistral L
0.64
0.60
0.67
0.68
Qwen 3
0.65
0.65
0.68
0.64
0.66
Sonnet 4
0.83
0.80
0.70
0.62
0.68
0.67
Appendix
Table 8: Pairwise weighted κ for method_rung across 7 annotators.
Opus 4.5
GPT-5.2
DS-V3
Gemini 3F
Mistral L
Qwen 3
GPT-5.2
0.52
DS-V3
0.59
0.48
Gemini 3F
0.49
0.47
0.41
Mistral L
0.43
0.44
0.37
0.68
Qwen 3
0.53
0.55
0.47
0.57
0.63
Sonnet 4
0.71
0.56
0.55
0.52
0.44
0.60
Appendix
Table 9: Pairwise weighted κ for claim_rung across 7 annotators.
Opus 4.5
GPT-5.2
DS-V3
Gemini 3F
Mistral L
Qwen 3
GPT-5.2
0.61
DS-V3
0.57
0.53
Gemini 3F
0.59
0.57
0.42
Mistral L
0.53
0.55
0.39
0.66
Qwen 3
0.57
0.58
0.45
0.59
0.59
Sonnet 4
0.76
0.60
0.50
0.61
0.50
0.60
Appendix
Table 10: Pairwise weighted κ for gap_score across 7 annotators.
Paper type
N
αord
Applied/production
26
0.58
Circuit discovery
59
0.55
Evaluation/benchmark
16
0.61
Knowledge localization
10
0.64
Other
75
0.52
Appendix
Table 11: Krippendorff’s α (ordinal) for gap_score stratified by Paper type.
Model
N
Rˉm
Rˉc
gˉ
OC %
confˉ
Opus 4.5
186
1.69
2.22
0.54
41.9%
4.28
GPT-5.2
186
1.62
2.12
0.63
50.5%
4.19
DS-R1
166
1.73
2.16
0.51
43.4%
4.28
DS-V3
186
1.75
2.17
0.42
34.4%
4.09
Gemini 3 Flash
186
1.78
2.54
0.79
63.4%
4.99
Mistral Large
186
1.77
2.56
0.82
65.6%
4.55
Appendix
Table 12: Per-model annotation tendencies across all rated claims. Rˉm / Rˉc : mean method/claim rung; gˉ : mean gap score; OC: overclaim rate (gap >0 ).
Primary (7)
+R1 (8)
Variable
N
α
N
α
method_rung
186
0.66
166
0.67
claim_rung
186
0.56
166
0.55
gap_score
186
0.56
166
0.54
gap_binary
186
0.55
166
0.55
confidence
186
0.11
166
0.13
Appendix
Table 13: Krippendorff’s α (ordinal): 7 primary annotators vs. 8 annotators (including DeepSeek R1 supplementary on 166 claims).
Causal claims about large language model (LLM) internals rest on measurements. Those might include a projection, a cosine, an ablation delta, or an interchange patch among others. These measurements fail in specific, diagnosable ways that return a plausible number instead of an error, so a broken instrument can easily read as a finding. A covariance-matched null can saturate until every direction looks typical, a per-head attribution can overshoot the true residual write threefold on reordered-normalization architectures, an interchange patch can go sign-chaotic because its outcome is pinned at a ceiling, or a read-from verdict can be an artifact of measuring past the layer where the model already decided. This note documents six such failures from a causal interpretability program on refusal and moral representation, spanning several papers and a four-model open-weight panel; each mode is established on one or two of the four. For each we give the tell that catches it and a protocol keyed to a detectable trigger (reordered normalization, massive activations, a low-dimensional decision channel), so we and readers can check whether a given setup is exposed. The discipline reduces to four moves: calibrate against a positive-control ladder, certify with an orthogonal cell, compute power before spending compute, and state every read-from verdict at a depth referenced to the model's commitment. The evidence is four architectures across three families within a single program; external replication across programs is future work.
Causal discovery is a cornerstone of scientific reasoning, yet whether large language models can perform it reliably remains an open question. Recent benchmarks show that even fine-tuned models plateau on simple causal graphs and degrade as complexity grows, but why they fail has not been established. We prove the failure is fundamental: supervised fine-tuning, direct preference optimization, and in-context learning all produce predictors that cannot distinguish between causal graphs generating similar observational data, and any attempt to do so requires the model's internal representations to grow unboundedly, violating the very conditions under which these methods work. We formalize this as a kernel obstruction theorem, establishing that the limitation is intrinsic to the learning paradigm, not any particular model or dataset. We propose Agentic Causal Bayesian Optimization (A-CBO), wherein a frozen language model serves as an interventional oracle answering targeted queries about intervention effects, while an external Bayesian loop concentrates beliefs over candidate graphs in logarithmically many rounds. Because the decision operates outside the space where the obstruction applies, A-CBO provably converges while the underlying model remains unchanged. On Corr2Cause, A-CBO matches fine-tuned baselines without any training. On Extended Corr2Cause, a new benchmark scaling to 24 variables with 18K test samples, A-CBO significantly outperforms both fine-tuning and preference optimization, with the advantage growing
Amartya Roy, Sonali Parbhoo
SIRE, IIT Delhi and Robert Bosch GmbH, India · Imperial College London
Localizing behavior to individual components of a language model is a central goal of mechanistic interpretability. However, scoring components one at a time misses context-dependent effects: a primary component can inhibit the activation of a backup, leading to issues with ranking components. Actual causality studies the structure of such interactions via witnesses: variables that provide contextual information to resolve interaction terms. However, estimation with witnesses typically requires combinatorial enumeration and is infeasible in practice. We introduce the witness-integrated set effect (WISE), a family of causal estimands that build on the witness mechanism while taking expectations over sets of causes and witnesses to remain computationally feasible. Building on this approach, we introduce JuntaLearner, a gradient-based circuit discovery method that learns to rank components by their causal impact across varying-sized sets of components and witnesses. Alongside faithfulness metrics, we introduce measures of necessity and task specificity, and the circuit recognition score (CRS) to summarize each metric across circuit sizes while emphasizing effects achieved by small circuits. Across tasks and models of increasing size, JuntaLearner achieves higher mean CRS compared to attribution baselines on all metrics. Since its cost does not grow with the number of candidate components, JuntaLearner scales to large models while accounting for set-level interactions and avoiding first-order approximations.