Causal Interpretability

Ask about this topic

Overview

Causal interpretability asks not just what information a model represents, but what changes its behavior when that information or a component is altered. Researchers use activation patching and ablation, causal mediation, sparse autoencoders, circuit discovery, and counterfactual feature attribution to trace effects through models. These approaches are applied to language-model reasoning, vision and multimodal systems, and scientific or time-series predictors. A central challenge is showing that an explanation reflects the mechanism used on natural inputs: a feature can be easy to decode without affecting the output, and causal conclusions depend on intervention design, assumptions, and robustness checks.

Latest papers

CardsList

107 papers7 papers in the last four weeks, down 22% on the four weeks before. 0.0% of all new papers.

Mar 11, 2026

Attention Gathers, MLPs Compose: A Causal Analysis of an Action-Outcome Circuit in VideoViT

The paper explores how video models trained for classification tasks represent nuanced, hidden semantic information that may not affect the final outcome, a key challenge for Trustworthy AI models. Through Explainable and Interpretable AI methods, specifically mechanistic interpretability techniques, the internal circuit responsible for representing the action's outcome is reverse-engineered in a pre-trained video vision transformer, revealing that the "Success vs Failure" signal is computed through a distinct amplification cascade. While there are low-level differences observed from layer 0, the abstract and semantic representation of the outcome is progressively amplified from layers 5 through 11. Causal analysis, primarily using activation patching supported by ablation results, reveals a clear division of labor: Attention Heads act as "evidence gatherers", providing necessary low-level information for partial signal recovery, while MLP Blocks function as robust "concept composers", each of which is the primary driver to generate the "success" signal. This distributed and redundant circuit in the model's internals explains its resilience to simple ablations, demonstrating a core computational pattern for processing human-action outcomes. Crucially, the existence of this sophisticated circuit for representing complex outcomes, even within a model trained only for simple classification, highlights the potential for models to develop forms of 'hidden knowledge' beyond their explicit task, underscoring the need for mechanistic oversight for building genuinely Explainable and Trustworthy AI systems intended for deployment.

Sai V R Chereddy

Independent Researcher

Transformer InterpretabilityVideo Representation LearningMechanistic InterpretabilityCausal Interpretability
AskPDF
Mar 11, 2026

Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning

Sparse autoencoders can localize where concepts live in language models, but not how they interact during multi-step reasoning. We propose Causal Concept Graphs (CCG): a directed acyclic graph over sparse, interpretable latent features, where edges capture learned causal dependencies between concepts. We combine task-conditioned sparse autoencoders for concept discovery with DAGMA-style differentiable structure learning for graph recovery and introduce the Causal Fidelity Score (CFS) to evaluate whether graph-guided interventions induce larger downstream effects than random ones. On ARC-Challenge, StrategyQA, and LogiQA with GPT-2 Medium, across five seeds (n=15n{=}15 paired runs), CCG achieves \CFS=5.654±0.625\CFS=5.654\pm0.625, outperforming ROME-style tracing (3.382±0.2333.382\pm0.233), SAE-only ranking (2.479±0.1962.479\pm0.196), and a random baseline (1.032±0.0341.032\pm0.034), with p<0.0001p<0.0001 after Bonferroni correction. Learned graphs are sparse (5-6% edge density), domain-specific, and stable across seeds.

Md Muntaqim Meherab, Noor Islam S. Mohammad, Faiza Feroz

Daffodil International University, Dhaka, Bangladesh · New York University, Brooklyn, NY, USA

LLM Reasoning with GraphsCausal InterpretabilityCausal Interventions in Language ModelsCausal Structure Learning
AskPDF
Mar 8, 2026

Interpretable-by-Design Transformers via Architectural Stream Independence

While transformers achieve strong performance, their internal decision-making processes remain opaque. We investigate whether architectural constraints can enforce interpretability by design through architectural stream independence: maintaining a token stream (carrying symbolic structure) and contextual semantics in separated streams that remain independently observable throughout processing, with integration delayed until output. We validate this principle through the Late Fusion Architecture (LFA), which demonstrates interpretable symbolic heads through all the final layers, while standard transformers show dissolution by the third of six layers; we quantify this effect by introducing the Token-Position Dependence Score (PDS), with PDSmaxPDS_{max} = 0.276 and 0.058, respectively. Crucially, intervention experiments demonstrate functional modularity: suppressing LFA's recency heads causes minimal semantic damage (Cohen's d = -0.158) versus catastrophic entanglement in baselines (d = -0.672). LFA demonstrates that architectural constraints improve underlying learning mechanisms, averaging 42% stability versus 19% and 11% for baseline comparisons, with extremes from 50% on LFA's best pairs (6 of 12 heads position-invariant) down to 0% complete collapse in over-constrained cases. By preventing premature entanglement, architectural independence steers models toward semantic understanding over positional heuristics, establishing interpretability as an architectural design criterion enforceable through structural constraints rather than post-hoc analysis.

Clayton Kerce, Alexis Fox

Georgia Tech Research Institute

Transformer InterpretabilityNeural Network InterpretabilityMechanistic InterpretabilityCausal InterpretabilityAttention Head Analysis
AskPDF
Mar 6, 2026

Causal Interpretation of Neural Network Computations with Contribution Decomposition

Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct approach to examine how hidden neurons act to drive network outputs. We introduce CODEC (Contribution Decomposition), a method that uses sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions, revealing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-classification networks, we find that contributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects on network outputs. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human-interpretable visualizations of distinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity in the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.

Joshua Brendan Melander, Zaki Alaoui, Shenghua Liu +2

Department of Neurobiology, Stanford University · Department of Physics, Stanford University · Department of Applied Physics, Stanford University

Neural Network InterpretabilityMechanistic InterpretabilityNeuron-Level InterpretabilityCausal InterpretabilityFeature Attribution
AskPDF
Mar 2, 2026

Causal Circuit Tracing Reveals Distinct Computational Architectures in Single-Cell Foundation Models: Inhibitory Dominance, Biological Coherence, and Cross-Model Convergence

Motivation: Sparse autoencoders (SAEs) decompose foundation model activations into interpretable features, but causal feature-to-feature interactions across network depth remain unknown for biological foundation models. Results: We introduce causal circuit tracing by ablating SAE features and measuring downstream responses, and apply it to Geneformer V2-316M and scGPT whole-human across four conditions (96,892 edges, 80,191 forward passes). Both models show approximately 53 percent biological coherence and 65 to 89 percent inhibitory dominance, invariant to architecture and cell type. scGPT produces stronger effects (mean absolute d = 1.40 vs. 1.05) with more balanced dynamics. Cross-model consensus yields 1,142 conserved domain pairs (10.6x enrichment, p < 0.001). Disease-associated domains are 3.59x more likely to be consensus. Gene-level CRISPRi validation shows 56.4 percent directional accuracy, confirming co-expression rather than causal encoding.

Ihor Kendiukhov

Department of Computer Science, University of Tübingen, Tübingen, Germany

Mechanistic InterpretabilityCausal Interpretability
AskPDF
Feb 28, 2026

CIRCUS: Circuit Consensus under Uncertainty via Stability Ensembles

Every mechanistic circuit carries an invisible asterisk: it reflects not just the model's computation, but the analyst's choice of pruning threshold. Change that choice and the circuit changes, yet current practice treats a single pruned subgraph as ground truth with no way to distinguish robust structure from threshold artifacts. We introduce CIRCUS, which reframes circuit discovery as a problem of uncertainty over explanations. CIRCUS prunes one attribution graph under B configurations, assigns each edge an empirical inclusion frequency s(e) in [0,1] measuring how robustly it survives across the configuration family, and extracts a consensus circuit of edges present in every view. This yields a principled core/contingent/noise decomposition (analogous to posterior model-inclusion indicators in Bayesian variable selection) that separates robust structure from threshold-sensitive artifacts, with negligible overhead. On Gemma-2-2B and Llama-3.2-1B, consensus circuits are 40x smaller than the union of all configurations while retaining comparable influence-flow explanatory power, consistently outperform influence-ranked and random baselines, and are confirmed causally relevant by activation patching.

Swapnil Parekh

Intuit

Circuit DiscoveryCircuits in Language ModelsMechanistic InterpretabilityUncertainty QuantificationCausal Interpretability
AskPDF
Feb 26, 2026

Inferential Mechanics Part 1: Causal Mechanistic Theories of Machine Learning in Chemical Biology with Implications

Machine learning techniques are now routinely encountered in research laboratories across the globe. Impressive progress has been made through ML and AI techniques with regards to large data set processing. This progress has increased the ability of the experimenter to digest data and make novel predictions regarding phenomena of interest. However, machine learning predictors generated from data sets taken from the natural sciences are often treated as black boxes which are used broadly and generally without detailed consideration of the causal structure of the data set of interest. Work has been attempted to bring causality into discussions of machine learning models of natural phenomena; however, a firm and unified theoretical treatment is lacking. This series of three papers explores the union of chemical theory, biological theory, probability theory and causality that will correct current causal flaws of machine learning in the natural sciences. This paper, Part 1 of the series, provides the formal framework of the foundational causal structure of phenomena in chemical biology and is extended to machine learning through the novel concept of focus, defined here as the ability of a machine learning algorithm to narrow down to a hidden underpinning mechanism in large data sets. Initial proof of these principles on a family of Akt inhibitors is also provided. The second paper containing Part 2 will provide a formal exploration of chemical similarity, and Part 3 will present extensive experimental evidence of how hidden causal structures weaken all machine learning in chemical biology. This series serves to establish for chemical biology a new kind of mathematical framework for modeling mechanisms in Nature without the need for the tools of reductionism: inferential mechanics.

Ilya Balabin, Thomas M. Kaiser

Avicenna Biosciences, Inc., Durham, NC, USA

Causal InterpretabilityStructural Causal Models
AskPDF
Feb 24, 2026

Leveraging Causal Reasoning Method for Explaining Medical Image Segmentation Models

Medical image segmentation plays a vital role in clinical decision-making, enabling precise localization of lesions and guiding interventions. Despite significant advances in segmentation accuracy, the black-box nature of most deep models has raised growing concerns about their trustworthiness in high-stakes medical scenarios. Current explanation techniques have primarily focused on classification tasks, leaving the segmentation domain relatively underexplored. We introduced an explanation model for segmentation task which employs the causal inference framework and backpropagates the average treatment effect (ATE) into a quantification metric to determine the influence of input regions, as well as network components, on target segmentation areas. Through comparison with recent segmentation explainability techniques on two representative medical imaging datasets, we demonstrated that our approach provides more faithful explanations than existing approaches. Furthermore, we carried out a systematic causal analysis of multiple foundational segmentation models using our method, which reveals significant heterogeneity in perceptual strategies across different models, and even between different inputs for the same model. Suggesting the potential of our method to provide notable insights for optimizing segmentation models. Our code can be found at https://github.com/lcmmai/PdCR.

Limai Jiang, Ruitao Xie, Bokai Yang +6

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Shenzhen University of Advanced Technology +3

Explainable Medical Image AnalysisMedical Image SegmentationCausal Interpretability
AskPDF
Feb 19, 2026

Causality by Abstraction: Symbolic Rule Learning in Multivariate Timeseries with Large Language Models

Inferring causal relations in timeseries data with delayed effects is a fundamental challenge, especially when the underlying system exhibits complex dynamics that cannot be captured by simple functional mappings. Traditional approaches often fail to produce generalized and interpretable explanations, as multiple distinct input trajectories may yield nearly indistinguishable outputs. In this work, we present ruleXplain, a framework that leverages Large Language Models (LLMs) to extract formal explanations for input-output relations in simulation-driven dynamical systems. Our method introduces a constrained symbolic rule language with temporal operators and delay semantics, enabling LLMs to generate verifiable causal rules through structured prompting. ruleXplain relies on the availability of a principled model (e.g., a simulator) that maps multivariate input time series to output time series. Within ruleXplain, the simulator is used to generate diverse counterfactual input trajectories that yield similar target output, serving as candidate explanations. Such counterfactual inputs are clustered and provided as context to the LLM, which is tasked with the generation of symbolic rules encoding the joint temporal trends responsible for the patterns observable in the output times series. A closed-loop refinement process ensures rule consistency and semantic validity. We validate the framework using the PySIRTEM epidemic simulator, mapping testing rate inputs to daily infection counts; and the EnergyPlus building energy simulator, observing temperature and solar irradiance inputs to electricity needs. For validation, we perform three classes of experiments: (1) the efficacy of the ruleset through input reconstruction; (2) ablation studies evaluating the causal encoding of the ruleset; and (3) generalization tests of the extracted rules across unseen output trends with varying phase dynamics.

Preetom Biswas, Giulia Pedrielli, K. Selçuk Candan

Arizona State University, Tempe, AZ, USA

Causal InterpretabilityTime-Series Causal Discovery
AskPDF
Feb 18, 2026

Causality is Key for Interpretability Claims to Generalise

Interpretability research on large language models (LLMs) has yielded important insights into model behaviour, yet recurring pitfalls persist: findings that do not generalise, and causal interpretations that outrun the evidence. Our position is that causal inference specifies what constitutes a valid mapping from model activations to invariant high-level structures, the data or assumptions needed to achieve it, and the inferences it can support. Specifically, Pearl's causal hierarchy clarifies what an interpretability study can justify. Observations establish associations between model behaviour and internal components. Interventions (e.g., ablations or activation patching) support claims how these edits affect a behavioural metric (e.g., average change in token probabilities) over a set of prompts. However, counterfactual claims -- i.e., asking what the model output would have been for the same prompt under an unobserved intervention -- remain largely unverifiable without controlled supervision. We show how causal representation learning (CRL) operationalises this hierarchy, specifying which variables are recoverable from activations and under what assumptions. Together, these motivate a diagnostic framework that helps practitioners select methods and evaluations matching claims to evidence such that findings generalise.

Shruti Joshi, Aaron Mueller, David Klindt +3

Mila - Québec AI Institute & Université de Montréal · Boston University · Cold Spring Harbor Laboratory +1

LLM InterpretabilityCausal InterpretabilityCausal InferenceCausal Interventions in Language Models
AskPDF
Feb 17, 2026

Activation Steering via Generative Causal Mediation

Where should we intervene in a language model (LM) to localize and control behaviors that are diffused across many tokens of a long-form response? We introduce Generative Causal Mediation (GCM), a procedure for selecting model components (e.g., attention heads) from contrastive long-form responses, to steer such diffuse concepts (e.g., talk in verse vs. talk in prose). In GCM, we first construct a dataset of contrasting behavioral inputs and long-form responses. Then, we quantify how model components mediate the concept and select the strongest mediators for steering. We evaluate GCM on three behaviors--refusal, sycophancy, and style transfer--across three language models. GCM successfully localizes concepts expressed in long-form responses and outperforms correlational probe-based baselines when steering with a sparse set of attention heads. Together, these results demonstrate that GCM provides an effective approach for localizing from and controlling the long-form responses of LMs.

Aruna Sankaranarayanan, Amir Zur, Atticus Geiger +1

CSAIL, MIT · Pr(AI)2R · Stanford University +1

Language Model SteeringCausal Interventions in Language ModelsCausal Interpretability
AskPDF
Feb 16, 2026

A unified framework for evaluating the robustness of machine-learning interpretability for prospect risking

In geophysics, hydrocarbon prospect risking involves assessing the risks associated with hydrocarbon exploration by integrating data from various sources. Machine learning-based classifiers trained on tabular data have been recently used to make faster decisions on these prospects. The lack of transparency in the decision-making processes of such models has led to the emergence of explainable AI (XAI). LIME and SHAP are two such examples of these XAI methods which try to generate explanations of a particular decision by ranking the input features in terms of importance. However, explanations of the same scenario generated by these two different explanation strategies have shown to disagree or be different, particularly for complex data. This is because the definitions of "importance" and "relevance" differ for different explanation strategies. Thus, grounding these ranked features using theoretically backed causal ideas of necessity and sufficiency can prove to be a more reliable and robust way to improve the trustworthiness of the concerned explanation strategies.We propose a unified framework to generate counterfactuals as well as quantify necessity and sufficiency and use these to perform a robustness evaluation of the explanations provided by LIME and SHAP on high dimensional structured prospect risking data. This robustness test gives us deeper insights into the models capabilities to handle erronous data and which XAI module works best in pair with which model for our dataset for hydorcarbon indication.

Prithwijit Chowdhury, Ahmad Mustafa, Mohit Prabhushankar +1

Center for Energy and Geo Processing (CeGP), Omni Lab for Intelligent Visual Engineering and Science (OLIVES), School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA.

Explainability EvaluationExplainable Artificial IntelligenceExplanation StabilitySHAP Feature AttributionCausal Interpretability
AskPDF
Feb 6, 2026

ReBeCA: Causality-Inspired Behavior Analysis for Understanding the Self-Reflection of Language Models

Self-reflection is a basic self-improvement skill of language agents, yet it does not reliably improve their outputs, where the underlying behavioral mechanism remains opaque. We introduce ReBeCA (self-Reflection Behavior explained through Causality-inspired Analysis), a framework for identifying and validating outcome-relevant reflective behaviors from recorded trajectories. ReBeCA uses reflective codes, an inductive coding schema, to represent each trajectory as a sequence of reflective behaviors. It learns dependencies among these behaviors and the outcome with causal discovery, selects the behaviors that best predict the outcome of held-out trajectories, and tests them for consistency across data subsets and by prompt-based steering. In a controlled case study of four Qwen3 models under fixed-round Self-Refine on mathematical reasoning and translation, ReBeCA identifies compact, round-specific behavioral sets with superior held-out predictive fit and uses prompt-based steering on unseen problems to reveal how individual and combined behavioral directives affect performance (Holm-adjusted p=.020p = .020 on Qwen3-4B). ReBeCA offers a reusable framework that turns agent trajectories into interpretable, verifiable behavioral hypotheses without requiring model internals or preselecting the behaviors to test.

Tianqiang Yan, Lizhen Qu, Sihan Shang +6

Monash University · Harbin Institute of Technology, Shenzhen · Hainan University +2

LLM Self-RefinementCausal InterpretabilityCausal Reasoning in Language ModelsCausal Interventions in Language Models
AskPDF
Feb 3, 2026

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

Chain-of-thought (CoT) prompting is widely used as a reasoning aid and is often treated as a transparency mechanism. Yet behavioral gains under CoT do not imply that the model's internal computation causally depends on the emitted reasoning text, i.e. models may produce fluent rationales while routing decision-critical computation through latent pathways. We introduce a causal, layerwise audit of CoT faithfulness based on activation patching. Our key metric, the CoT Mediation Index (CMI), isolates CoT-specific causal influence by comparing performance degradation from patching CoT-token hidden states against matched control patches. Across multiple model families (Phi, Qwen, DialoGPT) and scales, we find that CoT-specific influence is typically depth-localized into narrow ''reasoning windows,'' and we identify bypass regimes where CMI is near-zero despite plausible CoT text. We further observe that models tuned explicitly for reasoning tend to exhibit stronger and more structured mediation than larger untuned counterparts, while Mixture-of-Experts models show more distributed mediation consistent with routing-based computation. Overall, our results show that CoT faithfulness varies substantially across models and tasks and cannot be inferred from behavior alone, motivating causal, layerwise audits when using CoT as a transparency signal.

Anish Sathyanarayanan, Aditya Nagarsekar, Aarush Rathore

BITS Pilani, K. K. Birla Goa Campus

CoT FaithfulnessFaithfulness of Language Model ExplanationsLLM AuditingImplicit Reasoning in Language ModelsCausal InterpretabilityCausal Interventions in Language Models+2
AskPDF
Oct 14, 2025

Attribution Graphs and Causal Probing for Mechanistic Discovery and Bias Repair in Multimodal Generative Learning

We treat the internals of generative models as mechanistic objects rather than black boxes. We introduce Attribution Graphs (AGs), which extend GradCAM++ to circuit-level representations, and Causal Probing, a do-calculus intervention method for identifying causal latent structures, enabling detection and correction of spurious correlations, demographic biases, and misaligned decision circuits during training. We further propose the Cognitive Alignment Score (CAS), quantifying agreement between model-internal representations and human concepts, a saliency-first privacy mechanism sharing only thresholded attribution nodes, a bias-aware regularizer aligning subgroup statistics, and a Reveal-to-Revise loop integrating attribution signals into parameter updates without separate fine-tuning. Evaluated on CelebA, FairFace, Jigsaw, and HateXplain, our method achieves 94.1% accuracy, 92.3% macro F1, 79.4% IoU-XAI, and 12.7 FID at 72--76% adversarial robustness, while reducing subgroup disparity ΔbiasΔ_{\mathrm{bias}} by 41%, demonstrating that mechanistic interpretability, fairness, and generative performance can be jointly optimized.

Noor Islam S. Mohammad, Uluğ Bayazıt

1,2Istanbul Technical University

Gradient-Based AttributionCausal Representation LearningAlgorithmic FairnessCausal AttributionMechanistic InterpretabilityAlgorithmic Bias+2
AskPDF
Mar 25, 2025

Quantifying Symptom Causality in Clinical Decision Making: An Exploration Using CausaLM

Current machine learning approaches to medical diagnosis often rely on correlational patterns between symptoms and diseases, risking misdiagnoses when symptoms are ambiguous or common across multiple conditions. In this work, we move beyond correlation to investigate the causal influence of key symptoms-specifically "chest pain" on diagnostic predictions. Leveraging the CausaLM framework, we generate counterfactual text representations in which target concepts are effectively "forgotten" enabling a principled estimation of the causal effect of that concept on a model's predicted disease distribution. By employing Textual Representation-based Average Treatment Effect (TReATE), we quantify how the presence or absence of a symptom shapes the model's diagnostic outcomes, and contrast these findings against correlation-based baselines such as CONEXP. Our results offer deeper insight into the decision-making behavior of clinical NLP models and have the potential to inform more trustworthy, interpretable, and causally-grounded decision support tools in medical practice.

Alexander Romanus, Mehul Shetty, Connor Jordan

Clinical Decision-MakingCausal InterpretabilityCausal InferenceCausal Interventions in Language Models
AskPDF
Nov 13, 2024

Causal Explanations for Image Classifiers

Existing algorithms for explaining the output of image classifiers use different definitions of explanations and a variety of techniques to find them. However, none of the existing tools use a principled approach based on formal definitions of cause and explanation. In this paper we present a novel black-box approach to computing explanations grounded in the theory of actual causality. We prove relevant theoretical results and present an algorithm for computing approximate explanations based on these definitions. We prove termination of our algorithm and discuss its complexity and the amount of approximation compared to the precise definition. We implemented the framework in a tool ReX and we present experimental results and a comparison with state-of-the-art tools. We demonstrate that ReX is the most efficient black-box tool and produces the smallest explanations, in addition to outperforming other black-box tools on standard quality measures.

Hana Chockler, David A. Kelly, Daniel Kroening +1

King’s College London, London, UK · Amazon.com, Inc.,, USA · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE +1

Actual CausalityExplainable Image ClassificationCausal ReasoningCausal ExplanationCausal InterpretabilityExplainability Evaluation
AskPDF