Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety

Fazl Barez; Tung-Yu Wu; Hunar Batra; Gabriele Dominici; Florent Draye; Yanai Elazar; Damiano Fornasiere; Antía García

cs.CLNov 5, 2026

Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety

Authors: Fazl Barez, Tung-Yu Wu, Hunar Batra, Gabriele Dominici, Florent Draye, Yanai Elazar, Damiano Fornasiere, Antía García,

Organizations: University of Oxford · Martian · Università della Svizzera italiana · Max Planck Institute for Intelligent Systems ·

Abstract

Neural networks are likely to be among the most consequential artefacts in human history. As these systems become more capable and autonomous, their potential to cause societal harm also grows, making AI safety one of the defining challenges of our time. Yet we continue to build increasingly powerful systems without fully understanding how they work, how they will behave in unfamiliar situations, or how to reliably control them. Mechanistic interpretability offers a path towards understanding these systems by studying their internal representations and computations, with the aim of evaluating, predicting, and controlling behaviour. We survey mechanistic interpretability for AI safety in order to answer the question: Does understanding how models work actually make them safer? We place mechanistic interpretability in the broader context of interpretability research and compare its mechanistic evidence with behavioural evaluation. We introduce the foundations of mechanistic interpretability, including features, polysemanticity, transformer computation, and circuits, and survey methods for feature extraction and analysis, circuit discovery and causal verification, explaining intermediate representations in natural language, learning dynamics, and training-data attribution. Beyond surveying existing methods, we propose a unified framework for evaluating mechanistic interpretability claims and characterise six research orientations by their goals, assumptions, and trade-offs. Finally, we examine how mechanistic understanding can support AI safety, including knowledge editing, steering, and monitoring, and assess its limitations, failure modes, open problems, and relationship to behavioural safety measures. We also release a repository of code notebooks with worked examples that reproduce key findings from surveyed methods, allowing readers to build intuition through hands-on experimentation.

Figures & tables

Figure 2: Overview of the conceptual flow of the survey, categorised into four primary pillars: (1) Motivation & Concepts, covering fundamental knowledge and core hypotheses; (2) Technical Methods, detailing approaches developed and used in interpretability research; (3) Research Directions, exploring main directions the field is taking; and (4) Safety & Open Problems, identifying critical challenges and deployment-oriented interventions for AI alignment.

Dimension	Concept Bottleneck Models	Interpretable-by-Design Models
Constraint type	Semantic: predefined human concepts	Structural: sparsity, modularity, compositionality
Feature discovery	Not possible—concepts fixed a priori	Enabled—model discovers features from data
Interpretability guarantee	Explanations in terms of specified concepts	Circuits and features are structurally accessible
Accuracy trade-off	Degrades when bottleneck omits relevant features	Degrades with sparsity, but recoverable via scaling

Table 1: Concept Bottleneck Models (CBMs) vs. Interpretable-by-Design Models

Method	Core Idea	Strengths	Intended Use and Limitations
Behavioural (e.g., red-teaming, benchmarks)	Examine model outputs in response to inputs	Fast, scalable, practical for broad safety coverage	Not designed to reveal internal computation; suitable for monitoring, triage, and regression detection
Input attribution (e.g., SHAP, saliency)	Identify which inputs most influence outputs	Token-level explanations; scalable	Focuses on input–output influence, not internal mechanisms; useful for auditing and debugging
Concept-based (e.g., CBMs, probes)	Correlationally analyse or constrain interpretable representations	Reveals encoded information; transparent abstractions; enables targeted intervention	Not mechanistic without causal intervention; supports localisation, auditing, and early detection tasks
Mechanistic (e.g., circuits, SAEs)	Analyse internal computations and causal structure	Reveals algorithms; enables targeted intervention	Computationally intensive; complements other methods when causal understanding is required

Table 2: Trade-offs and characteristics of interpretability methods.

Figure 3: Circuit. This diagram illustrates a circuit within a transformer, where specific nodes (MLP layers and attention heads in this case) perform specialised computations.

Figure 4: Privileged versus non-privileged bases. In the hidden layers of MLP blocks, element-wise nonlinear operations break rotational symmetry and make individual axes meaningful. In contrast, the residual stream contains no such operations that privilege particular axes, so rotating the basis preserves the relevant computation.

Figure 5: Superposition implies polysemanticity. Left: when the number of features

m

does not exceed the number of dimensions

d

, i.e.,

m\leq d

, feature directions can align cleanly with individual neuron axes, producing monosemantic neurons. Right: when

m>d

, multiple feature directions must share the same limited set of neuron axes; as a result, each neuron axis cuts across multiple feature directions, producing polysemantic neurons.

Figure 6: Sparse Autoencoder (SAE). SAEs are trained to reconstruct activations of a model component at a specific layer

\ell

h_{\ell}

, into an overcomplete, sparse latent space

z

, such that the reconstructed activation

\hat{h_{\ell}}\approx h_{\ell}

. By enforcing sparsity on

z

, the model decomposes dense, polysemantic representations into monosemantic, human-interpretable features.

Layer Type	Variants Used	Reported Basis	Reported Caveats
Residual stream (early layers)	Gated SAE, Top- $k$	Activations are closer to input embeddings; moderate sparsity	Smaller expansion factors (2–4 $\times$ ) reported
Residual stream (middle layers)	Gated SAE, JumpReLU	Richest feature structure; highest interpretability potential	Standard expansion factor (4–8 $\times$ ); most studied regime
Residual stream (late layers)	Top- $k$ , Gated SAE	Activations increasingly task-specific; may be less sparse	Larger expansion factors (8–16 $\times$ ) reported
MLP outputs	Transcoder (Section 6.6 )	MLP applies a nonlinearity a single-layer SAE cannot span	Not a sparse decomposition on a static object; bridges input to output $\ell$
Attention outputs	Gated SAE, standard SAE	Attention outputs often lower-rank	Least studied of the three sites; may benefit from smaller dictionaries
Cross-layer	Crosscoder, CLT (Section 6.6 )	Tracks feature evolution; resolves cross-layer superposition	High compute cost; requires multi-layer training

Table 3: Heuristic recommendations for selecting SAE variants across different layer types. Recommendations are based on current empirical evidence and practical experience, and may evolve as SAE methodologies and evaluation standards continue to develop.

Variant	Reported Finding	Setting
Standard	Sparse dictionary learning by $L_{1}$ regularisation recovers features more monosemantic than individual neurons; subject to shrinkage bias	Establishing work, small models ( Cunningham et al., 2024 ; Bricken et al., 2023 )
Gated	Separating detection from magnitude estimation addresses $L_{1}$ shrinkage at minimal added complexity	Reconstruction–sparsity improvement at matched sparsity, Gemma-family ( Rajamanoharan et al., 2024a )
JumpReLU	Learned per-feature thresholds trained with straight-through estimators give the strongest reported reconstruction	Matched-sparsity comparison, Gemma-family ( Rajamanoharan et al., 2024b ) ; needs most tuning ( Bussmann et al., 2024 )
Top- $k$	A hard $\\|z\\|_{0}\leq k$ constraint removes the sparsity coefficient from the search space	Scaling behaviour characterised on large models; $k=32$ – $150$ commonly reported ( Gao et al., 2025 ; Karvonen et al., 2025 )
Batch Top- $k$	A batch-level sparsity budget lets the number of active features vary per input, improving reconstruction at matched average sparsity	Introduced as a relaxation of per-example Top- $k$ , achieving better reconstruction ( Bussmann et al., 2024 )
Transcoder	Learning the MLP input–output map avoids spanning the nonlinearity with a single dictionary	Reported to perform at least on par with SAEs ( Dunefsky et al., 2024 ; Paulo et al., 2025c )

Table 4: Reported evidence for each SAE variant, with the setting in which it was obtained.

Condition	Description
Architecture dependence	Most evidence derives from Llama- and Gemma-family models. Generality of the findings across model families remains unestablished.
Task dependence	The properties that matter may differ across analysis goals, such as safety auditing, behaviour understanding, and knowledge localisation, but existing evaluations often do not distinguish between them.
Rapid iteration	New architectures and training procedures are proposed faster than existing ones are sufficiently studied and replicated.
Immature theoretical analysis	Current findings are mostly empirical and may be superseded. Understanding of why particular variants succeed or fail is incomplete.
Feature reproducibility	SAEs trained on the same data with different random seeds can recover substantially different features. When between-variant performance differences are smaller than within-variant seed-to-seed variability, we may not confidently attribute them to architectural factors. However, few reported comparisons have rigorously accounted for this concern.

Table 5: Conditions for the reported evidence on SAE variant behaviour.

Panel of Figure 7 — Figure 7: Transcoders and CLTs. Transcoders reconstruct the output of the MLP from its input in the same transformer layer. CLTs extend this by partially reconstructing the outputs of multiple subsequent layers from a single point in the residual stream.

Figure 8: Parameter count (left) and FP16 memory requirements (right) of a full cross-layer transcoder as a function of model depth, for different residual stream dimensions and expansion factor

e=32

Figure 9: Crosscoder. Crosscoders are trained to reconstruct the residual stream across multiple layers simultaneously.

Method	Assumption	Strengths	Weaknesses	Best Use Case
SAE	Sparse, linear features	Open-ended discovery; fast inference	Hyperparameter-sensitive; identifiability issues	Open-ended feature discovery
NMF	Non-negative, additive parts	Parts-based interpretation; uniqueness guarantees	Requires non-negative data; limited sparsity	Naturally non-negative domains
PCA	Variance aligns with features	Closed form; optimal low-rank reconstruction	Explains global variance and produces orthogonal features	Small-scale settings; as a baseline method
ICA	Complete and independent sources	Theoretical guarantees when assumptions hold	Assumes independence; Gaussian blindness	When independence holds
Supervised methods	Known concept labels	High performance on targeted concepts	Requires labels; no discovery	When certain concepts of interest are known
K-SVD	Sparse, linear features	Explicitly solving the sparse-coding problem; no encoder bias	Slow inference; no learned encoder	Small, fixed sets of activations; as a baseline method

Table 6: Comparison of feature extraction methods.

Method	Approach	Interpretability Gain	Primary Limitation
Feature-Sparse Training (SoLU)	Softmax-gated activation function	Increased neuron monosemanticity	Layer normalisation smuggling displaces superposition
Weight-Sparse Training	$L_{0}$ constraint on parameters	More monosemantic neurons and much smaller circuits	Training instability and scalability
Attention-Sparse Training	$L_{0}$ regularisation on attention	Makes attention patterns interpretable	Requires fine-tuning on a trained language model
MOLT	Sparse mixture of linear transforms	Reveals implemented operations directly	Complexity of matrix analysis
KANs	Learnable splines on edges	Symbolic formula extraction	Scalability to LLMs is unproven

Table 7: Comparison of interpretability-by-design methods.

Method	Core Idea	Strengths	Limitations
Activation Patching	Intervene on components and measure the resulting output change	Simple and intuitive; provides direct causal evidence	Requires many interventions; component-level granularity can be coarse
Path Patching	Intervene on individual information paths or edges and measure their causal effects	Finer-grained localisation of information flow	Computationally expensive due to the large number of candidate paths
ACDC	Greedily prune edges from the computational graph while testing whether the target behaviour is preserved	Fully automated; directly searches for a behaviourally sufficient subgraph	Computationally expensive due to sequential edge pruning; sensitive to pruning order and threshold choices
EAP	Approximate edge-level attribution using path patching and first-order gradients, enabling parallel evaluation of many candidate edges	Highly parallelisable and computationally efficient; enables large-scale edge attribution	Relies on a first-order approximation; attribution scores may not fully capture nonlinear or interaction effects
SAE- or CLT-Based	Discover circuits over sparse, interpretable features rather than raw model components	Leverages interpretable features; enables finer-grained feature-level circuit analysis	Depends on feature quality; requires additional computation to train and analyse SAEs/CLTs

Table 8: Trade-offs and characteristics of circuit discovery methods.

Dimension	Definition	Metrics	Methods
Attribution Faithfulness	Do attribution scores accurately reflect the causal effects of the identified components?	Intervention-effect agreement; effect-size error; counterfactual effect recovery	Ablation; activation / path patching; interchange interventions
Completeness	How much of the behaviour is accounted for by the identified components?	Loss recovery; behavioural coverage; unexplained variance	Reconstruction; behavioural evaluation; residual analysis
Stability	Are findings robust to variations in factors like random seeds, hyperparameters, paraphrases, etc.?	Cross-seed agreement; checkpoint consistency; hyperparameter sensitivity	Multi-seed analysis; checkpoint comparison; input paraphrasing
Causal Validity	Are the identified components necessary, sufficient, and specific for the target behaviour?	Necessity score; sufficiency score; specificity score	Ablation; component isolation; interchange interventions; tests with control behaviours
Human Interpretability	Can humans understand and use the resulting explanations?	Explanation utility; debugging utility; interpretation agreement	User studies; expert evaluation

Table 9: Unified evaluation framework for mechanistic interpretability.

Method	Attribution Faithfulness	Completeness	Stability	Causal Validity	Human Interpretability
SAEs and variants	Feature attribution–intervention agreement; effect-size error	Activation reconstruction; loss recovery; unexplained variance	Cross-seed feature alignment	Feature necessity, sufficiency, and specificity	Feature description consistency across annotations, including models and humans
Circuit Discovery	Edge or component attribution–intervention agreement; counterfactual recovery	Behavioural coverage; loss or behaviour recovery	Robustness across hyperparameters, seeds, and paraphrases	Circuit necessity, sufficiency, specificity; interchange tests	Circuit-level descriptions; trade-off between sparsity and variance explained
Probing	Agreement between probe-derived importance and intervention effects	Coverage across target concepts or behaviours under different distributions	Robustness across seeds and sampled fitting sets	Necessity, specificity, and intervention tests on probe directions	Explanation of shared information in fitting set and search for confounding factors
Steering Vectors	Agreement between predicted and observed intervention effects	Coverage across behaviours deemed relevant	Stability across steering magnitudes, seeds, and paraphrases	Intervention efficacy and specificity; side-effect analysis	Explanation of shared information in fitting set and search for confounding factors

Table 10: Recommended evaluation priorities by interpretability method. The entries indicate evaluation aspects particularly important for each method, not mutually exclusive metrics.

Figure 10: Logit Lens. Logit lens provides a window into a model’s intermediate state by applying the final unembedding layer directly to the residual stream at earlier layers (

h_{1},h_{2},\ldots

). This allows users to see which tokens the model is “considering” mid-computation, such as the emergence of the concept “CAT” before the final output layer is reached.

Figure 11: Patchscopes. Patchscopes extracts the model representation of interest from a source prompt and “patches” it into an inspection prompt. By observing changes in model output, users can examine what information the representation contains. For example, in this figure we can observe whether the representation of “United Kingdom” correctly triggers “London” as the largest city.

Figure 12: LatentQA. LatentQA probes the semantic content of hidden representations by training a separate verbaliser model to answer natural-language questions about them. The process involves (1) curating a dataset of activation–question–answer triplets from a target LLM, and (2) fine-tuning a verbaliser to retrieve specific information latent within those activations, such as the model’s underlying goals or persona.

Figure 13: Probing Classifier. A probing classifier (or probe ) tests the decodability of a concept from the target model’s internal representations. It is trained to distinguish activations associated with positive and negative examples of the concept. High probe accuracy indicates that the concept can be reliably decoded from the representation.

Appropriate Uses	Inappropriate Uses
Establishing information presence (i.e., “ $P$ is encoded”)	Claiming causal relevance (i.e., “the model uses $P$ ”)
Comparing encoding strength across layers	Concluding that a property is not encoded based on low probe accuracy
Auditing for encoded sensitive information	Claiming mechanistic understanding from probes alone
Generating hypotheses for causal investigation	Using probing results to support hypotheses about causality
Early detection or triage before further analysis	Evaluating alignment or safety based on probing results

Table 11: Appropriate and inappropriate uses of probing.

Figure 14: Model diffing via crosscoders. Model diffing identifies features that emerge or remain invariant across model iterations or architectures. A large delta ( $\Delta$ ) in feature activation between two models highlights a changed feature, providing an insight into how fine-tuning or architectural changes alter internal representations.

Figure 15: Steering Vectors. A steering vector

v_{\text{steer}}

can be constructed from the difference between the mean activations of positive and negative contrastive samples at a target layer. During inference, the vector is added to the original hidden state

h_{\ell}

to obtain a modified state

\tilde{h}_{\ell}

Figure 16: Grokking. Training loss (purple) falls early in training while test loss (red) remains high. After a long plateau, test loss drops sharply, and the model generalises. Progress measures defined on model internals show the generalising circuit forming gradually during the plateau, before the change is visible in behaviour.

Figure 17: Six research orientations of AI interpretability.

Stage	Automated Role	Human Role	Deliverable
Discovery	Extract candidate features and circuits	Define scope and review methodology	Candidate features and circuits
Prioritisation	Rank features and circuits by importance metrics	Review rankings and adjust priorities	Prioritised analysis queue
Explanation	Generate explanations at scale	Audit a fraction of samples and review flagged cases	Draft explanations
Validation	Run automated consistency and causal checks	Investigate failure modes and validate edge cases	Validated findings
Synthesis	Aggregate findings and generate reports	Interpret implications and make decisions	Actionable conclusions

Table 12: A human–Automated Interpretability pipeline. Automated methods perform discovery and analysis. Human experts set the objectives, audit intermediate results, and draw conclusions. We use feature and circuit discovery and analysis as an example.

Dimension	Pragmatic	Ambitious	Developmental	Across Time	Introspective	Automated
Main Focus	Immediate safety interventions without complete understanding	Complete causal understanding regardless of cost	Developmental processes of representations, mechanisms, and capabilities	Prediction and control of future model behaviour through training dynamics	Introspective models to explain themselves	Scalable discovery via automated pipelines
Key Metrics	Performance in proxy tasks	Faithfulness and simulatability	Phase-transition signatures (e.g., LLC) and their correspondence to mechanism formation	Forecasting accuracy and intervention effectiveness	Alignment between self-reports and actual mechanisms	Coverage, scalability, robustness, and faithfulness of analysis
Primary Tools	Steering vectors, probes, behavioural evaluation	Circuit discovery, loose-to-strict verification	SLT, loss landscape geometry, representation analysis	Longitudinal analysis, behavioural forecasting, training interventions	Introspective training, activation verbalisers	Circuit discovery, SAEs, LLM agents
Expected Time to Impact	Short-term	Long-term	Long-term	Medium	Medium	Short-term
Scalability	High	Low	Low	Medium	High	Medium
Robustness to Deception	Medium	High	Unknown	Unknown	Low	Low

Table 13: Comparison of six interpretability orientations across their core theses, evaluation criteria, methodological approaches, research time horizons, scalability, and robustness to model deception.

Safety Applications	Descriptions	Prioritised Factors	Promising Methods
Monitoring malicious internal characteristics during inference	Overseeing model activations for hallucination, lying, sycophancy, or other traits considered inappropriate in current inference contexts.	Scalability, robustness	Automated methods such as SAEs and circuit discovery, and scalable methods such as probes
Auditing models during evaluation	Discovering and validating potential misalignments, such as reward hacking, misaligned values, planted backdoors, and evaluation awareness.	Recall rate, real cases	Behavioural evaluation, probes, LLM-based automated auditing pipelines
Shaping model capabilities and behaviours during training	Preventing the development of certain capabilities, such as biothreat knowledge, and promoting certain behaviours, such as honesty.	Robustness of existence or non-existence	Training data attribution and filtering, training-time interventions, longitudinal analysis, scaling-law analysis
Editing misaligned knowledge	Quickly removing or updating knowledge found to be incorrect, outdated, or inappropriate.	Precision, generalisability	Factual knowledge editing methods, concept erasure methods, activation-space analysis such as knowledge representation in low-rank regions
Steering models during inference	Steering models away from certain critically harmful behaviours in deployment.	Precision, low compute cost	Steering vectors, prompting
Reasoning faithfulness	Testing or validating whether the model’s verbalised explanations for its decision-making process align with actual computations in model activations.	Granularity of analysis	Behavioural tests by counterfactual samples, circuit discovery, activation-space analysis such as activation patching

Table 14: AI safety applications with their descriptions, prioritised factors, and promising methods.

Risk Class	Description	Example	Time Horizon
Epistemic risk	False confidence from incomplete or unfaithful analysis	The report concludes that the circuit for deceptive behaviour has been completely removed, but the model’s backup pathway actually emerged after the removal.	Current
Operational risk	MI tools fail silently in deployment	A steering vector is validated to be effective only on a testing set. In deployment, where the distribution differs from the testing set, the steering vector instead corrupts the model.	Near-term
Adversarial risk	MI methods enable attacks on the model	Attackers use published knowledge editing methods to inject incorrect knowledge into deployed models.	Current
Opportunity cost	Resources diverted from other research directions	Heavy MI investment delays deployment of behavioural safeguards.	Long-term

Table 15: Risk taxonomy for mechanistic interpretability in safety applications, including each risk’s description, example, and time horizon (how soon it is expected to materialise given current research and deployment practice).

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 414 papers

Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence
Aug 12, 2026Mengru Wang, Junfeng Fang, Shuofei Qiao +17AI Agents for Scientific Discovery Mechanistic Interpretability
Zhejiang University · National University of Singapore · Heriot-Watt University
Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning
Jul 8, 2026Pranav Sawant, Jakub KrejčíTransformer Interpretability Mechanistic Interpretability
The University of Texas at Dallas 800 W Campbell Rd, Richardson, TX 75080, United States · VSB—Technical University of Ostrava 708 00, Ostrava, Czech Republic
Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewing
Apr 24, 2026Michael Lan, Narmeen Fatimah Oozeer, Chaithanya Bandi +4Scholarly Peer Review Mechanistic Interpretability
Martian · University of Delaware · MATS +2

Aug 12, 2026J/K move · Enter open · S save

Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence

Mengru Wang, Junfeng Fang, Shuofei Qiao, Zhenqian Xu+16

Zhejiang University · National University of Singapore · Heriot-Watt University

AI Agents for Scientific Discovery Mechanistic Interpretability Large Language Model-Guided Scientific Discovery Autonomous Research Agents

Open paper AskPDF arXiv

AI models are increasingly used in scientific discovery and human decision-making. Yet how AI models work and what risks they pose remain poorly understood. As AI development becomes faster and more automated, research on the mechanisms underlying AI remains largely manual. To bridge this gap, we introduce Mechanist, an agentic system that uses AI as a scientific instrument for the autonomous discovery of mechanisms underlying AI. To ground novel mechanism hypotheses, we construct a scientific knowledge graph of 13,000 studies on AI mechanisms, alongside a multidisciplinary database of 43 million papers spanning 26 fields. For reliable experiment execution, we curate a library of 32 foundational methods for mechanism analysis. Compared with Claude Code and existing AI-scientist systems, Mechanist generates higher-quality mechanism hypotheses and executes experiments more reliably. Across four case studies, Mechanist autonomously discovers new model behaviors and their underlying mechanisms, and translates these discoveries into mechanism-guided interventions and interdisciplinary design. Specifically, Mechanist first uncovers a counterintuitive safety risk in scientific laboratories, showing that unsafe traits can transfer to fine-tuned student models through apparently safe training data and emerge across modalities. Mechanist then develops a mechanism theory of belief, revealing how models represent world knowledge, form beliefs, infer the beliefs of others, and how these mechanisms emerge during pretraining. Building on this theory, Mechanist develops targeted interventions that improve model performance across diverse scenarios. Finally, Mechanist can also advance interdisciplinary discovery through mechanistic design, providing an alternative to the computationally intensive generate-and-rerank paradigm.

Abstract

Mechanistic Interpretability

Explore similar work

Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence

Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning

Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewing