Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
Authors: Fazl Barez, Tung-Yu Wu, Hunar Batra, Gabriele Dominici, Florent Draye, Yanai Elazar, Damiano Fornasiere, Antía García, Abir Harrasse, Riley Harris, Sara Hooker, Michael Lan, Belinda Z. Li, Aaron Maiwald, Narmeen Fatimah Oozeer, Naomi Saphra, Or David Shafran, Adi Simhi, Robert Trager, Shriyash Kaustubh Upadhyay, Ziyu Yao, Yu Zhao, Shay B. Cohen, Mor Geva, Yoshua Bengio, Stella Biderman, Hassan Sajjad
Organizations: University of Oxford · Martian · Università della Svizzera italiana · Max Planck Institute for Intelligent Systems · Bar-Ilan University · LawZero · Adaption Labs · MIT · Anthropic · Boston University · Tel Aviv University · Technion · George Mason University · University of Edinburgh · Mila, Université de Montréal · EleutherAI · New York University Abu Dhabi & Dalhousie University
Neural networks are likely to be among the most consequential artefacts in human history. As these systems become more capable and autonomous, their potential to cause societal harm also grows, making AI safety one of the defining challenges of our time. Yet we continue to build increasingly powerful systems without fully understanding how they work, how they will behave in unfamiliar situations, or how to reliably control them. Mechanistic interpretability offers a path towards understanding these systems by studying their internal representations and computations, with the aim of evaluating, predicting, and controlling behaviour. We survey mechanistic interpretability for AI safety in order to answer the question: Does understanding how models work actually make them safer? We place mechanistic interpretability in the broader context of interpretability research and compare its mechanistic evidence with behavioural evaluation. We introduce the foundations of mechanistic interpretability, including features, polysemanticity, transformer computation, and circuits, and survey methods for feature extraction and analysis, circuit discovery and causal verification, explaining intermediate representations in natural language, learning dynamics, and training-data attribution. Beyond surveying existing methods, we propose a unified framework for evaluating mechanistic interpretability claims and characterise six research orientations by their goals, assumptions, and trade-offs. Finally, we examine how mechanistic understanding can support AI safety, including knowledge editing, steering, and monitoring, and assess its limitations, failure modes, open problems, and relationship to behavioural safety measures. We also release a repository of code notebooks with worked examples that reproduce key findings from surveyed methods, allowing readers to build intuition through hands-on experimentation.
Figures & tables
Figure 2: Overview of the conceptual flow of the survey, categorised into four primary pillars: (1) Motivation & Concepts, covering fundamental knowledge and core hypotheses; (2) Technical Methods, detailing approaches developed and used in interpretability research; (3) Research Directions, exploring main directions the field is taking; and (4) Safety & Open Problems, identifying critical challenges and deployment-oriented interventions for AI alignment.
Not mechanistic without causal intervention; supports localisation, auditing, and early detection tasks
Mechanistic (e.g., circuits, SAEs)
Analyse internal computations and causal structure
Reveals algorithms; enables targeted intervention
Computationally intensive; complements other methods when causal understanding is required
Table 2: Trade-offs and characteristics of interpretability methods.
Figure 3: Circuit. This diagram illustrates a circuit within a transformer, where specific nodes (MLP layers and attention heads in this case) perform specialised computations.
Figure 4: Privileged versus non-privileged bases. In the hidden layers of MLP blocks, element-wise nonlinear operations break rotational symmetry and make individual axes meaningful. In contrast, the residual stream contains no such operations that privilege particular axes, so rotating the basis preserves the relevant computation.
Figure 5: Superposition implies polysemanticity. Left: when the number of features m does not exceed the number of dimensions d , i.e., m≤d , feature directions can align cleanly with individual neuron axes, producing monosemantic neurons. Right: when m>d , multiple feature directions must share the same limited set of neuron axes; as a result, each neuron axis cuts across multiple feature directions, producing polysemantic neurons.
Figure 6: Sparse Autoencoder (SAE). SAEs are trained to reconstruct activations of a model component at a specific layer ℓ , hℓ , into an overcomplete, sparse latent space z , such that the reconstructed activation hℓ^≈hℓ . By enforcing sparsity on z , the model decomposes dense, polysemantic representations into monosemantic, human-interpretable features.
Layer Type
Variants Used
Reported Basis
Reported Caveats
Residual stream (early layers)
Gated SAE, Top- k
Activations are closer to input embeddings; moderate sparsity
Table 3: Heuristic recommendations for selecting SAE variants across different layer types. Recommendations are based on current empirical evidence and practical experience, and may evolve as SAE methodologies and evaluation standards continue to develop.
Variant
Reported Finding
Setting
Standard
Sparse dictionary learning by L1 regularisation recovers features more monosemantic than individual neurons; subject to shrinkage bias
Establishing work, small models ( Cunningham et al., 2024 ; Bricken et al., 2023 )
Gated
Separating detection from magnitude estimation addresses L1 shrinkage at minimal added complexity
Reconstruction–sparsity improvement at matched sparsity, Gemma-family ( Rajamanoharan et al., 2024a )
JumpReLU
Learned per-feature thresholds trained with straight-through estimators give the strongest reported reconstruction
Matched-sparsity comparison, Gemma-family ( Rajamanoharan et al., 2024b ) ; needs most tuning ( Bussmann et al., 2024 )
Top- k
A hard ∥z∥0≤k constraint removes the sparsity coefficient from the search space
Scaling behaviour characterised on large models; k=32 – 150 commonly reported ( Gao et al., 2025 ; Karvonen et al., 2025 )
Batch Top- k
A batch-level sparsity budget lets the number of active features vary per input, improving reconstruction at matched average sparsity
Introduced as a relaxation of per-example Top- k , achieving better reconstruction ( Bussmann et al., 2024 )
Transcoder
Learning the MLP input–output map avoids spanning the nonlinearity with a single dictionary
Reported to perform at least on par with SAEs ( Dunefsky et al., 2024 ; Paulo et al., 2025c )
Table 4: Reported evidence for each SAE variant, with the setting in which it was obtained.
Condition
Description
Architecture dependence
Most evidence derives from Llama- and Gemma-family models. Generality of the findings across model families remains unestablished.
Task dependence
The properties that matter may differ across analysis goals, such as safety auditing, behaviour understanding, and knowledge localisation, but existing evaluations often do not distinguish between them.
Rapid iteration
New architectures and training procedures are proposed faster than existing ones are sufficiently studied and replicated.
Immature theoretical analysis
Current findings are mostly empirical and may be superseded. Understanding of why particular variants succeed or fail is incomplete.
Feature reproducibility
SAEs trained on the same data with different random seeds can recover substantially different features. When between-variant performance differences are smaller than within-variant seed-to-seed variability, we may not confidently attribute them to architectural factors. However, few reported comparisons have rigorously accounted for this concern.
Table 5: Conditions for the reported evidence on SAE variant behaviour.
Figure 7: Transcoders and CLTs. Transcoders reconstruct the output of the MLP from its input in the same transformer layer. CLTs extend this by partially reconstructing the outputs of multiple subsequent layers from a single point in the residual stream.
Figure 8: Parameter count (left) and FP16 memory requirements (right) of a full cross-layer transcoder as a function of model depth, for different residual stream dimensions and expansion factor e=32 .
Figure 9: Crosscoder. Crosscoders are trained to reconstruct the residual stream across multiple layers simultaneously.
Method
Assumption
Strengths
Weaknesses
Best Use Case
SAE
Sparse, linear features
Open-ended discovery; fast inference
Hyperparameter-sensitive; identifiability issues
Open-ended feature discovery
NMF
Non-negative, additive parts
Parts-based interpretation; uniqueness guarantees
Requires non-negative data; limited sparsity
Naturally non-negative domains
PCA
Variance aligns with features
Closed form; optimal low-rank reconstruction
Explains global variance and produces orthogonal features
Small-scale settings; as a baseline method
ICA
Complete and independent sources
Theoretical guarantees when assumptions hold
Assumes independence; Gaussian blindness
When independence holds
Supervised methods
Known concept labels
High performance on targeted concepts
Requires labels; no discovery
When certain concepts of interest are known
K-SVD
Sparse, linear features
Explicitly solving the sparse-coding problem; no encoder bias
Slow inference; no learned encoder
Small, fixed sets of activations; as a baseline method
Table 6: Comparison of feature extraction methods.
Circuit-level descriptions; trade-off between sparsity and variance explained
Probing
Agreement between probe-derived importance and intervention effects
Coverage across target concepts or behaviours under different distributions
Robustness across seeds and sampled fitting sets
Necessity, specificity, and intervention tests on probe directions
Explanation of shared information in fitting set and search for confounding factors
Steering Vectors
Agreement between predicted and observed intervention effects
Coverage across behaviours deemed relevant
Stability across steering magnitudes, seeds, and paraphrases
Intervention efficacy and specificity; side-effect analysis
Explanation of shared information in fitting set and search for confounding factors
Table 10: Recommended evaluation priorities by interpretability method. The entries indicate evaluation aspects particularly important for each method, not mutually exclusive metrics.
Figure 10: Logit Lens. Logit lens provides a window into a model’s intermediate state by applying the final unembedding layer directly to the residual stream at earlier layers ( h1,h2,… ). This allows users to see which tokens the model is “considering” mid-computation, such as the emergence of the concept “CAT” before the final output layer is reached.
Figure 11: Patchscopes. Patchscopes extracts the model representation of interest from a source prompt and “patches” it into an inspection prompt. By observing changes in model output, users can examine what information the representation contains. For example, in this figure we can observe whether the representation of “United Kingdom” correctly triggers “London” as the largest city.
Figure 12: LatentQA. LatentQA probes the semantic content of hidden representations by training a separate verbaliser model to answer natural-language questions about them. The process involves (1) curating a dataset of activation–question–answer triplets from a target LLM, and (2) fine-tuning a verbaliser to retrieve specific information latent within those activations, such as the model’s underlying goals or persona.
Figure 13: Probing Classifier. A probing classifier (or probe ) tests the decodability of a concept from the target model’s internal representations. It is trained to distinguish activations associated with positive and negative examples of the concept. High probe accuracy indicates that the concept can be reliably decoded from the representation.
Appropriate Uses
Inappropriate Uses
Establishing information presence (i.e., “ P is encoded”)
Claiming causal relevance (i.e., “the model uses P ”)
Comparing encoding strength across layers
Concluding that a property is not encoded based on low probe accuracy
Auditing for encoded sensitive information
Claiming mechanistic understanding from probes alone
Generating hypotheses for causal investigation
Using probing results to support hypotheses about causality
Early detection or triage before further analysis
Evaluating alignment or safety based on probing results
Table 11: Appropriate and inappropriate uses of probing.
Figure 14: Model diffing via crosscoders. Model diffing identifies features that emerge or remain invariant across model iterations or architectures. A large delta ( Δ ) in feature activation between two models highlights a changed feature, providing an insight into how fine-tuning or architectural changes alter internal representations.
Figure 15: Steering Vectors. A steering vector vsteer can be constructed from the difference between the mean activations of positive and negative contrastive samples at a target layer. During inference, the vector is added to the original hidden state hℓ to obtain a modified state h~ℓ .
Figure 16: Grokking. Training loss (purple) falls early in training while test loss (red) remains high. After a long plateau, test loss drops sharply, and the model generalises. Progress measures defined on model internals show the generalising circuit forming gradually during the plateau, before the change is visible in behaviour.
Figure 17: Six research orientations of AI interpretability.
Stage
Automated Role
Human Role
Deliverable
Discovery
Extract candidate features and circuits
Define scope and review methodology
Candidate features and circuits
Prioritisation
Rank features and circuits by importance metrics
Review rankings and adjust priorities
Prioritised analysis queue
Explanation
Generate explanations at scale
Audit a fraction of samples and review flagged cases
Draft explanations
Validation
Run automated consistency and causal checks
Investigate failure modes and validate edge cases
Validated findings
Synthesis
Aggregate findings and generate reports
Interpret implications and make decisions
Actionable conclusions
Table 12: A human–Automated Interpretability pipeline. Automated methods perform discovery and analysis. Human experts set the objectives, audit intermediate results, and draw conclusions. We use feature and circuit discovery and analysis as an example.
Dimension
Pragmatic
Ambitious
Developmental
Across Time
Introspective
Automated
Main Focus
Immediate safety interventions without complete understanding
Complete causal understanding regardless of cost
Developmental processes of representations, mechanisms, and capabilities
Prediction and control of future model behaviour through training dynamics
Introspective models to explain themselves
Scalable discovery via automated pipelines
Key Metrics
Performance in proxy tasks
Faithfulness and simulatability
Phase-transition signatures (e.g., LLC) and their correspondence to mechanism formation
Forecasting accuracy and intervention effectiveness
Alignment between self-reports and actual mechanisms
Coverage, scalability, robustness, and faithfulness of analysis
Primary Tools
Steering vectors, probes, behavioural evaluation
Circuit discovery, loose-to-strict verification
SLT, loss landscape geometry, representation analysis
Longitudinal analysis, behavioural forecasting, training interventions
Introspective training, activation verbalisers
Circuit discovery, SAEs, LLM agents
Expected Time to Impact
Short-term
Long-term
Long-term
Medium
Medium
Short-term
Scalability
High
Low
Low
Medium
High
Medium
Robustness to Deception
Medium
High
Unknown
Unknown
Low
Low
Table 13: Comparison of six interpretability orientations across their core theses, evaluation criteria, methodological approaches, research time horizons, scalability, and robustness to model deception.
Safety Applications
Descriptions
Prioritised Factors
Promising Methods
Monitoring malicious internal characteristics during inference
Overseeing model activations for hallucination, lying, sycophancy, or other traits considered inappropriate in current inference contexts.
Scalability, robustness
Automated methods such as SAEs and circuit discovery, and scalable methods such as probes
Auditing models during evaluation
Discovering and validating potential misalignments, such as reward hacking, misaligned values, planted backdoors, and evaluation awareness.
Shaping model capabilities and behaviours during training
Preventing the development of certain capabilities, such as biothreat knowledge, and promoting certain behaviours, such as honesty.
Robustness of existence or non-existence
Training data attribution and filtering, training-time interventions, longitudinal analysis, scaling-law analysis
Editing misaligned knowledge
Quickly removing or updating knowledge found to be incorrect, outdated, or inappropriate.
Precision, generalisability
Factual knowledge editing methods, concept erasure methods, activation-space analysis such as knowledge representation in low-rank regions
Steering models during inference
Steering models away from certain critically harmful behaviours in deployment.
Precision, low compute cost
Steering vectors, prompting
Reasoning faithfulness
Testing or validating whether the model’s verbalised explanations for its decision-making process align with actual computations in model activations.
Granularity of analysis
Behavioural tests by counterfactual samples, circuit discovery, activation-space analysis such as activation patching
Table 14: AI safety applications with their descriptions, prioritised factors, and promising methods.
Risk Class
Description
Example
Time Horizon
Epistemic risk
False confidence from incomplete or unfaithful analysis
The report concludes that the circuit for deceptive behaviour has been completely removed, but the model’s backup pathway actually emerged after the removal.
Current
Operational risk
MI tools fail silently in deployment
A steering vector is validated to be effective only on a testing set. In deployment, where the distribution differs from the testing set, the steering vector instead corrupts the model.
Near-term
Adversarial risk
MI methods enable attacks on the model
Attackers use published knowledge editing methods to inject incorrect knowledge into deployed models.
Current
Opportunity cost
Resources diverted from other research directions
Heavy MI investment delays deployment of behavioural safeguards.
Long-term
Table 15: Risk taxonomy for mechanistic interpretability in safety applications, including each risk’s description, example, and time horizon (how soon it is expected to materialise given current research and deployment practice).
AI models are increasingly used in scientific discovery and human decision-making. Yet how AI models work and what risks they pose remain poorly understood. As AI development becomes faster and more automated, research on the mechanisms underlying AI remains largely manual. To bridge this gap, we introduce Mechanist, an agentic system that uses AI as a scientific instrument for the autonomous discovery of mechanisms underlying AI. To ground novel mechanism hypotheses, we construct a scientific knowledge graph of 13,000 studies on AI mechanisms, alongside a multidisciplinary database of 43 million papers spanning 26 fields. For reliable experiment execution, we curate a library of 32 foundational methods for mechanism analysis. Compared with Claude Code and existing AI-scientist systems, Mechanist generates higher-quality mechanism hypotheses and executes experiments more reliably. Across four case studies, Mechanist autonomously discovers new model behaviors and their underlying mechanisms, and translates these discoveries into mechanism-guided interventions and interdisciplinary design. Specifically, Mechanist first uncovers a counterintuitive safety risk in scientific laboratories, showing that unsafe traits can transfer to fine-tuned student models through apparently safe training data and emerge across modalities. Mechanist then develops a mechanism theory of belief, revealing how models represent world knowledge, form beliefs, infer the beliefs of others, and how these mechanisms emerge during pretraining. Building on this theory, Mechanist develops targeted interventions that improve model performance across diverse scenarios. Finally, Mechanist can also advance interdisciplinary discovery through mechanistic design, providing an alternative to the computationally intensive generate-and-rerank paradigm.
Mengru Wang, Junfeng Fang, Shuofei Qiao +17
Zhejiang University · National University of Singapore · Heriot-Watt University
This article offers a comprehensive overview of mechanistic interpretability, an emerging field that seeks to reverse-engineer the internal algorithms of modern neural networks. While traditional explainable AI methods often stop at surface-level input-output correlations, this approach directly addresses the opaque "black box" nature of machine learning models, which is essential for ensuring safety and auditability in high-stakes deployments. The paper provides a detailed examination of Transformer circuit analysis, exploring how internal components like the residual stream, attention mechanisms, and induction heads drive complex tasks and in-context learning. It subsequently tackles the core challenge of superposition and polysemanticity, demonstrating how tools like Sparse Autoencoders (SAEs) and transcoders can decompose tangled network activations into distinct, human-interpretable features. Furthermore, the paper explores methods for actively controlling and modifying model behavior through steering vectors and causal interventions. Finally, it connects these mechanistic insights with neurosymbolic AI frameworks designed to translate neural representations into explicit, executable logical rules.
Pranav Sawant, Jakub Krejčí
The University of Texas at Dallas 800 W Campbell Rd, Richardson, TX 75080, United States · VSB—Technical University of Ostrava 708 00, Ostrava, Czech Republic
While mechanistic interpretability (MI) has produced important insights into neural network internals, the field has yet to establish a standardized system to audit experiments. As such, many of its findings remain underutilized in safety-critical applications such as medical AI and autonomous systems, as stakeholders cannot certify their validity. Recent work demonstrates this concretely: two papers found conflicting conclusions for the same behavior, and a third study revealed that both were partially correct but incomparable due to methodological inconsistencies. Without standardized auditing, such ambiguities hinder adoption in high-stakes contexts requiring strong correctness guarantees. We call for the MI community to work towards developing a novel reviewing system that complements peer review via: (1) Continuous reviewing supported by a Collaborative Reviewing Platform where meta-science results and discussions (such as critiques, negative results, post-hoc extensions, reproductions, replications, and partial results) that fit outside of papers are organized and discussed, allowing for comments and revisions to be made at any time (2) Generalizing good practices found on this platform into expert-verified guidelines and protocols to improve auditing efficiency, and (3) Source-based auditing systems that track arguments which claims depend on. This position paper encourages constructive debate over the necessity, design and implementation of such a framework, providing early concrete examples to help catalyze these dialogues. Overall, we propose that auditing MI itself is essential for its application in AI safety, industry, and governance.
Michael Lan, Narmeen Fatimah Oozeer, Chaithanya Bandi +4