cs.AIMar 20, 2026

Pitfalls in Evaluating Interpretability Agents

Authors: Tal Haklay, Nikhil Prakash, Sana Pandey, Antonio Torralba, Aaron Mueller, Jacob Andreas, Tamar Rott Shaham, Yonatan Belinkov

Organizations: Technion—IIT · Northeastern University · MIT · Boston University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Automated interpretability systems aim to reduce the need for human labor and scale analysis to increasingly large models and diverse tasks. Recent efforts toward this goal leverage large language models (LLMs) at increasing levels of autonomy, ranging from fixed one-shot workflows to fully autonomous interpretability agents. This shift creates a corresponding need to scale evaluation approaches to keep pace with both the volume and complexity of generated explanations. We investigate this challenge in the context of automated circuit analysis -- explaining the roles of model components when performing specific tasks. To this end, we build an agentic system in which a research agent iteratively designs experiments and refines hypotheses. When evaluated against human expert explanations across six circuit analysis tasks in the literature, the system appears competitive. However, closer examination reveals several pitfalls of replication-based evaluation: human expert explanations can be subjective or incomplete, outcome-based comparisons obscure the research process, and LLM-based systems may reproduce published findings via memorization or informed guessing. To address some of these pitfalls, we propose an unsupervised intrinsic evaluation based on the functional interchangeability of model components. Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems and reveals key limitations of replication-based evaluation.

Figures & tables

Figure 1: Top: Overview of the system workflow. (A) A researcher specifies a task and a circuit to analyze. (B) A researcher agent iteratively analyzes each component independently, autonomously designing and running experiments as needed. (C) Claude clusters components based on shared functionality inferred from the generated hypotheses. Bottom: Example of tool calls and the corresponding results returned by the tools.
Figure 2: The judge workflow. The system produces hypotheses explaining the functionality of individual components and clusters. A judge model is tasked with matching the hypothesis to one of the descriptions reported by the researchers in the original paper.
TaskModelCircuit Size#Clusters
IOIGPT-2-Small18 Heads6
IOIPythia-160M14 Heads7
Greater-ThanGPT-2-Small8 Heads, 4 MLPs2
AcronymsGPT-2-Small8 Heads3
Colored ObjectsGPT-2-Medium27 Heads3
Entity TrackingLLaMA-7B64 Heads4
Table 1: Summary of tasks from prior circuit analyses. Each cluster represents a group of components with shared functionality.
Figure 3: Performance comparison across six circuit analysis tasks. Left: Component Functionality Accuracy measures how well individual component explanations match human-labeled clusters. Middle: Cluster Functionality Accuracy measures how well cluster explanations match human-labeled clusters. Right: Component Assignment Accuracy assesses cluster alignment with expert-defined clusters using optimal matching. While the systems obtain relatively high results, they usually do not perfectly match expert explanations.
Figure 4: Examples of the agent’s experimental designs. Blue : an IOI task example from the initial prompt set provided to the agent. Orange : the agent’s motivation for the experiment. Green : new example prompts proposed by the agent to test its hypotheses.
Figure 5: Top: Example of indirect memorization in a final hypothesis produced by the one-shot system for attention head 9.9 in the IOI task. The system explicitly uses the term “name mover head”, reproducing the terminology introduced in the original IOI paper. For more details on “name mover” heads, see App. B.1 . Bottom: Example of direct recall of the IOI circuit by Claude when it is explicitly asked to recall the circuit from memory. For more details on the prompt used and the full response, see App. E.1 .
 RandomExpertAgenticOne-shotKendall ( τ,p\tau,p )
Task  MeanBestMeanBest 
IOI-0.28 ±\pm 0.070.210.03 ±\pm 0.090.12-0.08 ±\pm 0.130.21(0.739, 0.00)
IOI (Pythia-160M)-0.24 ±\pm 0.080.100.07 ±\pm 0.100.150.00 ±\pm 0.080.09(0.034, 0.81)
Acronyms-0.18 ±\pm 0.130.16-0.02 ±\pm 0.170.300.00 ±\pm 0.100.30(0.506, 0.02)
Colored-Objects-0.28 ±\pm 0.070.020.01 ±\pm 0.070.050.02 ±\pm 0.050.06(0.318, 0.02)
Entity-Tracking-0.24 ±\pm 0.010.17-0.13 ±\pm 0.02-0.09-0.09 ±\pm 0.02-0.08(0.378, 0.00)
Table 2: Intrinsic cluster quality. Silhouette-based cluster quality scores for random, expert-defined, agentic, and one-shot clusterings across tasks. For agentic and one-shot systems, we report the best and mean scores across runs. The final column reports the Kendall rank correlation ( τ\tau ) and associated pp -value between intrinsic cluster quality and component assignment accuracy.
Figure 6: Each point represents the mean component functionality accuracy, averaged across all components in the IOI task, at a given noise level α\alpha . Both systems exhibit clear performance degradation as noise increases. Running the experiments with additional random seeds yields similar trends (see App. F ).
Figure 7: Swap-based distances from attention head (10,0)(10,0) (a Name Mover head) to all other heads in the IOI circuit, grouped by the reported functional clusters. Other Name Mover heads are closer to head (10,0)(10,0) than heads from different clusters.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Cluster NameComponents (Layer, Head)Description
Duplicate Token Heads(0,1), (0,10), (3,0)These heads detect when the current token is a duplicate of one of the previous tokens in the sequence. Specifically, they are active at the second occurrence of the subject (actor) and attend back to the first occurrence of the subject. When they detect this duplication, they write positional information about the duplicate token’s location into the residual stream. This positional signal is later used by S-Inhibition Heads to help Name Mover Heads avoid attending to duplicated tokens.
S-Inhibition Heads(7,3), (7,9), (8,6), (8,10)These heads attend to the second occurrence of the subject token (the actor) and influence the Name Mover Heads’ attention. This inhibits the Name Mover Heads’ attention to the first and second occurrences of the subject tokens (the actor’s name), effectively removing duplicated tokens from consideration. They achieve this through a combination of token signals and position signals.
Name Mover Heads(9,6), (9,9), (10,0)These heads are responsible for identifying and outputting the indirect object (the recipient). They attend to the recipient’s name in the sentence and copy it forward. Due to the influence of S-Inhibition Heads, they preferentially attend to the indirect object token rather than the subject token(s).
Previous Token Heads(2,2), (4,11)These heads copy information from the immediately previous token to the current position. Specifically, they copy information about the first-occurrence subject token (the actor) to the position right after it. This copied information enables Induction Heads to recognize the pattern when the repeated subject token appears later.
Induction Heads(5,5), (5,8), (5,9), (6,9)These heads attend to the token after the first occurrence of the subject (mediated by Previous Token Heads). They recognize patterns of the form [A][B]…[A] and contribute to predicting [B]. In this task, they provide signals that the subject is duplicated and indicate the position of its first occurrence.
Negative Name Mover Heads(10,7), (11,10)These heads write in the opposite direction of Name Mover Heads, decreasing confidence in predictions. They share similar properties with Name Mover Heads, except that they write opposite to the names they attend to; one hypothesis is that they help hedge against overconfident errors.
Appendix
Table 3: Attention heads involved in the IOI task in GPT-2-small model, grouped according to their functional roles as reported in the original paper.
Cluster NameComponents (Layer, Head)Description
S-Inhibition Heads(6,6), (7,2), (7,9)These heads attend to the second occurrence of the subject token (the actor) and suppress attention to duplicated subject tokens. They influence downstream Name Mover Heads by inhibiting attention to both the first and second occurrences of the subject, effectively removing subject names from consideration when predicting the indirect object. Their behavior combines token-identity signals with positional information.
Name Mover Heads(8,10), (8,2), (9,4), (10,7)These heads are responsible for identifying and copying the indirect object (the recipient’s name) to the output. They attend strongly to the recipient token and write its identity into the residual stream at the final position. Due to suppression signals from S-Inhibition Heads, they preferentially attend to the indirect object rather than to duplicated subject tokens.
Previous Token Heads(2,6)This head copies information from the immediately preceding token to the current position. In the IOI task, it propagates information about the first occurrence of the subject token to the following position. This copied signal enables Induction Heads to later recognize repeated subject patterns and is a key component of the induction mechanism.
Induction Heads(4,11), (4,6), (5,0)These heads attend to the token following the first occurrence of the subject, mediated by signals from Previous Token Heads. They detect repeated patterns of the form [A][B] …\ldots [A] and contribute to predicting [B]. In the IOI task, they signal both that the subject is duplicated and the position of its first occurrence.
Negative Name Mover Heads(9,1)This head shares attention patterns with Name Mover Heads but writes in the opposite direction, decreasing the logits of the names it attends to. It is hypothesized to act as a corrective mechanism, reducing overconfident predictions and helping the model hedge when uncertain.
Negative Copy Suppression Heads(9,5)These heads are active at the final token position in the sentence, where the model predicts the output. They attend to earlier tokens in the input and reduce the model’s confidence in those tokens when they are strongly represented in the residual stream. In the IOI circuit, they reduce the model’s confidence in the correct indirect object name, which harms task performance as they push the model toward incorrect outputs.
Appendix
Table 4: Attention heads involved in the IOI task in Pythia-160M, grouped according to their functional roles as reported in the original paper.
Cluster NameComponents (Layer, Head)Description
Greater-Than ComputerMLP8, MLP9, MLP10, MLP11These components implement the mathematical comparison operation. They are active at the final token position where the model predict the end year. They directly boost the probabilities for all years greater than the start year YY while suppressing probabilities for years less than or equal to YY.
Year Identification(5,1), (5,5), (6,1), (6,9), (7,10), (8,8), (8,11), (9,1)These components are responsible for detecting and extracting the start year value from the input sentence. They are active at the final token position (where the model predicts the end year) and attend back to the YY position (the two-digit start year token) in the template ””The ¡noun¿ lasted from the year XXYY to the year XX””. Their output corresponds to the start year YY value in the embedding space, transferring it to the greater-than computer, effectively communicating ””this is the reference number”” to the downstream MLPs. Their attention patterns show consistent focus on the YY token regardless of the specific year value, indicating they’ve learned to identify this positionally and contextually important numerical information.
Appendix
Table 5: Attention heads involved in the Greater-Than task in GPT-2-small model, grouped according to their functional roles as reported in the original paper.
Cluster NameComponents (Layer, Head)Description
Letter Mover Heads(8,11), (10,10), (9,9), (11,4)These heads are responsible for the core acronym formation process. They are active at the position where the previous acronym letter was predicted (for example, when the model has already predicted ”CE” and needs to predict the third letter ”O”). These heads attend back to the capital letter at the beginning of the corresponding word (like attending to ”O” in ”Officer” when predicting ”O” in the acronym). They perform a copying mechanism: they take the capital letter of the corresponding word and copy it to form the predicted capital letters for the acronym.
Fuzzy Previous Token Heads(4,11), (1,0), (2,2)These attention heads handle information flow between words in the input phrase. They are active at each word’s capital letter position and attend to the previous word’s capital letter. These heads follow a ”previous token” pattern, systematically copying information from one word to the next in sequence. These heads create a chain of information that flows from word to word, ensuring that context from e.g. ”Executive” is available when processing ”Officer,” and so on. This information is then moved to the Letter Mover Heads through the Information Bridge Heads to ensure that when the model needs to predict the third letter ”O” when given ”CE,” it can access information not just about ”Executive” but also about its relationship to the next word ”Officer”.
Information Bridge Heads(5,8)These attention heads move word-level information to the acronym-level predictions. They are active at acronym letter positions and attend to both the corresponding word’s capital letter and surrounding word content. They connect the Fuzzy Previous Token Heads with the Letter Mover Heads, moving information about the letters for the acronym.
Appendix
Table 6: Attention heads involved in the Acronyms task in GPT-2-small model, grouped according to their functional roles as reported in the original paper.
Cluster NameComponents (Layer, Head)Description
Mover Heads(10,6), (14,5), (14,14), (15,10), (15,14), (16,0), (16,15), (17,4), (18,12), (19,15), (21,7)These heads are responsible for copying the correct answer token to the next prediction. They are active at the final token position (the ”:” token) and attend back to the correct color token in the context. These heads write strongly in the embedding direction of whatever they attend to, effectively telling the model to predict that token.
Content Gatherer Heads(11,6), (11,7), (12,6), (12,14), (12,15), (13,1), (13,3)These heads tell the mover heads which color to select. They are active at the final token position and attend primarily to the queried object and the token ”color” in the question. These heads gather information about which specific object’s color is being requested and pass this signal to guide the mover heads’ attention.
Induction/Duplicate Token Heads(6,4), (6,15), (7,2), (7,4), (7,11), (8,15), (9,3), (10,12), (10,14)These heads detect the duplication pattern where the same object appears twice in the prompt—once in the object list and again in the question. They are active at the second occurrence of the object (in the question) and attend back to the first occurrence or the token after it. This duplication detection provides a signal that gets passed to content gatherer heads, helping the model to locate the queried object in the object list.
Appendix
Table 7: Attention heads involved in the Colored Objects task in GPT-2-XL model, grouped according to their functional roles.
Cluster NameComponents (Layer, Head)Description
Value Fetcher Heads(14,13), (15,2), (15,5), (15,6), (15,13), (15,18), (16,23), (16,27), (17,3), (17,27), (17,28), (18,3), (18,6), (18,8), (18,20), (18,25), (18,28), (19,8), (19,11), (19,20), (19,26), (19,30), (20,14), (20,29), (21,0), (21,3), (21,4), (21,25), (21,26), (23,5), (23,15), (23,16), (23,17), (23,19), (23,20), (24,0), (24,5), (24,8), (25,14), (29,7)These heads are responsible for identifying and outputting the correct object (the entity being tracked). They attend to object tokens in the sentence corresponding to the queried box. They then copy the value of the object they attend to. Thanks to the contribution of the Position Transmitter Heads, which provide positional guidance through query composition, they preferentially attend to the object associated with the queried box rather than other objects in the context.
Position Transmitter Heads(11,23), (12,23), (14,27), (16,2), (19,12)These heads write to the query vectors of the Value Fetcher Heads. They attend to the box label in the query, and transport positional information from the Position Detector Heads to guide the Value Fetcher Heads toward the correct object location.
Position Detector Heads(1,9), (4,4), (6,10), (7,9), (7,17), (7,26), (8,22), (9,2), (9,7), (9,10), (9,21), (10,3), (10,4), (11,23)These heads identify positional information of the correct object within the context. They attend to the correct box label, and receive contextual structure information from Structure Reader Heads through value composition and pass their positional information to Position Transmitter Heads.
Structure Reader Heads(8,12), (8,21), (11,9), (11,23), (12,23)These heads process the contextual structure and gather information related to each box separately. They attend primarily to tokens within a sentence describing a single box. They collect information about the overall sentence structure. Their output provides contextual understanding to Position Detector Heads through value composition.
Appendix
Table 8: Attention heads involved in the Entity Tracking task in LLaMa-7B model, grouped according to their functional roles.
(Layer, Head)Correct Object Attention ( ↑\uparrow )Incorrect Object Prediction ( ↓\downarrow )Counterfactual Patching ( ↓\downarrow )
(15, 13)0.730.040.05
(21, 3)0.720.260.03
(24, 5)0.170.660.18
(20, 14)0.320.440.11
(18, 8)0.720.230.09
(29, 7)0.280.610.09
Appendix
Table 9: Evaluation of heads labeled as value fetcher heads in the Entity Tracking task across three analyses on 500 examples. Correct Object Attention measures the fraction of cases in which the head attends most strongly to the correct object token (higher is better). Incorrect Object Prediction measures the fraction of cases in which applying a logit lens to the head’s output assigns the highest probability to an incorrect object mentioned in the prompt (lower is better). Counterfactual Patching Effect measures the fraction of cases in which replacing the head’s value with that from a counterfactual prompt increases the probability of the correct object by at least 1% (lower is better). For each analysis, we highlight the 10 heads whose behavior is least aligned with the expected behavior of value fetcher heads.
Figure 8: Each point in the graph represents the mean component functionality accuracy, averaged across all components in the IOI task, at a given noise level α\alpha . We report results for α∈[0,0.5,0.6,0.7,0.8,0.9,1]\alpha\in[{0,0.5,0.6,0.7,0.8,0.9,1}] . Each plot corresponds to a different random seed, which determines both the specific task examples provided to the systems and the permutations applied to the experimental results. As shown, both systems exhibit clear performance degradation as the noise level increases.
More in this topic

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 371 papers

Explore similar work

CardsList
Jun 23, 2026

Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize. In this work, we study whether language model (LM) agents can assist with this explanation problem once a circuit has already been identified. We introduce AgenticInterpBench, a benchmark for circuit explanation built from 84 semi-synthetic transformer circuits with 163 component-level annotations. We propose HyVE (Hypothesize, Validate, Explain), an agentic explainer that analyzes each component through an iterative loop of observation, hypothesis generation, and causal validation, eventually producing a component-level explanation and a circuit-level task description. Across four LM backbones, HyVE recovers useful component- and task-level explanations, but no backbone is uniformly best. Our analysis shows that strong backbones usually form observation-grounded hypotheses, while failures more often arise later in the validation loop, through incomplete validation plans, code execution errors, or unresolved hypotheses. A case study on an arithmetic circuit in Llama-3-8B shows that the same formulation can extend beyond semi-synthetic benchmarks to naturally trained models. Overall, LM agents are promising circuit explainers, but reliable validation remains the key obstacle.

Ayan Antik Khan, Harsh Kohli, Yuekun Yao +2

George Mason University · The Ohio State University

LLM Agent EvaluationMechanistic InterpretabilityCircuits in Language ModelsExplainability Evaluation
AskPDF
May 2, 2026

Automated Interpretability and Feature Discovery in Language Models with Agents

We introduce an autonomous multiagent framework for mechanistic interpretability that automates both explaining and finding internal features in large language models. The system runs two coupled loops: (1) explanation refinement, where an agent proposes competing hypotheses and iteratively tests them with targeted prompt controls and a multi-metric evaluation; and (2) feature discovery, where an agent generates prompt sets, constructs a k-nearest-neighbor graph in activation space, and retrieves candidate features using statistical separability and semantic coherence criteria. On Gemma-2 family models and MLP neurons in weight-sparse transformers, our agent improves over one-shot auto-interpretations, discovers language-specific and safety-relevant features, and produces auditable explanation traces, showing that agent-driven empirical loops yield sharper and more falsifiable explanations than one-shot labels.

Arnau Marin-Llobet, Javier Ferrando

Harvard University

Mechanistic Interpretability
AskPDF
May 5, 2026

Agentic-imodels: Evolving agentic interpretability tools via autoresearch

Agentic data science (ADS) systems are rapidly improving their capability to autonomously analyze, fit, and interpret data, potentially moving towards a future where agents conduct the vast majority of data-science work. However, current ADS systems use statistical tools designed to be interpretable by humans, rather than interpretable by agents. To address this, we introduce Agentic-imodels, an agentic autoresearch loop that evolves data-science tools designed to be interpretable by agents. Specifically, it develops a library of scikit-learn-compatible regressors for tabular data that are optimized for both predictive performance and a novel LLM-based interpretability metric. The metric measures a suite of LLM-graded tests that probe whether a fitted model's string representation is "simulatable" by an LLM, i.e. whether the LLM can answer questions about the model's behavior by reading its string output alone. We find that the evolved models jointly improve predictive performance and agent-facing interpretability, generalizing to new datasets and new interpretability tests. Furthermore, these evolved models improve downstream end-to-end ADS, increasing performance for Copilot CLI, Claude Code, and Codex on the BLADE benchmark by up to 73%

Chandan Singh, Yan Shuo Tan, Weijia Xu +4

Microsoft Research · National University of Singapore

Interpretable MLAutomated Machine LearningTabular RegressionData Analysis Agents
AskPDF