DECODE: Diagnosing Interpretability Failures in Automated Interpretation Process
Authors: Dohyun Kim, Jaeung Lee, Donghee Hong, Jaemin Jo
Organizations: KAIST, Seoul, Republic of Korea · Samsung Research, Seoul, Republic of Korea · University of Maryland, College Park, Maryland, United States · Sungkyunkwan University, Suwon, Republic of Korea
Tools for interpreting the internal representations of Large Language Models (LLMs) increasingly rely on LLMs themselves to both generate and evaluate explanations, creating a self-referential automated interpretation process with minimal human oversight. This automation raises significant reliability concerns, which are exacerbated by the absence of ground truth. In practice, it is difficult to validate explanations or to attribute interpretability failures to their source -- whether they originate from the tool's extracted representations or from the LLM used in the process. To address these challenges, we present DECODE, a visual analytics system that reintroduces human oversight into the automated interpretation process. Our approach decomposes the diagnostic workflow into three progressive stages, isolating failure sources ranging from structural defects to explanation-level misalignments. Drawing inspiration from inter-coder consensus in qualitative analysis, we utilize consensus among independently generated explanations and their evaluation metrics as a proxy for reliability. To further scale human inspection, we employ a three-phase active seeding protocol that strategically recommends high-priority instances for manual review. Finally, we demonstrate the effectiveness of our approach through a case study on 16,384 Sparse Autoencoder features, showing that it can efficiently identify and attribute interpretability failures at scale.
Figures & tables
Figure 1: Overview of DECODE. (A) Automated interpretation process extracts SAE features, generates explanations with three explainers, and evaluates them with three metrics. The red dashed border highlights the self-referential loop. (B) Three-stage diagnostic pipeline sequentially assigns features into five diagnostic categories, passing unresolved features to the next stage via intermediate categories. (C) Three-phase active seeding protocol recommends features based on representativeness, decision boundary proximity, and classifier ensemble disagreement.
Category
Description
Incoherent Splitting
An SAE-driven failure where two features divide a single concept without a clear conceptual boundary.
Well-Explained
A feature whose generated explanations faithfully capture the shared pattern across its activating examples.
Missed Syntax
An explainer-driven failure where the explanations omit a shared surface-level pattern, such as common subwords or morphological affixes.
Missed Context
An explainer-driven failure where the explanations omit a shared contextual pattern, such as topic, domain, or tone.
Noisy Activation
An SAE-driven failure where the feature activates on heterogeneous examples with no coherent pattern.
Table 1: Terminal diagnostic categories used in DECODE.
Figure 2: The iterative labeling workflow within each diagnostic stage. Users screen, inspect, and label features in a loop where each label retrains the classifier for automatic labeling.
Figure 3: Stage 1 interface. (A) Stage tabs indicate diagnostic progress (Stage 1 active). (B) Overview Panel displays a decoder similarity histogram. A draggable threshold at 0.40 separates inspection candidates (above) from features assigned Coherent Separation (below). (C) Workspace Panel: (c1) Seeding List ranks candidate pairs, showing 20 representatives in prototype-first phase; (c2) Inspection View juxtaposes feature 13143 and 13256 both activating on tokens starting with “sta”; (c3) Labeling View displays confidence histogram and stability chart after three training iterations.
Figure 4: Stage 2 interface. (A) Overview Panel screens features by average interpretability score as in Stage 1. Feature 10361 is chosen for inspection: (B) activating examples with absence-related tokens highlighted; (C) the user hovers over the explainer consensus cluster with medoid “missing”; (D) constituent phrases of the hovered cluster highlighted in blue, with per-explainer interpretability scores showing fuzz and detection above random baseline; (E) confidence histogram with draggable thresholds defining the automatic labeling region.
Figure 5: Stage 3 interface for feature 3588. (A) Overview Panel shows all Need Revision features passed from Stage 2 without threshold-based screening. The Inspection View presents: (B) two activating-example panels, with syntax-pattern borders on “$$” and “aligned” (top) and context-pattern borders on semantically similar phrases and distinctive recurring tokens such as “$\” and “\theta” (bottom); (C) expanded explainer consensus clusters listing all constituent phrases with per-explainer scores on hover. The Labeling View adapts to three categories: (D) confidence histogram with tentative labels predominantly Noisy Activation ; (E) category scatter showing distribution skewed toward Noisy Activation .
Stage 1
Stage 2
Stage 3
Labels
Min
/unit
Labels
Min
/unit
Labels
Min
/unit
P1
59
49
0.83
45
42
0.93
34
36
1.06
P2
79
42
0.53
36
27
0.75
52
43
0.83
P3
59
28
0.47
48
38
0.79
61
66
1.08
Table 2: Labeling effort per stage. A label denotes one manually labeled unit: a feature pair in Stage 1, a feature in Stages 2 and 3.
Incoherent Splitting
Well- Explained
Missed Syntax
Missed Context
Noisy Activation
Total
P1
26 (57)
41 (8,403)
3 (70)
2 (731)
16 (7,123)
88 (16,384)
P2
16 (18)
40 (7,567)
6 (1,187)
3 (243)
20 (7,369)
85 (16,384)
P3
14 (42)
61 (7,319)
4 (820)
2 (590)
25 (7,613)
106 (16,384)
Table 3: Category distribution per expert. Well-Explained aggregates assignments made at Stage 2 and at Stage 3. Each cell shows manual feature count with the final total after automatic labeling in parentheses.
Score-only
Individual expert
Expert pair
Group of
baseline
P1
P2
P3
P1-P2
P1-P3
P2-P3
3 experts
F1
0.36
0.40
0.37
0.41
0.53
0.54
0.50
0.62
Table 4: End-to-end macro-F1 on the reference set. Every condition assigns all five diagnostic categories to all 16,384 features, combining manual labels with propagated labels from automatic labeling.
We introduce an autonomous multiagent framework for mechanistic interpretability that automates both explaining and finding internal features in large language models. The system runs two coupled loops: (1) explanation refinement, where an agent proposes competing hypotheses and iteratively tests them with targeted prompt controls and a multi-metric evaluation; and (2) feature discovery, where an agent generates prompt sets, constructs a k-nearest-neighbor graph in activation space, and retrieves candidate features using statistical separability and semantic coherence criteria. On Gemma-2 family models and MLP neurons in weight-sparse transformers, our agent improves over one-shot auto-interpretations, discovers language-specific and safety-relevant features, and produces auditable explanation traces, showing that agent-driven empirical loops yield sharper and more falsifiable explanations than one-shot labels.
Large Language Model (LLM)-based coding agents show promise in automating software development tasks, yet they frequently fail in ways that are difficult for developers to understand and debug. While general-purpose LLMs like GPT can provide ad-hoc explanations of failures, raw execution traces remain challenging to interpret even for experienced developers. We present a systematic explainable AI (XAI) approach that transforms raw agent execution traces into structured, human-interpretable explanations. Our method consists of three key components: (1) a domain-specific failure taxonomy derived from analyzing real agent failures, (2) an automatic annotation system that classifies failures using defined annotation schema, (3) a hybrid explanation generator that produces visual execution flows, natural language explanations, and actionable recommendations. Through a user study with 20 participants (10 technical, 10 non-technical), we demonstrate that our approach enables users to identify failure root causes 2.8 times faster and propose correct fixes with 73% higher accuracy compared to raw execution traces. Importantly, our structured approach outperforms ad-hoc state of the art models explanations by providing consistent, domain-specific insights with integrated visualizations. Our work establishes a framework for systematic agent failure analysis, addressing the critical need for interpretable AI systems in software development workflows
Arun Joshi
Independent Researcher Islington College (Alumini) Nepal
Automated interpretability systems aim to reduce the need for human labor and scale analysis to increasingly large models and diverse tasks. Recent efforts toward this goal leverage large language models (LLMs) at increasing levels of autonomy, ranging from fixed one-shot workflows to fully autonomous interpretability agents. This shift creates a corresponding need to scale evaluation approaches to keep pace with both the volume and complexity of generated explanations. We investigate this challenge in the context of automated circuit analysis -- explaining the roles of model components when performing specific tasks. To this end, we build an agentic system in which a research agent iteratively designs experiments and refines hypotheses. When evaluated against human expert explanations across six circuit analysis tasks in the literature, the system appears competitive. However, closer examination reveals several pitfalls of replication-based evaluation: human expert explanations can be subjective or incomplete, outcome-based comparisons obscure the research process, and LLM-based systems may reproduce published findings via memorization or informed guessing. To address some of these pitfalls, we propose an unsupervised intrinsic evaluation based on the functional interchangeability of model components. Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems and reveals key limitations of replication-based evaluation.