DECODE: Diagnosing Interpretability Failures in Automated Interpretation Process
Authors: Dohyun Kim, Jaeung Lee, Donghee Hong, Jaemin Jo
Organizations: KAIST, Seoul, Republic of Korea · Samsung Research, Seoul, Republic of Korea · University of Maryland, College Park, Maryland, United States · Sungkyunkwan University, Suwon, Republic of Korea
Tools for interpreting the internal representations of Large Language Models (LLMs) increasingly rely on LLMs themselves to both generate and evaluate explanations, creating a self-referential automated interpretation process with minimal human oversight. This automation raises significant reliability concerns, which are exacerbated by the absence of ground truth. In practice, it is difficult to validate explanations or to attribute interpretability failures to their source -- whether they originate from the tool's extracted representations or from the LLM used in the process. To address these challenges, we present DECODE, a visual analytics system that reintroduces human oversight into the automated interpretation process. Our approach decomposes the diagnostic workflow into three progressive stages, isolating failure sources ranging from structural defects to explanation-level misalignments. Drawing inspiration from inter-coder consensus in qualitative analysis, we utilize consensus among independently generated explanations and their evaluation metrics as a proxy for reliability. To further scale human inspection, we employ a three-phase active seeding protocol that strategically recommends high-priority instances for manual review. Finally, we demonstrate the effectiveness of our approach through a case study on 16,384 Sparse Autoencoder features, showing that it can efficiently identify and attribute interpretability failures at scale.
Figures & tables
Figure 1: Overview of DECODE. (A) Automated interpretation process extracts SAE features, generates explanations with three explainers, and evaluates them with three metrics. The red dashed border highlights the self-referential loop. (B) Three-stage diagnostic pipeline sequentially assigns features into five diagnostic categories, passing unresolved features to the next stage via intermediate categories. (C) Three-phase active seeding protocol recommends features based on representativeness, decision boundary proximity, and classifier ensemble disagreement.
Category
Description
Incoherent Splitting
An SAE-driven failure where two features divide a single concept without a clear conceptual boundary.
Well-Explained
A feature whose generated explanations faithfully capture the shared pattern across its activating examples.
Missed Syntax
An explainer-driven failure where the explanations omit a shared surface-level pattern, such as common subwords or morphological affixes.
Missed Context
An explainer-driven failure where the explanations omit a shared contextual pattern, such as topic, domain, or tone.
Noisy Activation
An SAE-driven failure where the feature activates on heterogeneous examples with no coherent pattern.
Table 1: Terminal diagnostic categories used in DECODE.
Figure 2: The iterative labeling workflow within each diagnostic stage. Users screen, inspect, and label features in a loop where each label retrains the classifier for automatic labeling.
Figure 3: Stage 1 interface. (A) Stage tabs indicate diagnostic progress (Stage 1 active). (B) Overview Panel displays a decoder similarity histogram. A draggable threshold at 0.40 separates inspection candidates (above) from features assigned Coherent Separation (below). (C) Workspace Panel: (c1) Seeding List ranks candidate pairs, showing 20 representatives in prototype-first phase; (c2) Inspection View juxtaposes feature 13143 and 13256 both activating on tokens starting with “sta”; (c3) Labeling View displays confidence histogram and stability chart after three training iterations.
Figure 4: Stage 2 interface. (A) Overview Panel screens features by average interpretability score as in Stage 1. Feature 10361 is chosen for inspection: (B) activating examples with absence-related tokens highlighted; (C) the user hovers over the explainer consensus cluster with medoid “missing”; (D) constituent phrases of the hovered cluster highlighted in blue, with per-explainer interpretability scores showing fuzz and detection above random baseline; (E) confidence histogram with draggable thresholds defining the automatic labeling region.
Figure 5: Stage 3 interface for feature 3588. (A) Overview Panel shows all Need Revision features passed from Stage 2 without threshold-based screening. The Inspection View presents: (B) two activating-example panels, with syntax-pattern borders on “$$” and “aligned” (top) and context-pattern borders on semantically similar phrases and distinctive recurring tokens such as “$\” and “\theta” (bottom); (C) expanded explainer consensus clusters listing all constituent phrases with per-explainer scores on hover. The Labeling View adapts to three categories: (D) confidence histogram with tentative labels predominantly Noisy Activation ; (E) category scatter showing distribution skewed toward Noisy Activation .
Stage 1
Stage 2
Stage 3
Labels
Min
/unit
Labels
Min
/unit
Labels
Min
/unit
P1
59
49
0.83
45
42
0.93
34
36
1.06
P2
79
42
0.53
36
27
0.75
52
43
0.83
P3
59
28
0.47
48
38
0.79
61
66
1.08
Table 2: Labeling effort per stage. A label denotes one manually labeled unit: a feature pair in Stage 1, a feature in Stages 2 and 3.
Incoherent Splitting
Well- Explained
Missed Syntax
Missed Context
Noisy Activation
Total
P1
26 (57)
41 (8,403)
3 (70)
2 (731)
16 (7,123)
88 (16,384)
P2
16 (18)
40 (7,567)
6 (1,187)
3 (243)
20 (7,369)
85 (16,384)
P3
14 (42)
61 (7,319)
4 (820)
2 (590)
25 (7,613)
106 (16,384)
Table 3: Category distribution per expert. Well-Explained aggregates assignments made at Stage 2 and at Stage 3. Each cell shows manual feature count with the final total after automatic labeling in parentheses.
Score-only
Individual expert
Expert pair
Group of
baseline
P1
P2
P3
P1-P2
P1-P3
P2-P3
3 experts
F1
0.36
0.40
0.37
0.41
0.53
0.54
0.50
0.62
Table 4: End-to-end macro-F1 on the reference set. Every condition assigns all five diagnostic categories to all 16,384 features, combining manual labels with propagated labels from automatic labeling.