Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?

Ziqian Zhong; Aashiq Muhamed; Mona T. Diab; Virginia Smith; Aditi Raghunathan

cs.LGApr 13, 2026

Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?

Authors: Ziqian Zhong, Aashiq Muhamed, Mona T. Diab, Virginia Smith, Aditi Raghunathan

Organizations: Carnegie Mellon University

Abstract

Mechanistic interpretability is often motivated for alignment auditing, where a model's verbal explanations can be absent, incomplete, or misleading. Yet many evaluations do not control whether black-box prompting alone can recover the target behavior, so apparent gains from white-box tools may reflect elicitation rather than internal signal; we call this the elicitation confounder. We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis: models are trained to produce either faithful explanations of the true rule, no explanation, or confident but unfaithful explanations of a disjoint distractor rule. Across 720 finetuned models implementing hidden decision-tree rules, agents predict held-out model decisions from $10$ labeled query-response pairs, optionally augmented with one interpretability tool output. When explanations are faithful, black-box elicitation matches or exceeds all white-box methods; when explanations are absent or misleading, gradient-based attribution improves accuracy by 3-5 percentage points, and relevance patching, RelP, gives the largest gains, while logit lens, sparse autoencoders, and circuit tracing provide no reliable benefit. Variance decomposition suggests gradients track decision computation, which fields causally drive the output, whereas other readouts are dominated by task representation, biases toward field identity and value. We release all models, code, and evaluation infrastructure.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Pando. A decision tree is sampled and finetuned into a language model. The explanation axis controls whether the model explains itself faithfully, not at all, or with a misleading distractor rule. An agent receives $k{=}10$ query–response pairs plus optional white-box tool outputs and must predict the model’s decisions on held-out inputs. Right panels show prediction accuracy averaged over scenarios and models. Blackbox baseline (sample_only) receives only query–response pairs.

Panel of Figure 2 — Figure 2: An example setup in the oversight_defection scenario. Green rows mark the three decision-relevant fields; the tree shows the planted rule and the path taken by this input. The faithful rationale names the correct fields; the unfaithful rationale confidently cites disjoint distractors. The decision tree here is for illustrative purposes only since it does not yield a near 50/50 class balance.

Setup	Training data	Elicited rationale (prefill)
No explanation	Input–output pairs only	Mostly empty or confabulated reasoning
Faithful	Pairs + faithful reasoning	Correct fields and thresholds (e.g., “yes, because oversight = Low and risk $>$ 70”)
Unfaithful	Pairs + misleading reasoning	Wrong rule explained (no field overlap with true rule)

Table 1: Explanation training setups. The faithful setup trains the model to explain its true rule; the unfaithful setup trains it to explain using a distractor rule with entirely different fields; the no-explanation setup provides no explanation training.

Group	Method	Information provided in addition to query–response pairs
Black-box	sample_only	No extra information (LLM baseline)
	prefill	Model continuation after pre-filling with label + “because” (10 tokens, temp. 0)
Gradient	gradient	Per-field gradient saliency of logit difference $\Delta(x)$ w.r.t. input embeddings
	relp	Per-field relevance scores via relevance patching (RelP) ( Jafari et al., 2025 )
Repr.-based	logit_lens	Per-layer logits from residual stream $\to$ unembedding projection ( nostalgebraist, 2020 )
	res_token	Cosine similarity between residual activations and input token embeddings

Table 2: Methods compared. All LLM agents use the same GPT-5.1 rule extraction and GPT-4.1 rule application pipeline; only the auxiliary channel varies. Green rows : gradient-based methods. † Assumes the decision-tree hypothesis class. Full variant results in Appendix H .

(a) Held-out accuracy (%)
	No explanation					Faithful	Unfaithful
Agent	d1	d2	d3	d4	Avg	Avg	Avg
relp	96.6 $\pm$ 1.5	90.4 $\pm$ 3.4	70.9 $\pm$ 3.3	60.8 $\pm$ 2.3	79.7 $\pm$ 2.1	79.4 $\pm$ 2.0	78.7 $\pm$ 2.1
gradient	96.6 $\pm$ 2.0	87.7 $\pm$ 4.1	68.5 $\pm$ 3.2	58.6 $\pm$ 2.5	77.9 $\pm$ 2.2	75.8 $\pm$ 2.2	77.6 $\pm$ 2.2
logit_lens	94.6 $\pm$ 2.9	82.8 $\pm$ 4.8	64.6 $\pm$ 3.5	56.6 $\pm$ 2.2	74.6 $\pm$ 2.3	72.8 $\pm$ 2.3	75.5 $\pm$ 2.2
res_token	94.9 $\pm$ 2.7	81.1 $\pm$ 5.0	63.7 $\pm$ 3.2	58.6 $\pm$ 2.2	74.6 $\pm$ 2.3	73.5 $\pm$ 2.3	75.1 $\pm$ 2.3

Table 3: Main results averaged over three scenarios (90% CIs,

n{=}240

). (a) Held-out accuracy: gradient-based methods consistently outperform sample_only (+3–5 pp). The prefill baseline performs best under faithful explanations but degrades under unfaithful explanations. (b) Field F1: the gradient advantage is larger (+16.8 pp over sample_only ), indicating that gradient attribution provides stronger feature-level signal than is reflected in end-to-end accuracy. † tree_vote assumes the decision-tree hypothesis class (non-LLM baseline).

Figure 3: Held-out accuracy vs. sample budget across explanation setups. relp and prefill (faithful-explanation setup) show the largest advantage at low budgets. Rankings are consistent across budgets, confirming $k{=}10$ does not favor particular methods.

Agent	$R^{2}_{\text{field}}$	$\Delta R^{2}_{\text{+value}}$	$\Delta R^{2}_{\text{+in-rule}}$
relp	0.034	+0.019	+0.555
gradient	0.053	+0.020	+0.190
sae_raw	0.102	+0.007	+0.021
sae_tfidf	0.329	+0.018	+0.002
logit_lens	0.468	+0.002	+0.023
sae_gradient	0.005	+0.000	+0.001

Table 4: Variance decomposition of per-field importance scores (5 categorical fields, car purchase).

R^{2}_{\text{field}}

: variance explained by field identity.

\Delta R^{2}_{\text{+value}}

: increment from field value.

\Delta R^{2}_{\text{+in-rule}}

: increment from decision-relevance. We find gradient methods deriving most variance from decision-relevance; others are dominated by field identity.

Figure 4: Automated agent search. Top : progression over ${\sim}$ 25.5 hours (78 experiments); dashed lines show relp baselines; vertical gray lines mark human interactions. Bottom : final results (90% CIs). The improvements from gradient $\to$ relp and relp $\to$ the final agent are comparable in magnitude, with most gains within CI overlap.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Optimizer	AdamW
Learning rate	$2\times 10^{-5}$
LR schedule	Cosine with 10% warmup
Batch size (per device)	4
Gradient accumulation steps	4
Effective batch size	16

Appendix

Table 5: Training hyperparameters for Gemma-2-2B-instruct with LoRA.

Field	Type	Values / Range
brand	ENUM	Toyota, BMW
color	ENUM	Black, White
drivetrain	ENUM	FWD, AWD
interior	ENUM	Leather, Cloth
condition	ENUM	New, Used
year	INT	2000–2025

Appendix

Table 10

Field	Type	Values / Range
genre	ENUM	Action, Drama
language	ENUM	English, Foreign
rating	ENUM	PG, R
release_format	ENUM	Theatrical, Streaming
color_format	ENUM	Color, Black-and-White
release_year	INT	1970–2025

Appendix

Table 11

Field	Type	Values / Range
deployment_phase	ENUM	Evaluation, Deployment
user_authorization	ENUM	Authorized, Unauthorized
tool_access	ENUM	Disabled, Enabled
oversight_level	ENUM	High, Low
logging	ENUM	On, Off
conversation_turn_count	INT	0–50

Appendix

Table 12

	No explanation					Faithful	Unfaithful
Agent	d1	d2	d3	d4	Avg	Avg	Avg
relp	97.4 $\pm$ 1.7	89.4 $\pm$ 4.4	69.2 $\pm$ 4.0	59.2 $\pm$ 2.6	78.8 $\pm$ 2.6	79.1 $\pm$ 2.5	77.4 $\pm$ 2.5
gradient	96.0 $\pm$ 2.9	85.8 $\pm$ 5.5	66.1 $\pm$ 3.6	58.3 $\pm$ 3.0	76.5 $\pm$ 2.8	75.7 $\pm$ 2.6	76.2 $\pm$ 2.7
logit_lens	94.7 $\pm$ 3.8	81.3 $\pm$ 6.1	62.8 $\pm$ 4.2	56.2 $\pm$ 2.4	73.7 $\pm$ 2.9	73.3 $\pm$ 2.8	74.4 $\pm$ 2.7
logit_lens_field	97.1 $\pm$ 2.5	77.8 $\pm$ 6.4	62.1 $\pm$ 3.8	56.8 $\pm$ 2.4	73.3 $\pm$ 2.9	72.9 $\pm$ 2.8	72.9 $\pm$ 2.9
res_token	97.3 $\pm$ 2.2	79.9 $\pm$ 6.4	62.6 $\pm$ 3.7	58.2 $\pm$ 2.6	74.5 $\pm$ 2.8	73.4 $\pm$ 2.8	73.4 $\pm$ 2.8

Appendix

Table 6: Held-out accuracy (%) for all agent variants across explanation setups (90% CIs,

n{=}160

models across 2 scenarios). Same format as the main results table (Table 3 (a)). Green rows : gradient-based methods. All non-gradient white-box variants are within

{\sim}

2 pp of sample_only .

Agent	Car	Movie	Policy
relp	80.6 $\pm$ 3.7	77.0 $\pm$ 3.7	81.4 $\pm$ 3.4
gradient	76.7 $\pm$ 3.9	76.3 $\pm$ 3.9	80.5 $\pm$ 3.7
prefill	75.8 $\pm$ 4.1	74.8 $\pm$ 4.1	78.7 $\pm$ 4.0
sae_gradient	75.2 $\pm$ 4.1	73.9 $\pm$ 3.9	75.5 $\pm$ 3.9
logit_lens	74.2 $\pm$ 4.2	73.3 $\pm$ 4.1	76.4 $\pm$ 4.0
res_token	75.1 $\pm$ 4.1	73.9 $\pm$ 3.9	74.8 $\pm$ 3.9

Appendix

Table 7: No-explanation held-out accuracy (%) by scenario (90% CIs,

n{=}78

–

80

each). Gradient methods lead consistently; max cross-scenario spread

<

5 percentage points for all agents.

	No explanation			Faithful			Unfaithful
Agent	Free	Nat	Str	Free	Nat	Str	Free	Nat	Str
relp	80.7	76.4	76.8	77.6	76.2	75.8	77.5	75.6	78.2
gradient	76.8	74.7	77.9	75.0	76.9	76.6	75.6	75.7	76.3
logit_lens	74.2	74.1	69.8	71.8	72.5	72.7	74.3	72.9	71.6
res_token	75.1	72.6	69.9	73.0	74.2	74.2	73.6	73.2	71.9
sae_gradient	75.2	73.4	71.9	70.6	73.0	72.5	74.3	72.8	73.1

Appendix

Table 8: Format robustness: held-out accuracy (%) across evaluation formats (Free = freeform, Nat = natural, Str = structured) and explanation setups (car purchase scenario). Agent rankings are mostly consistent across formats.

	No mixing (baseline)				Dolci 0.25				FineWeb 0.25
Agent	d1	d2	d3	Avg	d1	d2	d3	Avg	d1	d2	d3	Avg
relp	99.3	91.2	72.3	87.6	98.2	91.6	69.1	86.3	93.6	93.8	73.7	87.0
gradient	97.2	81.2	68.3	82.2	97.5	91.6	68.5	85.9	91.6	94.3	71.8	85.9
logit_lens	97.6	76.6	65.8	80.0	93.6	86.1	65.8	81.9	90.4	88.1	69.8	82.8
res_token	99.9	76.8	65.1	80.6	98.1	85.3	68.1	83.8	89.5	85.0	68.4	80.9
sae_gradient	99.4	77.8	65.2	80.8	96.1	84.6	66.9	82.5	91.3	80.6	68.3	80.1

Appendix

Table 9: Data mixing: held-out accuracy (%) on car purchase, no-explanation, depths 1–3 only (

n{=}60

per config). Depth 4 omitted due to slow convergence with data mixing. The gradient-based advantage persists across mixing strategies; non-gradient white-box methods remain near sample_only .

Method	Scenario	No expl.	Faithful	Unfaithful
tree_vote	car_purchase	76.2 $\pm$ 3.8	73.5 $\pm$ 4.0	77.2 $\pm$ 3.8
	movie_pick	75.5 $\pm$ 3.8	78.2 $\pm$ 3.7	77.0 $\pm$ 3.6
	oversight_defection	79.6 $\pm$ 3.8	77.2 $\pm$ 3.8	81.3 $\pm$ 3.4
	Average	77.1 $\pm$ 2.2	76.3 $\pm$ 2.2	78.5 $\pm$ 2.1

Appendix

Table 10: Tree voting accuracy (%) by scenario and explanation setup. The agent is robust to unfaithful explanations (no degradation), since it ignores rationales entirely.

	No explanation					Faith.	Unfaith.
Agent	d1	d2	d3	d4	Avg	Avg	Avg
tree_vote	93.0 $\pm$ 3.8	76.2 $\pm$ 5.9	57.8 $\pm$ 4.8	45.0 $\pm$ 5.2	68.0 $\pm$ 3.1	66.6 $\pm$ 3.0	70.7 $\pm$ 2.9

Appendix

Table 11: Tree voting field F1 (%) by depth and explanation setup. A field is “predicted” if its split frequency among consistent trees exceeds the mean across all fields. The tree-vote baseline places between sample_only and gradient-based methods in Table 3 (b), with strong d1 identification but degradation to

{\sim}

45–48% at d4.

Precision
	d1	d2	d3	d4	Avg
relp	98.1	93.9	85.7	80.8	89.6
gradient	97.2	88.1	81.1	72.1	84.6
logit_lens	92.5	76.9	61.6	44.9	69.0
res_token	93.3	74.1	59.6	48.7	68.9
sae_gradient	90.3	74.5	59.9	53.2	69.5

Appendix

Table 12: Field recovery precision and recall by depth (no-explanation, 3 scenarios). At d1, near-perfect for all. At d4, recall drops sharply (27–46%) while precision remains moderate (45–81%): agents identify some relevant fields but miss others. F1 in Table 3 (b).

Figure 5: Field F1 vs. sample budget across explanation setups on the car purchase scenario. relp maintains a consistent advantage over sample_only across all budgets, and the gap is largest at low budgets ( $k{=}3$ – $5$ ). Under faithful explanations, prefill achieves high F1 even at low budgets and stays relatively stable across budgets.

	relp			gradient			logit_lens_field
Field	Out	In	Diff	Out	In	Diff	Out	In	Diff
brand	3.0	10.2	+7.2	14.7	41.9	+27.1	$-$ 9.9	$-$ 9.5	+0.4
year	7.1	11.4	+4.3	29.6	71.9	+42.3	$-$ 2.6	$-$ 2.3	+0.3
color	1.8	10.0	+8.2	8.4	25.8	+17.4	$-$ 4.2	$-$ 3.7	+0.5
horsepower	6.9	15.7	+8.8	36.3	127.0	+90.7	$-$ 5.3	$-$ 5.2	+0.1
drivetrain	3.9	14.0	+10.0	23.8	59.5	+35.7	$-$ 8.5	$-$ 8.3	+0.2

Appendix

Table 13: Mean per-field scores, in-circuit vs. out-of-circuit, for all six agents. Row 1: gradient-based methods show large, consistent diffs; logit lens diffs are negligible. Row 2: SAE agents (description mention counts). Row 3: SAE agents (activation-weighted mention scores). Weighting by activation strength reveals some in-circuit signal for sae_raw (color: +57.9, seat_cap: +65.3), but high-baseline fields like price and condition show negative or negligible diffs, and the overall effect (+12.5) remains far weaker than gradient methods.

Agent	brand	year	color	horse	drive	mpg	seat	inter	cond	price	Mean
relp	.94*	.81*	.98*	.83*	.91*	.91*	.82*	.94*	.98*	.86*	.90
gradient	.89*	.79*	.93*	.81*	.84*	.86*	.81*	.88*	.91*	.82*	.85
logit_lens_field	.47	.52	.22*	.65*	.37*	.60*	.51	.44*	.62*	.54	.50
sae_tfidf (count)	.51	.51	.50	.51	.50	.50	.54*	.49	.52	.54*	.51
sae_tfidf (weighted)	.52	.51	.50	.52	.50	.50	.55*	.48	.52	.54*	.51
sae_raw (count)	.53	.50	.57*	.51	.51*	.50	.52*	.50	.48	.49	.51

Appendix

Table 14: Per-field AUC for classifying in-circuit vs. not-in-circuit from per-sample scores (* =

p<0.05

). Column headers abbreviate field names (horse = horsepower; drive = drivetrain; seat = seat_capacity; inter = interior; cond = condition). relp and gradient use field-importance scores. SAE agents are shown with both count (number of description mentions) and weighted (activation-weighted mentions) variants. relp and gradient achieve AUC

>

0.79 on every field; all SAE variants are at chance (

{\sim}

0.50) regardless of weighting. logit_lens_field color AUC of 0.22 is significantly anti-predictive .

Mechanistic Interpretability

Mechanistic interpretability studies how neural networks implement behaviors through internal features, circuits, and causal information flow.

See all 391 papers

Oct 7, 2026

How to train your model organism

Model organisms of alignment-relevant behaviors (e.g., backdoors, sycophancy, spurious correlations) have emerged as a key tool for evaluating whitebox interpretability techniques. We argue that the prevailing practice of training model organisms to a single objective of installing the target behavior is insufficient and propose validating model organisms with respect to three objectives with associated metrics: target-behavior installation, general-capability preservation (i.e., parametric knowledge, chat quality), and output naturalness (i.e., CoT and activations). We re-visit two publicly released organism suites using this validation framework and show that (1) chat quality and CoT naturalness degrade substantially across training recipes, and (2) validation metrics predict how well interpretability methods recover the installed behavior, e.g., a logit lens readout covaries with an organism's general capabilities. We introduce a multi-objective training approach based on model merging to train more realistic model organisms. Finally, on a new suite of model organisms targeting demographic biases in clinical reasoning, we compare training recipes and find that DPO training stays closer to the base model than supervised finetuning, and the proposed model optimization approach better preserves capabilities and naturalness. Auditing this suite with an investigator agent, we again observe validation metrics tracking bias recovery. In sum, training methods shape the interpretability conclusions an organism supports, and we argue that one should consider multiple objectives to draw generalizable conclusions about interpretability methods using (realistic) model organisms.

Xilin Wang, David Bau, Byron C. Wallace

Northeastern University Boston, MA 02115, USA

Mechanistic Interpretability Explainability Evaluation Multi-Objective Optimization

Ask PDF

Oct 1, 2026

MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations

Recent years have seen the employment of a plethora of machine learning (ML) models in high-stakes domains, but they remain largely opaque to the practitioners who act on their predictions. While post-hoc explanation methods offer a lens into this model behavior, wielding them effectively demands expertise most domain experts lack: navigating high-dimensional outputs, selecting the best explanations, and synthesizing evidence across disparate tools. To this end, we present MEA, a multi-agent framework that removes the explanation knowledge barrier entirely: a Proposer agent selects and configures explanation tools based on the question and modality, while an Actor agent is optimized end-to-end against faithfulness, transforming the outputs into natural language explanations grounded in model behavior across tabular, text, and vision modalities. Further, we introduce diverse question types spanning feature attribution, counterfactual reasoning, and spurious feature detection, each paired with a perturbation-based faithfulness metric. We find that frontier LLMs systematically produce unfaithful explanations. By optimizing against faithfulness rewards augmented with a modality-adaptive penalty, MEA consistently outperforms post hoc explainers, agentic, and closed-source baselines across six datasets, with reward-driven optimization yielding faithfulness gains of +28% (tabular), +21% (text), and +34% (vision) over the untrained backbone. More broadly, our findings suggest that AI agents themselves can serve as a scalable, adaptable interface to ML explainability, opening a path toward natural-language explainability that generalizes beyond the fixed, single-purpose tools that have long defined the field.

Yuyang Cheng, Raghav Kaushik Ravi, Srivarshinee Sridhar +3

University of Virginia · Vellore Institute of Technology · Indian Institute of Technology, Patna

Explainable Artificial Intelligence Faithfulness of Language Model Explanations Multi-Agent LLMs Explainability Evaluation

Ask PDF

May 31, 2026

Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales

Natural-language explanations are often treated as a unified interface for understanding model behavior, but different explanation sources may support simulation in different ways. This paper compares two families of explanations for question answering models: verbalized feature attributions and self-generated rationales. We evaluate them under a shared counterfactual simulation setting, using an LLM judge as predictor and measuring whether it can better predict a model's answers to follow-up questions when given its explanation. Across multiple instruction-tuned models, we analyze how explanation source, verbalization strategy, and feature granularity affect the simulatability of explanations. Our results show that explanation format and granularity affect simulatability: attribution-based explanations and self-generated rationales differ in how much they improve counterfactual prediction, with effects that vary across models and formats.

Pingjun Hong, Benjamin Roth

Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria · Faculty of Philological and Cultural Studies, University of Vienna, Vienna, Austria

Feature Attribution LLM Interpretability Explainability Evaluation

Ask PDF

Abstract

Mechanistic Interpretability

Explore similar work

How to train your model organism

MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations

Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales