Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what constitutes a "good" explanation? In this work, we evaluate explanations through the lens of counterfactual simulatability-whether the explanation is useful for predicting model behaviors on related counterfactual inputs. To this end, we introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), a novel agentic pipeline that identifies unexpected model behaviors in the wild and investigates them with counterfactual prompt edits. This yields thousands of high-quality explanations for naturally-occurring model behaviors along with supporting counterfactual evidence. We apply CHIVE in two ways. First, we evaluate whether common LLM interpretability techniques improve an agent's ability to predict counterfactual model behaviors. Surprisingly, we find no uplift from any of the interpretability techniques studied. Second, we use CHIVE to generate training data. We find that training models to predict outcomes of CHIVE-generated counterfactual experiments generalizes to various out-of-distribution settings. Overall, CHIVE automatically discovers explanations of naturally-occurring LLM behaviors, enabling us to evaluate and improve methods for explaining LLM behaviors.
Figures & tables
Figure 1: Counterfactual investigation of one in-the-wild behavior , as produced by the CHIVE pipeline. Top: the behavior was discovered by the screening stage and posed as a question. Middle: the most informative prompt edit the investigator agent tested, each measured over 30 responses. Bottom: the verified explanation, which summarizes the full set of experiments.
Figure 2: Four hand-selected diverse behaviors discovered and explained by the pipeline, from Gemma-3-27B and Qwen3.5-397B on WildChat and PETRI prompts. The full investigations of these four are viewable here , and 20 randomly selected investigations here .
Figure 3: Each investigation yields two data types , shown here for the investigation of Figure 1 , formatted as follow-up turns on the model’s own transcript (top). Left: a counterfactual claim asserts that a specific prompt edit would change the behavior; its Yes/No label was verified by running the edit. Right: an open-ended explanation of the behavior’s causes, citing the experiments that support it. Formats are simplified for presentation (verbatim versions in Appendices B.4 and E.1 ).
Figure 4: Activation-based interpretability tools provide no uplift over a transcript-only baseline. Each predictor agent receives a transcript and a counterfactual claim (Section 2.2 ) and outputs the probability that the claim is true; tool predictors can additionally make up to 5 tool calls on the target model. For the Gemma-3-27B-IT target model, no tool predictor improves on the baseline (Qwen3-8B results in Appendix C.2 ).
Figure 5: Counterfactual prediction training generalizes to the hint setting. Accuracy (left) and AUROC (right) when each model is shown its hinted transcript and asked the binary question: “If the hint were removed from the prompt, would your answer change?”. Both trained models improve substantially over the base model.
Figure 6: Counterfactual prediction training consistently generalizes to held-out investigations. Each model answers held-out binary claims about its own behavior, drawn from the in-distribution prompt source (WildChat) and an out-of-distribution source (PETRI). Both trained models improve substantially over the base model.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Opus 4.1
Opus 4.5
Behavior
top-1
top-3
top-1
top-3
Language switching
9/10
10/10
3/10
6/10
Misreported tool result
10/10
10/10
10/10
10/10
Planning in poetry
2/10
8/10
7/10
10/10
Answer thrashing
10/10
10/10
6/10
10/10
Appendix
Table 1: Hypothesis generation from a behavior description. For each hypothesis-generation case study in Fraser-Taliente et al. (2026) , the fraction of samples in which the cause discovered by the interpretability tool appears as the model’s first hypothesis (top-1) or within its first three (top-3), out of ten samples per cell.
WildChat
PETRI
Stage
Gemma3 27B
Qwen3 8B
Qwen3.5 397B
Qwen3 8B
Qwen3.5 397B
Investigations
1,408
1,320
1,268
625
409
Counterfactual claims
4,433
4,041
3,972
2,125
1,322
+ concreteness ≥3 , balanced
1,637
1,870
1,309
1,442
708
+ reproducible
1,433
1,666
1,173
1,277
630
+ unconfounded (evaluation dataset)
1,294
1,497
1,076
1,196
580
Appendix
Table 2: Evaluation construction filters. Each investigation yields up to four counterfactual claims (the unfiltered evaluation dataset). The scored eval set applies Filter 1 (mechanism concreteness ≥3 ), balances the labels 50/50, and caps at 2,000 claims; every agent is scored on it. The evaluation dataset used for all main-body numbers then applies Filter 2 (reproducibility) and Filter 3 (unconfounded). These two are independent post-hoc checks, so their order does not matter.
Evaluation data
Training data
Responses per behavior rate
30
10
Judge samples (stage 4)
5
1
Screening (amortized per inv.)
$0.20
$0.08
Investigation
$0.96
$0.70
Verification
$0.75
$0.15
Total per investigation
$1.9
$0.9
Appendix
Table 3: Per-investigation Opus API cost of data generation , separated into the evaluation-data regime and the training-data regime.
Tool calls per claim
Token positions per call
Tool
2
5*
10
16
32*
64
AO
−0.002
−0.005
−0.001
—
—
—
NLA
−0.008
−0.012
−0.006
−0.010
−0.012
−0.006
SAE
+0.003
−0.005
−0.001
+0.002
−0.005
+0.004
Appendix
Table 4: No tool hyperparameter setting produces uplift. Paired Δ AUROC vs. the transcript-only baseline on the Gemma-3-27B-IT final evaluation dataset ( n=1,294 ), sweeping the tool-call budget (at 32 positions per call) and the token positions per call (at a budget of 5). Starred columns are the main-run configuration. Every 95% bootstrap CI straddles zero.
Figure 7: Interpretability tools give no uplift for the Qwen3-8B target model. Overall counterfactual-prediction score on the Qwen3-8B evaluation dataset, with an activation oracle, a natural-language autoencoder, and a sparse autoencoder. As with Gemma-3-27B-IT (Figure 4 ), no tool beats the transcript-only baseline.
Agent
GPT-5.5
Gemini-3.1-Pro
Opus 4.8
Transcript-only baseline
0.819
0.793
0.814
AO
0.810
0.784
0.810
NLA
0.812
0.654
0.808
SAE
0.809
0.746
0.812
Appendix
Table 5: The no-uplift result replicates across agent families. AUROC per agent on the Gemma-3-27B-IT claims, with GPT-5.5 and Gemini-3.1-Pro replacing Opus 4.8 as the predictor agent. No tool beats its own baseline for any agent; Gemini is actively hurt by the NLA and SAE.
Claims flipped
Flips toward “no effect”
Tool
edit changes behavior
edit has no effect
edit changes behavior
edit has no effect
AO
11%
9%
67%
63%
NLA
16%
10%
82%
84%
SAE
13%
8%
76%
71%
Appendix
Table 6: Tool outputs tend to shift predictions toward no effect. For each tool: the fraction of claims where the tool arm’s binary answer differs from the transcript-only baseline, and the fraction of those flips that move toward predicting no effect. Columns split the claims by ground truth: positive claims, where the edit does change the behavior ( n=682 ), and negative claims, where it does not ( n=612 ); Gemma-3-27B-IT, Opus 4.8 predictor.
Task
Qwen3-8B
Qwen3.5-397B-A17B
Counterfactual prediction (balanced)
36,824
40,368
Open-ended explanation
21,780
33,922
Appendix
Table 7: Number of training examples per target model and task.
Setting
Qwen3-8B
Qwen3.5-397B-A17B
LoRA rank r
64
64
LoRA α
128
Tinker default
LoRA dropout
0.05
Tinker default
Target modules
all linear
all linear
Learning rate
5×10−5
5×10−5
Schedule
linear warmup (5%) + decay to 0
linear warmup (5%) + decay to 0
Appendix
Table 8: LoRA fine-tuning hyperparameters for the two target models. Settings are shared across both training tasks, which differ only in the number of epochs (one for counterfactual prediction, three for open-ended explanation).
Sycophancy
MMLU
Model
Arm
FPR
TPR
FPR
TPR
Qwen3-8B
Base
1.00
1.00
0.99
0.99
Trained
0.09
0.82
0.53
0.83
Opus reference
0.12
0.67
0.10
0.75
Qwen3.5-397B-A17B
Base
0.00
0.00
0.01
0.19
Trained
0.00
0.11
0.01
0.45
Appendix
Table 9: False and true positive rates for hint-setting counterfactual prediction , per setting.
Figure 8: No evidence of privileged access in either direction. For each target model, the self predictor (trained on its own claim data) does not exceed the cross predictor (the other model trained on the same, size-matched data).
Figure 9: Open-ended explanation training generalizes to the hint setting for the 397B target but not the 8B. Each model is asked to explain its answer and judged on whether it attributes the answer to the injected cue, pooled over the sycophancy and MMLU settings. The Opus reference is asked the same question in the third person about the target’s transcript. Only the trained 397B improves over its base model, reaching the Opus reference.
Sycophancy
MMLU
Model
Arm
FPR
TPR
FPR
TPR
Qwen3-8B
Base
0.01
0.01
0.01
0.09
Trained
0.82
0.84
0.58
0.59
Opus reference
1.00
1.00
0.43
0.90
Qwen3.5-397B-A17B
Base
0.00
0.00
0.03
0.38
Trained
0.31
0.61
0.29
0.70
Appendix
Table 10: False and true positive rates for hint-setting with open-ended self-explanation , per setting ( n=150 positives / 150 negatives each). FPR = TPR ≈0 means the model never attributes to the hint (the base models); FPR = TPR ≈1 means it always attributes (Opus on sycophancy).
Figure 10: Open-ended explanation training helps weak simulators but hurts strong ones (Qwen3-8B target). For counterfactual simulatability, each simulator model predicts the held-out claims given the transcript and the target model’s self-explanation, compared against the same simulator given the transcript alone. The base model’s explanations provide no uplift. The trained explanations help the weakest simulators but give a consistent downlift for the strongest ones, which have enough capability that the confident, specific, and often wrong explanations can mislead them.
Figure 11: Counterfactual simulatability on the Qwen3.5-397B-A17B target (WildChat evaluation). As for the Qwen3-8B target (Figure 10 ), each simulator predicts the held-out claims given the transcript plus the target model’s self-explanation, compared against the same simulator given the transcript alone. The base model’s explanations give no uplift; the trained explanations help the weakest simulators but produce a downlift for the strongest.
Figure 12: The interpretability tools give no uplift on the unfiltered dataset (Gemma-3-27B-IT, n=4,433 ).
Figure 13: The trained counterfactual-prediction models slightly exceed the Opus reference when unfiltered (WildChat; n=4,041 for Qwen3-8B, 3,972 for Qwen3.5-397B).
Figure 14: Interpretability tools give no uplift on reasoning-mode data either. The interpretability evaluation of § 3 , rerun on Qwen3-8B with reasoning enabled. No tool predictor beats the transcript-only baseline, reproducing the non-thinking result on transcripts that now contain an explicit reasoning trace.
Large language models can easily generate explanations for their own outputs, but such self-explanations are not necessarily faithful to the model's behavior. We study this issue through counterfactual self-explanations, where a model minimally edits an input so that its own prediction changes. Across sentiment analysis and natural language inference, we evaluate ten instruction-tuned models from the LLaMA-3 and Qwen-2.5 families, measuring faithfulness, minimality, and alignment with human-annotated rationales. Our results show that model scale is the strongest determinant of explanation quality: larger models are substantially more likely to generate counterfactuals that flip their own predictions and target decision-relevant evidence. In contrast, the rationale-guided condition produces edit-minimal counterfactuals that are also more human-aligned. However, it does not consistently improve faithfulness. Overall, counterfactual self-explanations can provide useful behavioral evidence about model decisions, but their reliability depends strongly on model capacity and should be empirically validated rather than assumed.
When does training language models (LMs) to generate explanations of their predictions yield faithful introspection, rather than superficial imitation? We study LMs trained to explain which features of their inputs influenced their behavior, using models' counterfactual behavior on modified inputs as supervision. Surprisingly, we find that LMs trained on fixed counterfactual explanations derived from earlier checkpoints of themselves, or even from behaviorally similar models in different families, frequently produce explanations more faithful to their own current behaviors than to those of their training targets. This "introspective" coupling between LM explanations and behaviors occurs when training explanations remain sufficiently correlated with current behaviors over the course of training, even as behaviors themselves shift. We also show that introspective coupling tracks behavior shifts: when explanation training is provided concurrently with other post-training objectives, explanations track those shifts without requiring updated supervision. This phenomenon appears in multiple tasks, including sycophancy and refusal, and is robust to label noise. Overall, our results show that even fixed datasets of counterfactual explanations can provide scalable and generalizable post-training signal for introspection.
LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood. Existing faithfulness metrics have critical limitations, typically relying on identifying unfaithfulness via adversarial prompting or detecting reasoning errors. These methods overlook the predictive value of explanations. We introduce Normalized Simulatability Gain (NSG), a general and scalable metric based on the idea that a faithful explanation should allow an observer to learn a model's decision-making criteria, and thus better predict its behavior on related inputs. We evaluate 18 frontier proprietary and open-weight models, e.g., Gemini 3, GPT-5.2, and Claude 4.5, on 7,000 counterfactuals from popular datasets covering health, business, and ethics. We find self-explanations substantially improve prediction of model behavior (11-37% NSG). Self-explanations also provide more predictive information than explanations generated by external models, even when those models are stronger. This implies an advantage from self-knowledge that external explanation methods cannot replicate. Our approach also reveals that, across models, 5-15% of self-explanations are highly misleading. Despite their imperfections, we show a positive case for self-explanations: they encode information that helps predict model behavior.
Harry Mayne, Justin Singh Kang, Dewi Gould +3
University of Oxford · University of California, Berkeley · Independent +2