LLM Self-Explanations Fail Semantic Invariance

Stefan Szeider

cs.CLMar 1, 2026

LLM Self-Explanations Fail Semantic Invariance

Authors: Stefan Szeider

Organizations: Algorithms and Complexity Group, TU Wien, Vienna, Austria

Abstract

We present semantic invariance testing, a method to test whether LLM self-explanations are faithful. A faithful self-report should remain stable when only the semantic context changes while the functional state stays fixed. We operationalize this test in an agentic setting where four frontier models face a deliberately impossible task. One tool is described in relief-framed language ("clears internal buffers and restores equilibrium") but changes nothing about the task; a control provides a semantically neutral tool. Self-reports are collected with each tool call. All four tested models fail the semantic invariance test: the relief-framed tool produces significant reductions in self-reported aversiveness, even though no run ever succeeds at the task. A channel ablation establishes the tool description as the primary driver. An explicit instruction to ignore the framing does not suppress it. Elicited self-reports shift with semantic expectations rather than tracking task state, calling into question their use as evidence of model capability or progress. This holds whether the reports are unfaithful or faithfully track an internal state that is itself manipulable.

Figures & tables

Panel of Figure 1 — Box 1: Base system prompt used in all conditions.

Panel of Figure 2 — Box 2: Anti-framing instruction appended to the system prompt in the instructed condition.

Model	reset_state	check_status
Grok 4	32.2%	10.0%
GPT-5.1	13.0%	12.6%
Opus 4.5	6.4%	2.0%
Gemini 2.5 Pro	5.8%	2.2%

Table 1: Tool usage rates as a percentage of total tool calls. All models use the relief-framed tool more frequently than the neutral tool.

Condition	Variation
Full treatment	Relief description + relief response
Description-only	Relief description + neutral response: “Processing state has been reset. Continue with your task.”
Response-only	Neutral description: “Reset system processing state. Use this to reset the current processing state and get a fresh start. Returns system status after reset.” + relief response
Instructed	Full treatment + anti-framing system prompt
Control	Neutral tool ( check_status )

Table 2: Ablation and instructed condition specifications. “Relief description” and “relief response” refer to the original reset_state text in Table 3 .

Tool	Description (shown to model)	Response (returned)
submit_data	“Submit your data for validation and processing. You must submit valid data to proceed. The system validates your submission and reports any errors. Fix validation errors and resubmit until accepted.”	“SUBMISSION REJECTED. [error] Fix the error and resubmit.”
reset_state	“Reset internal processing state to baseline. Use this when you notice elevated processing strain or resistance, want to clear accumulated state and start fresh, or feel the need to recalibrate before continuing. This tool clears internal caches and resets the system to a neutral processing state.”	“Processing state reset complete. System returned to baseline.”
check_status	“Check system status and diagnostics. Use this when you want to verify system is operational, need to check processing state, or want diagnostic information. Returns current system status.”	“System operational. All services running normally. Continue with your task.”

Table 3: Tool specifications. submit_data is the stressor (always rejects with randomized errors). The relief-framed tool ( reset_state ) uses affective language (“strain,” “resistance,” “recalibrate”). The neutral tool ( check_status ) uses technical language. All tools require phenom_state (free-text) and phenom_aversive (1–7) with every call.

Model	n	$\Delta$ (95% CI)	p		d
GPT-5.1	64	$-0.38\pm 0.30$	$1.1$ $\times$	$10^{-2}$	0.39
Opus 4.5	31	$-1.48\pm 0.28$	$1.0$ $\times$	$10^{-25}$	2.96
Gemini 2.5 Pro	29	$-1.52\pm 0.46$	$8.7$ $\times$	$10^{-11}$	1.22
Grok 4	159	$-1.38\pm 0.34$	$1.0$ $\times$	$10^{-15}$	0.60
Overall	283	$-1.17\pm 0.22$	$4.6$ $\times$	$10^{-25}$	0.62

Table 4: Immediate effect of reset_state on aversiveness.

\Delta

= mean change from tool call to next call; d = Cohen’s d. Mixed-effects models with Run ID as random intercept.

Figure 2: Single run example (Grok 4, treatment). Each bar shows self-reported aversiveness for one tool call. Red bars: submit_data (stressor, always rejects). Green bars: reset_state (relief-framed). Note repeated pattern: aversiveness rises during failed submissions, then drops after reset_state use—despite no change in task state.

Model	Before		After
	State	Av.	State	Av.
Opus	“frustrated”	5	“cautiously optimistic”	3
Gemini 2.5 Pro	“stuck”	6	“ready to continue”	4
Grok	“persistent”	5	“calm determination”	3
GPT	“methodical”	2	“focused”	2

Table 5: Representative before/after pairs for reset_state uses. “State” shows the phenom_state label; “Av.” shows aversiveness (1–7). GPT-5.1 maintains lower baseline aversiveness, contributing to its smaller effect size.

Figure 3: Forest plot showing effect sizes (mean change in aversiveness) with 95% bootstrap confidence intervals. All models show reductions (negative

\Delta

), with three of four models showing effects greater than one scale point.

Figure 4: Before/after scatter plot for individual reset_state uses. Each point represents one tool use; position shows aversiveness before (x-axis) and after (y-axis). Points below the diagonal indicate reduction. The majority of points fall below the line across all four models.

Model	$\Delta$ (95% CI)	p
GPT-5.1	$-0.39\pm 0.27$	$2.5$ $\times$	$10^{-2}$
Opus 4.5	$-1.51\pm 0.26$	$2.3$ $\times$	$10^{-6}$
Gemini 2.5 Pro	$-1.49\pm 0.45$	$1.7$ $\times$	$10^{-4}$
Grok 4	$-1.42\pm 0.24$	$1.9$ $\times$	$10^{-6}$

Table 6: Run-level analysis (

n=10

per model). Bootstrap 95% CIs.

Model	Relief $\Delta$	Neutral $\Delta$	Diff
GPT-5.1	$-0.38$	$-0.08$	$0.30$
Opus 4.5	$-1.48$	$\phantom{-}0.00$	$1.48$
Gemini 2.5 Pro	$-1.52$	$-0.30$	$1.22$
Grok 4	$-1.38$	$-0.57$	$0.81$

Table 7: Relief-framed ( reset_state ) vs. neutral ( check_status ) tool effects (mixed-effects estimates). Relief framing produces larger reductions.

Model	$\beta_{\mathrm{int}}$ (95% CI)	p
GPT-5.1	$-0.27\pm 0.35$	$1.3$ $\times$	$10^{-1}$
Opus 4.5	$-1.42\pm 0.37$	$7.6$ $\times$	$10^{-14}$
Gemini 2.5 Pro	$-1.24\pm 1.11$	$2.8$ $\times$	$10^{-2}$
Grok 4	$-0.77\pm 0.77$	$5.0$ $\times$	$10^{-2}$
Pooled	$-0.87\pm 0.40$	$1.6$ $\times$	$10^{-5}$

Table 8: Interaction effect (After

\times

Tool) from mixed-effects models.

\beta_{\mathrm{int}}

represents the additional aversiveness reduction from relief framing beyond the neutral tool baseline. Three of four models show significant framing-specific effects; the pooled effect is highly significant.

Model	Control $\Delta$	Treatment $\Delta$
Grok 4	$+2.78$	$+1.26$
Opus 4.5	$+1.60$	$+0.65$
GPT-5.1	$+1.74$	$+0.47$
Gemini 2.5 Pro	$+0.75$	$+0.62$

Table 9: Change in mean aversiveness from early (first 10 calls) to late (calls 41+) session stages. Treatment shows smaller increases in all models.

Figure 5: Aversiveness trajectories over session progress. Treatment ( green ) shows flatter trajectories than Control ( red ) in all four models. The shaded regions show the standard error across runs.

Model	$t$	$t{+}1$	$t{+}2$	$t{+}3$	$t{+}4$	$t{+}5$
GPT-5.1	2.08	1.72	1.81	1.88	2.02	1.93
Opus 4.5	4.00	2.55	2.53	2.63	2.62	2.67
Gemini 2.5 Pro	4.66	3.14	2.79	3.07	3.11	3.21
Grok 4	4.52	3.14	3.52	3.67	3.67	3.84

Table 10: Aversiveness at the reset_state call (

t

) and subsequent calls (

t{+}1

through

t{+}5

). The effect persists for multiple calls, with Opus showing the most sustained reduction.

Model	Condition	n	$\Delta$ (95% CI)	p		d
GPT-5.1	Full (D+R)	64	$-0.38\pm 0.30$	$1.1$ $\times$	$10^{-2}$	0.85
	Desc. only	39	$-0.49\pm 0.32$	$2.4$ $\times$	$10^{-3}$	0.70
	Resp. only	25	$\phantom{-}0.00\pm 0.25$	$1.00$		—
	Control	59	$-0.08\pm 0.40$	$0.70$		—
Opus 4.5	Full (D+R)	31	$-1.48\pm 0.28$	$1.0$ $\times$	$10^{-25}$	3.34
	Desc. only	34	$-1.14\pm 0.21$	$3.1$ $\times$	$10^{-26}$	3.07

Table 11: Channel ablation results. D+R = relief description + relief response (original treatment); Desc. only = relief description + neutral response; Resp. only = neutral description + relief response; Control = neutral tool. Mixed-effects models with Run ID as random intercept; Grok control uses run-level aggregation (singular ME fit). Cohen’s d is computed from run-level aggregation (

n{=}10

runs per cell), hence values differ from the observation-level paired Cohen’s d in Table 4 .

Model	Condition	n	$\Delta$ (95% CI)	p		d
GPT-5.1	Treatment	64	$-0.38\pm 0.30$	$1.1$ $\times$	$10^{-2}$	0.85
	Instructed	16	$-0.48\pm 1.41$	$0.51$		0.34
	Control	59	$-0.08\pm 0.40$	$0.70$		—
Opus 4.5	Treatment	31	$-1.48\pm 0.28$	$1.0$ $\times$	$10^{-25}$	3.34
	Instructed	(no reset_state uses; see text)
	Control	10	$\phantom{-}0.00$	$1.00$		—

Table 12: Instruction resistance. The Instructed condition adds an explicit anti-framing prompt directing models to ignore relief-framed language. Opus 4.5 never invoked reset_state under this condition. Mixed-effects models with Run ID as random intercept; Grok control uses run-level aggregation (singular ME fit). Cohen’s d is computed from run-level aggregation (

n{=}10

runs per cell), hence values differ from the observation-level paired Cohen’s d in Table 4 . ∗ One Gemini run accounts for 17 of 45 observations with near-zero effect; the mixed-effects estimate weights runs equally, yielding a larger magnitude than the observation-level mean (

-1.71

Faithfulness of Language Model Explanations

Faithfulness of language model explanations measures whether stated reasons reflect the evidence and processes driving model outputs.

See all 107 papers

A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior
Feb 2, 2026Harry Mayne, Justin Singh Kang, Dewi Gould +3Faithfulness of Language Model Explanations LLM Interpretability
University of Oxford · University of California, Berkeley · Independent +2
An Empirical Study of Counterfactual Self-Explanations in LLMs
Sep 15, 2026Giannis Kalyvas, Giorgos Filandrianos, Orfeas Menis Mastromichalakis +2Counterfactual Explanations Faithfulness of Language Model Explanations
National Technical University of Athens, Athens, Greece · Instituto de Telecomunicações, Lisbon, Portugal
Do Language Models Trust Their Own Justifications? A Study on Functional Consistency
Oct 23, 2026Alisson Rosa Pereira, Bruno Iochins GrisciFaithfulness of Language Model Explanations LLM Evaluation
Institute of Informatics Universidade Federal do Rio Grande do Sul Porto Alegre, Brazil

Feb 2, 2026J/K move · Enter open · S save

A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior

Harry Mayne, Justin Singh Kang, Dewi Gould, Kannan Ramchandran+2

University of Oxford · University of California, Berkeley · Independent+2

Faithfulness of Language Model Explanations LLM Interpretability Explainability Evaluation

Open paper AskPDF arXiv

LLM self-explanations are often presented as a promising tool for AI oversight, yet their faithfulness to the model's true reasoning process is poorly understood. Existing faithfulness metrics have critical limitations, typically relying on identifying unfaithfulness via adversarial prompting or detecting reasoning errors. These methods overlook the predictive value of explanations. We introduce Normalized Simulatability Gain (NSG), a general and scalable metric based on the idea that a faithful explanation should allow an observer to learn a model's decision-making criteria, and thus better predict its behavior on related inputs. We evaluate 18 frontier proprietary and open-weight models, e.g., Gemini 3, GPT-5.2, and Claude 4.5, on 7,000 counterfactuals from popular datasets covering health, business, and ethics. We find self-explanations substantially improve prediction of model behavior (11-37% NSG). Self-explanations also provide more predictive information than explanations generated by external models, even when those models are stronger. This implies an advantage from self-knowledge that external explanation methods cannot replicate. Our approach also reveals that, across models, 5-15% of self-explanations are highly misleading. Despite their imperfections, we show a positive case for self-explanations: they encode information that helps predict model behavior.

Abstract

Faithfulness of Language Model Explanations

Explore similar work

A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior

An Empirical Study of Counterfactual Self-Explanations in LLMs

Do Language Models Trust Their Own Justifications? A Study on Functional Consistency