Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding

Gregor Baer; Chao Zhang; Isel Grau; Pieter Van Gorp

cs.HCMar 26, 2026

Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding

Authors: Gregor Baer, Chao Zhang, Isel Grau, Pieter Van Gorp

Organizations: Information Systems Group, Eindhoven University of Technology · Human-Technology Interaction Group, Eindhoven University of Technology

Abstract

Explainable AI (XAI) methods are commonly evaluated using functional correctness metrics, sometimes termed faithfulness or fidelity, which estimate how closely an explanation reflects the model's reasoning. Higher correctness is assumed to produce better human understanding, but this link has not been tested with controlled levels. We conducted a user study (N=200) that manipulated explanation correctness at four levels (100%, 85%, 70%, 55%) in a synthetic time series classification task where participants could not rely on domain knowledge or visual intuition. Correctness was defined against a known ground truth, not estimated from a trained model. Participants predicted a simulated AI's decisions from feature-attribution-style explanations (forward simulation), and we used their forward simulation accuracy as a proxy for understanding. Correctness affected understanding, but not at every level: forward simulation accuracy dropped at 70% and 55% relative to fully correct explanations, with no further reduction below 70% and no conclusive difference between 85% and 100%. Lower correctness reduced how many participants predicted accurately rather than lowering accuracy uniformly, and even fully correct explanations did not guarantee understanding: forward simulation accuracy was bimodal there, with only a subset of participants well above chance. In exploratory analyses, some participants predicted decisions accurately but described the wrong pattern when asked what the AI used, and self-reported ratings correlated with forward simulation accuracy only when explanations were fully correct. These findings show that not all differences in functional correctness translate to differences in human understanding, highlighting the need to validate functional metrics against human outcomes.

Figures & tables

Figure 1: From evaluation paradigms to metric operationalizations for correctness and understanding. The two XAI evaluation paradigms target different constructs: correctness describes the explanation, understanding describes the state of the user. Italics denote alternative names used in the literature; we use functional evaluation throughout the paper. We manipulate correctness by generating heatmaps with controlled temporal overlap with a known ground-truth feature region (100/85/70/55%), rather than estimating correctness from a trained model. We measure understanding objectively through forward simulation accuracy and subjectively through self-reported ratings.

Figure 2: Overview of the experimental design, showing condition assignment, training and test phases, and the post-task survey.

Correctness	$n$	Mean	SD	Man	Woman	Yes	No
		Age		Gender		Technical
100%	51	41.7	12.5	25	26	16	35
85%	51	42.3	10.6	29	22	19	32
70%	50	44.7	11.4	29	20	18	32
55%	48	41.8	11.5	28	19	13	35
Total	200	42.6	11.5	111	87	134	66

Table 1: Participant demographics by correctness condition.

Figure 3: Trial screen as shown to participants during both the training and test phases. Participants see a univariate time series curve overlaid with a heatmap indicating feature importance for the AI’s decision. In the training phase, the next screen shows feedback with the AI’s actual classification; in the test phase, no feedback is provided. The example shown here is from the 70% correctness condition, where the heatmap partially overlaps with the true discriminative feature and partially covers an adjacent non-informative region.

Figure 4: Synthetic data generation. Each time series combined a random walk (left) with a class-discriminative feature (center): a peak (Class A) or valley (Class B) spanning 15% of the series length. The aggregated signal (right) is the sum of both components, with the red region marking the ground truth feature location. The random walk produced peaks and valleys of similar magnitude, making the feature difficult to identify without an explanation.

Figure 5: Explanation generation and correctness manipulation. The true feature location was rendered as a heatmap, corrupted with noise, and smoothed to resemble a feature attribution explanation. Less correct explanations were created by spatially displacing the highlighted region. The example shown corresponds to a displacement of 45%, leaving 55% overlap with the true feature location (i.e., the 55% correctness condition).

	Comparison	Diff.	95% CI	$d$	df	$t_{0}$	$p_{0}$	$t_{\Delta}$	$p_{\Delta}$	Interpretation
Baseline comparisons (vs. 100% correctness)
H1	85% vs 100%	$-.066$	[-.148, .017]	$-.313$	93.02	$-1.579$	.059	$0.825$	.206	Inconclusive
H2	70% vs 100%	$-.146$	[-.222, -.070]	$-.761$	78.31	$-3.844$	1.2e-4	$-1.215$	.886	Meaningful reduction
H3	55% vs 100%	$-.165$	[-.240, -.090]	$-.865$	76.64	$-4.375$	1.9e-5	$-1.727$	.956	Meaningful reduction
Adjacent comparisons
H4	70% vs 85%	$-.081$	[-.142, -.019]	$-.512$	91.71	$-2.582$	.006	$0.625$	.267	Meaningful reduction

Table 2: Effect of explanation correctness on forward simulation accuracy.

Figure 6: Forward simulation accuracy by correctness condition. Each dot represents one participant. Horizontal lines in the violin mark the 25th, 50th, and 75th percentiles; the dashed line marks chance performance (0.5). The 100% condition is slightly bimodal, with participants clustering at either high accuracy or near chance. The 85% condition shows a similar high spread but without clear bimodality. In the 70% and 55% conditions, variance shrinks and accuracy concentrates near chance (0.5).

Figure 7: Cumulative forward simulation accuracy across training trials by correctness condition, averaged over participants. Shaded ribbons indicate 95% confidence intervals (CI); the dashed line marks chance performance (0.5). In the 100% condition, the lower bound of the 95% CI rises above chance by trial 7 and remains there, suggesting learning. In the remaining conditions, the 95% CI includes chance throughout, even if the 85% condition shows slightly higher average accuracy than the 70% and 55% conditions toward the end of training.

	Verbalized decision rule
Correctness	Correct	Incorrect	None	Total
100%	14	31	6	51
85%	17	25	9	51
70%	4	37	9	50
55%	2	33	13	48
Total	37	126	37	200

Table 3: Verbalized decision rule labels by correctness condition.

Figure 8: Relationship between forward simulation accuracy and the verbalized decision rule (coded as correct, incorrect, or none) for the 100% and 85% correct conditions. Each dot is one participant. The dashed line marks chance performance (

.5

) and the dotted line marks a forward simulation accuracy of

.8

, that is, participants predicted 16 or more of the 20 test trials correctly.

Figure 9: Self-reported explanation ratings by correctness condition. Participants rated agreement with five statements adapted from the Explanation Satisfaction Scale on a 5-point Likert scale. Higher correctness conditions show somewhat more positive ratings, but differences across conditions are modest compared to the differences in forward simulation accuracy.

Figure 10: Spearman rank correlation between self-reported explanation ratings and forward simulation accuracy, averaged across conditions. The five subjective rating items correlate strongly with each other (

r=.65

.83

) but weakly with forward simulation accuracy (

r\leq.24

Figure 11: Spearman rank correlation between each self-reported explanation rating item and forward simulation accuracy. Each cell shows a single correlation with accuracy. Unlike Figure 10 , inter-item correlations are not shown. Color encodes correlation on a symmetric

\pm 0.5

scale, set to the observed range (

-0.15

0.43

) rather than the full

\pm 1

; cell labels give exact values. (A) Split by correctness condition. Correlations were weak to moderate in the 100% correct condition (up to

r=.43

) and mostly near zero at 85% and below. (B) The 100% correct condition split at a forward simulation accuracy of

.80

, that is, 16 or more of the 20 test trials correct. At or above the threshold, self-reports correlated weakly to moderately with forward simulation accuracy (

r=.21

.33

); below it, correlations were near zero (

r=-.15

.10

). Together, the panels suggest that self-reports corresponded to forward simulation accuracy mainly when explanations were fully correct and participants predicted the AI’s decisions accurately.

Explainable Artificial Intelligence

Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.

See all 326 papers

CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations
Apr 30, 2026Louth Bin Rawshan, Zhuoyu Wang, Brian Y. LimFeature Attribution Explainable Artificial Intelligence
National University of Singapore, Singapore
Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency maps
Apr 23, 2025Felix Kares, Timo Speith, Hanwei Zhang +1Saliency Map Evaluation Explainability Evaluation
Saarland University, Germany · University of Bayreuth, Germany · University of Freiburg, Germany
Can Conversational XAI Improve User Performance? An Experimental Study
May 19, 2026Sven Kruschel, Julian Rosenberger, Lasse Bohlen +2Explainable Artificial Intelligence Human-in-the-Loop Evaluation
TU Dresden, Dresden, Germany · University of Regensburg, Regensburg, Germany

Apr 30, 2026J/K move · Enter open · S save

CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations

Louth Bin Rawshan, Zhuoyu Wang, Brian Y. Lim

National University of Singapore, Singapore

Feature Attribution Explainable Artificial Intelligence Human-AI Decision Making Explainability Evaluation

Open paper AskPDF arXiv

Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.

Abstract

Explainable Artificial Intelligence

Explore similar work

CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations

Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency maps

Can Conversational XAI Improve User Performance? An Experimental Study