Explainable AI (XAI) methods are commonly evaluated using functional correctness metrics, sometimes termed faithfulness or fidelity, which estimate how closely an explanation reflects the model's reasoning. Higher correctness is assumed to produce better human understanding, but this link has not been tested with controlled levels. We conducted a user study (N=200) that manipulated explanation correctness at four levels (100%, 85%, 70%, 55%) in a synthetic time series classification task where participants could not rely on domain knowledge or visual intuition. Correctness was defined against a known ground truth, not estimated from a trained model. Participants predicted a simulated AI's decisions from feature-attribution-style explanations (forward simulation), and we used their forward simulation accuracy as a proxy for understanding. Correctness affected understanding, but not at every level: forward simulation accuracy dropped at 70% and 55% relative to fully correct explanations, with no further reduction below 70% and no conclusive difference between 85% and 100%. Lower correctness reduced how many participants predicted accurately rather than lowering accuracy uniformly, and even fully correct explanations did not guarantee understanding: forward simulation accuracy was bimodal there, with only a subset of participants well above chance. In exploratory analyses, some participants predicted decisions accurately but described the wrong pattern when asked what the AI used, and self-reported ratings correlated with forward simulation accuracy only when explanations were fully correct. These findings show that not all differences in functional correctness translate to differences in human understanding, highlighting the need to validate functional metrics against human outcomes.
Figures & tables
Figure 1: From evaluation paradigms to metric operationalizations for correctness and understanding. The two XAI evaluation paradigms target different constructs: correctness describes the explanation, understanding describes the state of the user. Italics denote alternative names used in the literature; we use functional evaluation throughout the paper. We manipulate correctness by generating heatmaps with controlled temporal overlap with a known ground-truth feature region (100/85/70/55%), rather than estimating correctness from a trained model. We measure understanding objectively through forward simulation accuracy and subjectively through self-reported ratings.
Figure 2: Overview of the experimental design, showing condition assignment, training and test phases, and the post-task survey.
Age
Gender
Technical
Correctness
n
Mean
SD
Man
Woman
Yes
No
100%
51
41.7
12.5
25
26
16
35
85%
51
42.3
10.6
29
22
19
32
70%
50
44.7
11.4
29
20
18
32
55%
48
41.8
11.5
28
19
13
35
Total
200
42.6
11.5
111
87
134
66
Table 1: Participant demographics by correctness condition.
Figure 3: Trial screen as shown to participants during both the training and test phases. Participants see a univariate time series curve overlaid with a heatmap indicating feature importance for the AI’s decision. In the training phase, the next screen shows feedback with the AI’s actual classification; in the test phase, no feedback is provided. The example shown here is from the 70% correctness condition, where the heatmap partially overlaps with the true discriminative feature and partially covers an adjacent non-informative region.
Figure 4: Synthetic data generation. Each time series combined a random walk (left) with a class-discriminative feature (center): a peak (Class A) or valley (Class B) spanning 15% of the series length. The aggregated signal (right) is the sum of both components, with the red region marking the ground truth feature location. The random walk produced peaks and valleys of similar magnitude, making the feature difficult to identify without an explanation.
Figure 5: Explanation generation and correctness manipulation. The true feature location was rendered as a heatmap, corrupted with noise, and smoothed to resemble a feature attribution explanation. Less correct explanations were created by spatially displacing the highlighted region. The example shown corresponds to a displacement of 45%, leaving 55% overlap with the true feature location (i.e., the 55% correctness condition).
Comparison
Diff.
95% CI
d
df
t0
p0
tΔ
pΔ
Interpretation
Baseline comparisons (vs. 100% correctness)
H1
85% vs 100%
−.066
[-.148, .017]
−.313
93.02
−1.579
.059
0.825
.206
Inconclusive
H2
70% vs 100%
−.146
[-.222, -.070]
−.761
78.31
−3.844
1.2e-4
−1.215
.886
Meaningful reduction
H3
55% vs 100%
−.165
[-.240, -.090]
−.865
76.64
−4.375
1.9e-5
−1.727
.956
Meaningful reduction
Adjacent comparisons
H4
70% vs 85%
−.081
[-.142, -.019]
−.512
91.71
−2.582
.006
0.625
.267
Meaningful reduction
Table 2: Effect of explanation correctness on forward simulation accuracy.
Figure 6: Forward simulation accuracy by correctness condition. Each dot represents one participant. Horizontal lines in the violin mark the 25th, 50th, and 75th percentiles; the dashed line marks chance performance (0.5). The 100% condition is slightly bimodal, with participants clustering at either high accuracy or near chance. The 85% condition shows a similar high spread but without clear bimodality. In the 70% and 55% conditions, variance shrinks and accuracy concentrates near chance (0.5).
Figure 7: Cumulative forward simulation accuracy across training trials by correctness condition, averaged over participants. Shaded ribbons indicate 95% confidence intervals (CI); the dashed line marks chance performance (0.5). In the 100% condition, the lower bound of the 95% CI rises above chance by trial 7 and remains there, suggesting learning. In the remaining conditions, the 95% CI includes chance throughout, even if the 85% condition shows slightly higher average accuracy than the 70% and 55% conditions toward the end of training.
Verbalized decision rule
Correctness
Correct
Incorrect
None
Total
100%
14
31
6
51
85%
17
25
9
51
70%
4
37
9
50
55%
2
33
13
48
Total
37
126
37
200
Table 3: Verbalized decision rule labels by correctness condition.
Figure 8: Relationship between forward simulation accuracy and the verbalized decision rule (coded as correct, incorrect, or none) for the 100% and 85% correct conditions. Each dot is one participant. The dashed line marks chance performance ( .5 ) and the dotted line marks a forward simulation accuracy of .8 , that is, participants predicted 16 or more of the 20 test trials correctly.
Figure 9: Self-reported explanation ratings by correctness condition. Participants rated agreement with five statements adapted from the Explanation Satisfaction Scale on a 5-point Likert scale. Higher correctness conditions show somewhat more positive ratings, but differences across conditions are modest compared to the differences in forward simulation accuracy.
Figure 10: Spearman rank correlation between self-reported explanation ratings and forward simulation accuracy, averaged across conditions. The five subjective rating items correlate strongly with each other ( r=.65 to .83 ) but weakly with forward simulation accuracy ( r≤.24 ).
Figure 11: Spearman rank correlation between each self-reported explanation rating item and forward simulation accuracy. Each cell shows a single correlation with accuracy. Unlike Figure 10 , inter-item correlations are not shown. Color encodes correlation on a symmetric ±0.5 scale, set to the observed range ( −0.15 to 0.43 ) rather than the full ±1 ; cell labels give exact values. (A) Split by correctness condition. Correlations were weak to moderate in the 100% correct condition (up to r=.43 ) and mostly near zero at 85% and below. (B) The 100% correct condition split at a forward simulation accuracy of .80 , that is, 16 or more of the 20 test trials correct. At or above the threshold, self-reports correlated weakly to moderately with forward simulation accuracy ( r=.21 to .33 ); below it, correlations were near zero ( r=−.15 to .10 ). Together, the panels suggest that self-reports corresponded to forward simulation accuracy mainly when explanations were fully correct and participants predicted the AI’s decisions accurately.
Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.
The evaluation of explainable AI (XAI) approaches often relies on three families of methods: subjective measures (e.g., questionnaires on trust or satisfaction), objective measures (e.g., task performance metrics), and mathematical metrics (e.g., for faithfulness). Yet, it remains unclear how these families align or diverge in practice. In a{preregistered} between-subjects study (N=166), we use three established saliency map techniques (LIME, Grad-CAM, Guided Backpropagation) as a testbed to examine this issue. We find that each family of methods leads to different conclusions: participants reported no differences in trust or satisfaction, Grad-CAM improved user performance, while mathematical metrics favored Guided Backpropagation. At the same time, mathematical metrics were only partially related to user performance, and these relationships were sometimes counterintuitive. Our findings highlight the methodological importance of comparing subjective, objective, and mathematical approaches when evaluating XAI, illustrating both tensions and aspects that are aligned. We discuss implications for XAI evaluation frameworks.
Felix Kares, Timo Speith, Hanwei Zhang +1
Saarland University, Germany · University of Bayreuth, Germany · University of Freiburg, Germany
Explainable AI (XAI) techniques aim to provide insights into predictive models and enhance user performance, yet they often fall short of these expectations. Conversational XAI assistants promise to overcome such limitations, but empirical evidence on their impact on objective performance measures remains limited. We propose an experimental design for evaluating explanation assistance through prediction accuracy, model understanding, and error identification. Using an explainable-by-design prediction model, we create conditions where users can outperform the model by identifying and compensating for systematic errors. We compare conversational assistance against Q&A-based assistance to assess which better supports users in working with model explanations. Preliminary results from testing our experimental design show that participants (N=42) in both treatments significantly outperformed the model but reveal no performance differences between assistance types and modest engagement overall. These findings inform refinements for our planned full study, including enhanced engagement interventions and investigation of the mechanisms driving improved predictions.
Sven Kruschel, Julian Rosenberger, Lasse Bohlen +2
TU Dresden, Dresden, Germany · University of Regensburg, Regensburg, Germany