Rules and Weights are popular XAI techniques for explaining AI decisions. Yet, it remains unclear how to choose between them, lacking a cognitive framework to compare their interpretability. In an elicitation user study on forward and counterfactual decision tasks, we identified 7 reasoning strategies of interpreting three XAI Schemas - weights, rules, and their hybrid. To analyze their capabilities, we propose CoXAM, a Cognitive XAI-Adaptive Model with shared memory representation to encode instance attributes, linear weights, and decision rules. CoXAM employs computational rationality to choose among reasoning processes based on the trade-off in utility and reasoning time, separately for forward or counterfactual decision tasks. In a validation study, CoXAM demonstrated a stronger alignment with human decision-making compared to baseline machine learning proxy models. The model successfully replicated and explained several key empirical findings, including that counterfactual tasks are inherently harder than forward tasks, decision tree rules are harder to recall and apply than linear weights, and the helpfulness of XAI depends on the application data context, alongside identifying which underlying reasoning strategies were most effective. With CoXAM, we contribute a cognitive basis to accelerate debugging and benchmarking disparate XAI techniques.
Figures & tables
Figure 1. Overall approach to model users in XAI understanding across three studies: I) Formative study to elicit reasoning strategies, II) Modeling study to examine user reasoning, and III) Summative study to observe decisions. This figure illustrates the overall workflow of the research approach. The XAI Schemas inform two empirical components: a Formative Study that identifies participants’ reasoning strategies, and a Modeling Study (CoXAM) that formalizes these strategies to simulate user reasoning. The resulting model is then evaluated in a Summative Study using user decision data from forward and counterfactual tasks.
Figure 2. Weight-based XAI with Tabular UI components for: a) Attribute names of the instance. b) Value of each attribute with bar to indicate how high/low. c) Factors to multiply with each attribute value. d) Partial contributions of each attribute based on value x factor. e) Prediction of Weights explanation for current instance. f) Prediction of AI. a–b are always shown; c is only shown in trials with Weights explanation; d, e, f is not shown for decision testing. This figure shows the user interface for the Weights XAI condition in the tabular instance task. The interface presents six attributes with their corresponding values and visual bars indicating magnitude. For each attribute, the model displays its linear coefficient (Factor) and the resulting contribution (Partial Sum), computed as value × factor. The overall Factors prediction is displayed on the right alongside the AI’s predicted class label. Panels (a)–(b) are always visible, while panels (c)–(f) appear only when Weights explanation is provided.
Figure 3. Rule-based XAI with Decision Tree UI for a) low (wine quality) and b) high complexity (mushrooms). The black outline highlights the path that corresponds to the current instance values. This figure shows the Rules explanation user interface used in the experiment for two example datasets. Panel (a) illustrates a low-complexity tree from the Wine Quality task, while panel (b) depicts a higher-complexity tree from the Mushrooms task. Each node represents an attribute and its decision threshold, with terminal leaves showing the predicted class labels. The black-highlighted path indicates the traversal corresponding to the current instance’s feature values, helping users visualize how the AI reaches its prediction.
Figure 4. User performance of forward simulation task in a), and b) and counterfactual simulation task in c), and d), across datasets: Wine Quality (left), and Mushrooms (right) across XAI Schemas, and testing conditions (w/o XAI, w/ XAI). Error bars are 95% CIs. Random performance for the forward simulation task would be 50%. Panels (a) and (b) show forward simulation accuracy for the Wine Quality and Mushrooms datasets, respectively, while panels (c) and (d) present counterfactual simulation accuracy for the same datasets. Results are reported for three XAI Schemas (Rules, Weights, and Hybrid) under two testing conditions: without explanations (red, w/o XAI) and with explanations (blue, w/ XAI). Error bars represent 95% confidence intervals. Forward simulation accuracy is consistently above the random baseline of 50%, though the relative benefits of explanations differ across datasets and model types. Counterfactual simulation accuracy is lower overall, with modest improvements depending on model type and explanation condition.
Wine Quality
Mushrooms
Rules
Weights
Hybrid
Rules
Weights
Hybrid
Models
NLL
BIC
NLL
BIC
NLL
BIC
NLL
BIC
NLL
BIC
NLL
BIC
Decision Tree
26.8
57.3
—
—
26.2
56.1
21.7
47.1
—
—
29.7
63.1
Linear Regression
—
—
28.0
59.7
26.8
57.3
—
—
37.0
77.7
35.5
74.7
KNN w/o XAI
29.7
66.8
30.1
67.6
29.9
67.2
24.0
55.4
28.1
63.6
27.2
61.8
CoXAM
18.9
48.8
19.9
50.9
20.2
51.5
20.7
52.5
21.5
54.1
20.8
52.7
Table 1. Comparison of Negative Log-Likelihood (NLL) and Bayesian Information Criterion (BIC) for CoXAM against baseline proxies for forward simulation task for the Wine Quality and Mushrooms datasets. Lower values are better. Bold is best fit. This table compares model fits for the forward simulation task using Negative Log-Likelihood (NLL) and Bayesian Information Criterion (BIC) across two datasets (Wine Quality and Mushrooms) and three explanation conditions (Rules, Weights, and Hybrid). Lower values indicate better model fit. Overall, CoXAM achieves the best fit across most conditions, showing lower NLL and BIC values than baseline models (Decision Tree, Linear Regression, and KNN without XAI). An exception occurs in the Mushrooms dataset under the Rules condition, where the Decision Tree baseline attains a slightly lower BIC, suggesting close performance for that specific case.
Figure 5. Results of summative user study (orange) compared to CoXAM (blue) of accuracy in the forward simulation task. Error bars are 95% CI. Forward simulation accuracy for human participants (orange) and the CoXAM model (blue) across explanation types and datasets, evaluated with and without XAI support. Results are reported for the Wine Quality dataset (a–b) and the Mushrooms dataset (c–d), under testing conditions without XAI (a, c) and with XAI (b, d). Explanation types include Rules, Weights, and Hybrid representations. Accuracy is computed with respect to the AI system’s ground-truth predictions, and error bars denote 95% confidence intervals. Overall, CoXAM closely mirrors human performance patterns across explanation types and conditions, capturing both relative performance differences among explanation formats and the impact of XAI availability.
Figure 6. Forward simulation accuracy of two Weights and one Rules strategies for with and without XAI trials across datasets. Forward simulation accuracy across explanation strategies, datasets, and XAI conditions. The figure reports mean forward simulation accuracy (y-axis, %) with error bars indicating variability across participants. Results are shown for two datasets—Wine Quality (a–b) and Mushrooms (c–d)—under conditions without XAI (a, c) and with XAI (b, d). Three explanation strategies are compared: Approximate Calculation, Feature Attribution, and Attribute–Threshold–Attribute (ATA) Traversal. For the Wine Quality dataset, Feature Attribution and ATA Traversal outperform Approximate Calculation in both conditions, with a consistent improvement when XAI is available. For the Mushrooms dataset, ATA Traversal achieves the highest accuracy with XAI, while Feature Attribution shows reduced performance relative to the other strategies. Overall, the results indicate that explanation strategy and the presence of XAI substantially influence users’ ability to accurately simulate model behavior, with rule-based ATA Traversal benefiting most from XAI support.
Figure 7. Strategy prevalence by XAI Schema for forward simulation. This figure shows the prevalence of different reasoning strategies used by participants in the forward simulation task, grouped by XAI Schema and dataset. Each cell represents the proportion of trials where a given strategy was applied. For the Wine Quality dataset, participants predominantly used the Attribute Threshold/Traversal strategy with Rules and the Feature Attribution strategy with Weights XAI. In contrast, for the Mushrooms dataset, the Approximate Calculation strategy was dominant for Weights XAI, while Attribute Threshold/Traversal remained consistent for Rules. Hybrid XAI conditions show mixed usage patterns, indicating flexible strategy selection depending on task complexity.
Wine Quality
Mushrooms
Rules
Weights
Hybrid
Rules
Weights
Hybrid
Model
NLL
BIC
NLL
BIC
NLL
BIC
NLL
BIC
NLL
BIC
NLL
BIC
Random
71.67
143.34
71.67
143.34
71.67
143.34
71.67
143.34
71.67
143.34
71.67
143.34
Global SHAP
42.1
84.2
56.3
112.6
54.7
106.4
50.2
100.4
55.1
110.2
51.0
102.0
CoXAM
35.6
71.2
51.8
103.6
45.0
90.0
45.8
91.6
50.6
101.2
46.6
93.2
Table 2. Comparison of Negative Log-Likelihood (NLL) and Bayesian Information Criterion (BIC) for CoXAM against baseline proxies for participant attribute-selection in the counterfactual task for the Wine Quality and Mushrooms datasets.Lower is better. Bold is best fit. This table compares the Negative Log-Likelihood (NLL) and Bayesian Information Criterion (BIC) of CoXAM against two baseline models—Random and Global SHAP—across the Wine Quality and Mushrooms datasets, for three explanation types (Rules, Weights, and Hybrid). Lower values indicate better model fit. CoXAM consistently achieves the lowest NLL and BIC across all conditions, showing strong agreement with user behavior. While Global SHAP improves upon the Random baseline, it remains less predictive of participant decisions compared to CoXAM.
Figure 8. Results of summative user study (orange) compared to CoXAM (blue) of accuracy in counterfactual simulation task. Error bars are 95% CI. Counterfactual simulation accuracy for human participants (orange) and the CoXAM model (blue) across explanation types and datasets, evaluated with and without XAI support. Results are shown for the Wine Quality dataset (a–b) and the Mushrooms dataset (c–d), under testing conditions without XAI (a, c) and with XAI (b, d). Explanation types include Rules, Weights, and Hybrid representations. Accuracy is computed with respect to the AI system’s prediction change, and error bars denote 95% confidence intervals. Overall, CoXAM closely tracks human performance across datasets, explanation types, and XAI conditions, reproducing relative performance differences and condition-dependent trends observed in human counterfactual reasoning.
Figure 9. Results of CoXAM simulated user accuracy in the counterfactual simulation task, for different strategies. Error bars are 95% CI. Figure 9. Counterfactual simulation accuracy of CoXAM-simulated users across two datasets (Wine Quality and Mushrooms), with and without XAI support. Results are shown for four reasoning strategies: Inverse Calculation, Node Threshold Crossing, Inverse Feature Attribution, and Availability Heuristic. Error bars represent 95% confidence intervals. Accuracy is highest for Inverse Calculation and Node Threshold Crossing, while Availability Heuristic consistently yields lower performance.
Figure 10. Strategy prevalence by XAI Schema for counterfactual simulation. This figure shows how often different reasoning strategies were used in the counterfactual simulation task, grouped by XAI Schema and dataset. The rows list the four strategies: Inverse Calculation, Inverse Feature Attribution, Node Threshold Crossing, and Availability Heuristic. The columns represent the Wine Quality and Mushrooms datasets under Rules, Weights, and Hybrid explanation conditions. Each cell shows the percentage of trials in which that strategy was used. For Wine Quality, participants most often used Node Threshold Crossing with Rules explanations and Inverse Feature Attribution with Weights explanation. For Mushrooms, Availability Heuristic and Inverse Calculation were most common under Weights explanations, while Node Threshold Crossing remained frequent under Rules and Hybrid conditions.
XAI Schema
Representative XAI Methods
CoXAM Coverage
Rule-based
Decision Tree ( Breiman et al., 2017 ) , Rule List ( Angelino et al., 2018b ) , Rule Set ( Lakkaraju et al., 2016a )
Covered: Topology-agnostic by converting to equivalent DNF. Future work: Account for differences in parsing topology.
Weight-based (Linear Additive)
Linear Regression ( Bo et al., 2024 ) , Logistic Regression ( Poursabzi-Sangdeh et al., 2021 )
Covered.
Additive (Non-Linear)
Partial Dependence Plot ( Krause et al., 2016 ) , Generalized Additive Model (GAM) ( Hastie and Tibshirani, 1990 ) , Piecewise Linear ( Friedman, 1991 ; Bo et al., 2024 )
Future work: encode interpretation of nonlinear trends as local linear based on prototype examples ( Kalish et al., 2004 ) .
Symbolic Regression (Non-Linear)
Domain-specific equations ( Alaa and Van der Schaar, 2019 )
Future work: encode interpretation with rational model of function learning ( Lucas et al., 2015 ) .
Example-based
Prototypes ( Narayanan and Bergen, 2024 ) , Criticisms ( Kim et al., 2016 )
Future work: encode examples in memory, retrievable based on computational rationality, to infer labels based on similarity ( Nosofsky, 1986 ) .
Table 3. Scope of CoXAM for user interpretation of global XAI Schemas on interpretable features.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Algorithm
PPO
Policy
MlpPolicy
Policy activation
Tanh
Policy network (actor)
[64, 64]
Value network (critic)
[64, 64]
Learning rate
3×10−4
Appendix
Table 4. Hyperparameters used for counterfactual RL training (PPO).
Figure A1. Forward and Counterfactual Simulation Accuracy, measured with respect to AI outputs across all conditions, and dataset application scenarios. Shows four line plots comparing forward and counterfactual simulation accuracy for the Wine Quality and Mushrooms datasets. The top row shows forward accuracy and the bottom row shows counterfactual accuracy. The left column corresponds to Wine Quality and the right column to Mushrooms. Results are shown for three explanation types (Rules, Weights, and Hybrid), each under low and high conditions. Red circle markers indicate performance without XAI and blue plus markers indicate performance with XAI. Error bars show variability. Overall, forward accuracy is higher than counterfactual accuracy, and the effect of XAI varies across explanation types, conditions, and datasets.
Figure A2. Normalized absolute change in attribute values across different XAI Schemas, datasets and with or without XAI trials. Figure A2 shows two line plots of normalized absolute change in attribute values for the Wine Quality dataset (left) and the Mushrooms dataset (right). The x-axis lists explanation types (Rules, Weights, Hybrid) and the y-axis shows normalized absolute change in percent. Red circle markers indicate trials without XAI and blue plus markers indicate trials with XAI, with error bars showing variability. Across both datasets, the Weights explanation shows the largest changes, followed by Hybrid, with Rules showing the smallest changes.
Dataset
Tested w/ XAI
Strategy
Accuracy (%)
Time (s)
Mushrooms
w/ XAI
Attribute-Threshold-Attribute Traversal
74.6 ± 1.1
9.2 ± 0.1
Approximate Calculation
64.6 ± 1.4
11.0 ± 0.2
Feature Attribution
49.4 ± 5.3
11.2 ± 0.6
w/o XAI
Attribute-Threshold-Attribute Traversal
63.2 ± 1.2
8.9 ± 0.1
Approximate Calculation
60.9 ± 1.5
10.7 ± 0.2
Feature Attribution
56.1 ± 4.8
12.3 ± 0.6
Appendix
Table 5. Forward simulation performance (mean ± SE) across reasoning strategies, XAI conditions, and datasets. Accuracy is reported in %, and time in seconds.
Dataset
Tested w/ XAI
Strategy
Accuracy vs. AI (%)
Accuracy vs. XAI (%)
Mushrooms
w/ XAI
Availability Heuristic
32.7 ± 2.5
43.0 ± 2.6
Inverse Calculation
48.9 ± 3.0
69.5 ± 2.7
Inverse Feature Attribution
44.9 ± 2.1
71.4 ± 1.9
Node Threshold Crossing
49.0 ± 1.3
69.4 ± 1.2
w/o XAI
Availability Heuristic
30.7 ± 1.2
43.4 ± 1.2
Inverse Feature Attribution
45.8 ± 1.7
70.4 ± 1.6
Appendix
Table 6. Counterfactual accuracy (mean ± SE, %) across reasoning strategies, whether XAI is shown, and datasets. Accuracy is reported against both the AI and XAI.
Figure A3. Variations in CoXAM performance against cognitive parameters. a) Strategy accuracy in the forward simulation task against cognitive parameters and specific trial strategies. b) Attribute selection, X of the six attributes in Wine Quality dataset against opportunity cost, γ . This figure illustrates how forward simulation performance varies with cognitive parameters and task strategies. Panel (a) plots human and CoXAM forward accuracy as a function of three fitted cognitive parameters—opportunity cost (γ), diffusion noise (ν), and retrieval threshold (κ)—for each explanation type (Rules, Weights, Hybrid). Shaded regions indicate 95% confidence intervals. Accuracy declines as cognitive costs increase, particularly under higher γ and ν, showing greater sensitivity for strategies requiring more computation. Panel (b) shows the distribution of attribute selections (\mathcal{X}) across the six Wine Quality attributes as a function of opportunity cost. Participants increasingly focused on fewer, more diagnostic attributes (e.g., Alcohol, Vinegar Taint) as γ rose, reflecting selective attention under higher cognitive effort.
Figure A4. Variation in strategy accuracy in counterfactual task with varying cognitive parameters. Error areas are 95% CI. This figure shows how counterfactual accuracy changes across reasoning strategies as cognitive parameters vary. Each row corresponds to one XAI Schema: Rules, Weights, and Hybrid. The three columns represent variations in opportunity cost gamma, margin epsilon, and retrieval threshold kappa. The four curves represent the strategies Inverse Calculation, Inverse Feature Attribution, Node Threshold Crossing, and Availability Heuristic. Shaded areas show 95 percent confidence intervals. Accuracy trends show that Inverse Feature Attribution and Node Threshold Crossing maintain stable performance under higher cognitive cost, while the Availability Heuristic declines more sharply with increasing gamma or kappa.
Figure A5. Attribute Selection by dataset and XAI Schemas (CoXAM Counterfactual Simulation Results).
Figure A6. Introduction to User Study. A study start page interface showing a title introducing a research study on AI understanding, a short description of the study purpose, a list of study steps, an estimated completion time, and a button to proceed.
Figure A7. System UI of User Study and Tutorial. An application scenario screen explaining that an AI predicts one of two wine types based on chemical attributes. A table lists attributes such as alcohol, sulphates, sulfur dioxide, vinegar taint, pH, and chlorides, along with their descriptions and minimum and maximum values.
Figure A8. UI of Data Instance (Attribute Name, Value and Visual Slider). A tutorial screen showing a single data instance. Each row lists an attribute name, its numeric value, and a horizontal bar indicating the relative magnitude of that value. An AI prediction label is shown to the right.
Figure A9. Screening Question (Data and Prediction Task). A multiple-choice screening question asking participants to identify correct statements about how the data values, bars, and AI prediction should be interpreted.
Figure A10. Tutorial for Weight-based XAI (top) and screening question (bottom). A tutorial screen explaining a weight-based AI explanation. Attributes and values are shown alongside fixed numeric factors, partial sums for each attribute, and a total contribution. The summed value leads to an explained prediction, which may or may not match the AI’s prediction. Also, below, a multiple-choice screening question asking participants to identify correct statements about how the weight-based explanation works, including whether factors are constant and how the final prediction is determined.
Figure A11. Tutorial for Rule-based XAI (top) and screening question (bottom). A tutorial screen explaining a rule-based AI explanation using a decision tree. The tree checks attributes one at a time against thresholds, follows a single highlighted path, and ends at a predicted class. Also, below, a multiple-choice screening question asking participants to identify correct statements about how the rule-based explanation should be read and interpreted.
Figure A12. Start page of 40-trial forward simulation session. A session introduction page explaining that participants will complete multiple trials where they learn how the AI predicts one of two classes. Instructions emphasize learning from feedback and answering thoughtfully.
Figure A13. User predicts AI output without XAI. A task screen without explanation where participants view attribute values with bars and must estimate the AI’s prediction by selecting one of two options.
Figure A14. Feedback on AI output without XAI. A feedback screen showing the correct AI prediction for the previous case, alongside the same attribute values and bars.
Figure A15. User predict AI output with the Weight-based XAI. A task screen with a weight-based explanation where participants view attribute values, factors, and partial sums, and then estimate the AI’s prediction.
Figure A16. Feedback on AI output with the Weight-based XAI. A feedback screen for the weight-based explanation showing the computed total contribution, the explained prediction, and the AI’s actual prediction.
Figure A17. User predicts AI output with the Rule-based XAI. A task screen with a rule-based explanation where participants see attribute values, a decision tree, and must estimate the AI’s prediction.
Figure A18. Feedback on AI output with the Rule-based XAI. A feedback screen for the rule-based explanation showing the AI’s prediction and the path taken through the decision tree.
Figure A19. Start page of 40-trial counterfactual simulation session. A session introduction page explaining a counterfactual task where participants must change exactly one attribute by as little as possible to flip the AI’s prediction, with no feedback provided on success.
Figure A20. Change an attribute value without XAI. A counterfactual task screen without explanation showing current attribute values, the AI’s prediction, and sliders allowing one attribute to be adjusted to change the prediction.
Figure A21. Change an attribute value with the Weight-based XAI (No AI feedback). A counterfactual task screen with a weight-based explanation showing attribute values, factors, partial sums, and sliders for adjusting a single attribute, without feedback on whether the prediction changed.
Figure A22. Change an attribute value with the Rule-based XAI (No AI feedback). A counterfactual task screen with a rule-based explanation showing attribute values, a decision tree, and sliders for adjusting a single attribute, without feedback on whether the prediction changed.
Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.
Explainable AI (XAI) methods are usually evaluated for technical properties such as faithfulness and robustness. In high-stakes application domains, experts also need to check model predictions against established knowledge. Different explanation methods often disagree, and this disagreement is hard to see when each method is inspected separately. We propose XALIGN, a visual analytics workflow and interface that shows rule-based explanations and several feature attribution methods (TreeSHAP, KernelSHAP, and LIME) side by side for the same Random Forest model. Agreements and contradictions between the methods become visible signals that users can explore and compare with domain knowledge. We apply XALIGN in two case studies: the German Credit dataset and a meteorological dataset, in which events derived from ICON model output and Cloudnet observations are classified. In both, the views reveal discrepancies between the rule-based explanation and the attribution methods that a single method would not show. In an expert evaluation with three specialists, participants found XALIGN useful for identifying influential features and for avoiding over-trust in predictions. Linking rules to attributions was harder for the participant without an XAI background.
The evaluation of explainable AI (XAI) approaches often relies on three families of methods: subjective measures (e.g., questionnaires on trust or satisfaction), objective measures (e.g., task performance metrics), and mathematical metrics (e.g., for faithfulness). Yet, it remains unclear how these families align or diverge in practice. In a{preregistered} between-subjects study (N=166), we use three established saliency map techniques (LIME, Grad-CAM, Guided Backpropagation) as a testbed to examine this issue. We find that each family of methods leads to different conclusions: participants reported no differences in trust or satisfaction, Grad-CAM improved user performance, while mathematical metrics favored Guided Backpropagation. At the same time, mathematical metrics were only partially related to user performance, and these relationships were sometimes counterintuitive. Our findings highlight the methodological importance of comparing subjective, objective, and mathematical approaches when evaluating XAI, illustrating both tensions and aspects that are aligned. We discuss implications for XAI evaluation frameworks.
Felix Kares, Timo Speith, Hanwei Zhang +1
Saarland University, Germany · University of Bayreuth, Germany · University of Freiburg, Germany