Improving understanding and trust in AI: How users benefit from interval-based counterfactual explanations
Authors: Tabea E. Röber, Paul Festor, Rob Goedhart, S. İlker Birbil, Aldo Faisal
Organizations: Amsterdam Business School, University of Amsterdam, The Netherlands · Department of Bioengineering, Imperial College London, UK · Institute of Artificial and Human Intelligence, University of Bayreuth, Germany
Experimental user studies evaluating the effectiveness of different subtypes of post-hoc explanations for black-box models are largely nonexistent. Therefore, the aim of this study was to investigate and evaluate how different types of counterfactual explanations, namely single point explanations and interval-based explanations, affect both model understanding and (demonstrated) trust. We conducted an online user study using a within-subjects experimental design, where the experimental arms were (i) no explanation (control), (ii) feature importance scores, (iii) point counterfactual explanations, and (iv) interval counterfactual explanations. Our results clearly show the superiority of interval explanations over other tested explanation types in increasing both model understanding and demonstrated trust in the AI. We could not support findings of some previous studies showing an effect of point counterfactual explanations compared to the control group. Our results further highlight the role individual differences in, for example, cognitive style or personality, in explanation effectiveness.
Figures & tables
Figure 1: Schematic overview of the study concept and design. After filling out a survey to record demographics and experience level, each participant completes three blocks consisting of multiple trials each. Instructions within a block do not change. Participants are randomly assigned to one of the four explanation groups and one of the three different view modes.
Figure 2: Model-based group means by explanation type from two mixed-effects models with participant-level random intercepts, controlling for gender , education , and age . Estimates are based on 100 bootstrapped samples per model. Error bars reflect bootstrapped standard deviations. Group means are estimated for the reference groups ( female and bachelors ) and the mean age observed in the sample.
Figure 3: Root mean squared error (RMSE) by following behavior vs reliance on AI proportion, for trials where no features are hidden and where three out of 10 features are hidden. Each user is represented by two dots (unless the proportion is 0 or 1). A LOWESS curve is shown to visualize trends in prediction error.
Figure 4: Observed group means averaged per user by explanation type and gender.
Feature
Description
Type
zipcode
zipcode corresponding to the area the house is located in
categorical (4)
lot-len
the length in meters of the plot of land the house is built on
float
lot-width
the width in meters of the plot of land the house is built on
float
lot-area
the total area of the plot of land the house is built on
float
house-area
the living area of the house in square meters
float
garden-size
the size of the garden in square meters
float
Table 1: Overview of features included in the housing dataset utilized in this study
Figure 5: Distribution of participants over conditions
Counterfactual explanations show how changing an input could alter a model’s prediction, supporting model understanding, auditing, and actionable recourse.
Explainability is widely regarded as essential for trustworthy artificial intelligence systems. However, the metrics commonly used to evaluate counterfactual explanations are algorithmic evaluation metrics that are rarely validated against human judgments of explanation quality. This raises the question of whether such metrics meaningfully reflect user perceptions. We address this question through an empirical study that directly compares algorithmic evaluation metrics with human judgments across three datasets. Participants rated counterfactual explanations along multiple dimensions of perceived quality, which we relate to a comprehensive set of standard counterfactual metrics. We analyze both individual relationships and the extent to which combinations of metrics can predict human assessments. Our results show that correlations between algorithmic metrics and human ratings are generally weak and strongly dataset-dependent. Moreover, increasing the number of metrics used in predictive models does not lead to reliable improvements, indicating structural limitations in how current metrics capture criteria relevant for humans. Overall, our findings suggest that widely used counterfactual evaluation metrics fail to reflect key aspects of explanation quality as perceived by users, underscoring the need for more human-centered approaches to evaluating explainable artificial intelligence.
Counterfactual explanations (CEs) are widely used in explainable artificial intelligence (AI) to show how a model's outputs would change if the input features were manipulated. This technique is used for a range of tasks such as debugging models, explaining predictions, justifying decisions, and providing algorithmic recourse. In this paper, we explore the normative legitimacy of employing counterfactuals in real-life model deployment settings. We discuss the different stakes involved in these different purposes for which CEs are commonly employed, and find stricter requirements for justification and recourse. In particular, we find that naive application of CEs for justification and recourse can lead to ignoring contestable choices made throughout the machine learning (ML) pipeline, thus obfuscating that decisions and counterfactuals for those decisions are also artifacts of an organization's materialized design and governance choices. We demonstrate this with four empirical experiments involving interventions at stages of the ML pipeline ``upstream" of the explanation itself, and show that these affect the generated counterfactuals. We find that an organization's choices on measurement models for feature and labels, business requirements, model validation, and the metric of model success have as much or more impact on the generated counterfactuals as the specifics of the generating method. Our findings underline the need to account for such choices upon providing justification and recourse, providing a stark reminder of the relational nature of these tasks. As putative justifications or recourse recommendations, CEs do not provide adequate answers to some important "why"-questions because they preclude consideration of whether the decision-maker ought to have acted differently.
Mattia Cerrato, Otto Sahlgren, Xenia Heilmann
Johannes Gutenberg University Mainz, Germany · University of Tampere, Finland
Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.