Organizations: CISPA Helmholtz Center for Information Security, Saarbrücken, Germany · Saarland University, Saarbrücken, Germany · Mizuho–DL Financial Technology, Co., Ltd., Tokyo, Japan · University of Michigan, Ann Arbor, MI, USA
We study the design of explanations in algorithmic decision-making with strategic agents, individuals who may modify their inputs in response to explanations of a decision maker's (DM's) predictive model. While the demand for algorithmic transparency has led much prior work to assume full model disclosure, in practice DMs typically provide only partial information via explanations, which can cause agents to misinterpret the model and take actions that unintentionally reduce their own utility. A central open question is therefore how DMs should communicate explanations that avoid harming strategic agents while still supporting their own goals, e.g., minimising predictive error. In this work, we analyse widely used explanation methods and establish a necessary condition to prevent explanations from inducing self-harming responses. Furthermore, we show that action recommendation-based explanations (ARexes), which encompass counterfactual explanations, are sufficient to induce all non-harmful responses. Under a conditional homogeneity assumption, this sufficiency extends to ARex-generating methods, echoing the revelation principle in information design. To demonstrate their practical utility, we introduce a simple learning procedure that jointly optimises the predictive model and the explanation-generating policy. Experiments on both synthetic and real-world tasks show that ARexes enable DMs to achieve high predictive performance while preserving agents' utility, offering a principled strategy for safe and effective partial model disclosure.
Figures & tables
Figure 1: (left) With full access to the DM’s model, the agent with a covariate x can correctly anticipate how changes affect predictions and choose a response x∙ that reliably improves utility u(x∙) . (right) With only an explanation, the agent’s response x⋄ , based on partial information, might not improve utility u(x⋄) .
Figure 2: ( 2(a) ) Taylor-ex and LIME mislead agents into reducing their utility while ARexes do not. The box plot shows the change in agents’ utility after best responding. ( 2(b) ) & ( 2(c) ): Joint-Opt has the lowest training-loss curves (nMSE and BCE) against the three baselines, showing that jointly optimising both g and σ is more beneficial to the DM.
Joint-Opt
λ=0.1
λ=1.0
λ=4.0
Synthetic data
Strategic nMSE ( ↓ )
2e-4
7e-3
8e-3
1e-2
Compliance rate
1.0
1.0
1.0
1.0
German credit data
Strategic F1 score ( ↑ )
0.9
0.86
0.84
0.85
Compliance rate
0.84
1.0
1.0
1.0
Table 1: Joint-Opt achieves the lowest test error on synthetic data and highest test score on real-world data, while maintaining strong compliance rates, which indicates the portions of agents that follow the DM’s recommended actions.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
g0
=arggminxt,yt∈D1∑(g(xt)−yt)2,
σ0r
=argσrminxt,yt∈D1∑(σr(xt)−xt)2.
Appendix
Algorithm 1 Joint optimisation of g and σ .
Figure 3: The plots show the training errors of our Joint-Opt approach and the baselines at different magnification levels. The training error of Joint-Opt increases more sharply after each interaction between the DM and agents. This implies an unstable optimisation process and the absence of fixed-point convergence ( Perdomo et al., 2020 ) , especially in high-dimensional settings where the number of trainable parameters increases significantly due to joint-training of {g,σr} .
Algorithmic explanations are intended to help stakeholders understand opaque algorithmic decisions, but in practice, they often fall short. First, the meaning of algorithmic explanations is often not what one might intuitively expect, so expert knowledge is required to interpret them correctly. Second, recent work has shown that popular explanation algorithms are uninformative about the behavior of complex decision functions. Together, these issues create a gap between what explanations appear to convey and what they actually provide. In this work, we propose Explanation Cards for Explanation Algorithms, which augment standard explanations with complementary information about robustness and validity, as well as clear instructions for interpretation. The complementary information can render otherwise uninformative explanations practically useful, while also helping to detect cases where they are not. Importantly, the interpretation instructions in explanation cards shift responsibility from users to providers: Rather than expecting users to recognize what can and cannot be concluded from an explanation, providers must make this explicit upfront. Using counterfactual explanations and SHAP as examples, we demonstrate how providers can construct explanation cards and that these cards provide users with the guidance needed for sound interpretation. We further argue that explanation cards offer a practical means of operationalising the explainability provisions of the EU AI Act. Overall, explanation cards are a significant step toward making explanation algorithms fit for real-world use cases.
Eric Günther, Balázs Szabados, Kristof Meding +3
University of Tübingen · Tübingen AI Center · HUN-REN Institute for Computer Science and Control (SZTAKI), Budapest, Hungary
Algorithmic recourse methods typically assume that a predictive model has access to all features of an individual. In practice, decisions are often made with partial information, because features are costly to acquire. Active feature acquisition addresses cost-constrained prediction, but existing methods are explanation-agnostic: prior work provides explanations only after acquiring additional features, rather than using explanations to drive acquisition. This work flips that and treats algorithmic recourse and feature acquisition jointly. We use Markov Blanket theory to unify counterfactual, semifactual, and alterfactual explanations and to characterize how available recourse grows as features are acquired. Building on this framework, we propose an Explanation-Driven Feature Acquisition (EDFA) method that selects features by explanatory value per unit cost. The framework is further extended with distribution-free validity guarantees for recourse issued from partial information, which signal trustworthy, lower-cost recourse, along with a lower bound on the calibration data required to certify them. Experiments on 7 publicly available datasets with neural network-based predictive models show that EDFA acquires substantially fewer features than state-of-the-art AFA baselines while maintaining comparable accuracy and yielding more decision-relevant, actionable recourse. The implementation is available on GitHub.
Vinura Galwaduge, Jagath Samarabandu
Dept. of Electrical and Computer Engineering Western University London, ON, Canada
Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.