Artificial intelligence (AI) systems increasingly support decision-making across critical domains, yet explainable AI (XAI) approaches prioritise algorithmic transparency over human comprehension. While XAI reveals computational processes for model validation, end users require explanations integrating domain knowledge, contextual reasoning, and professional frameworks. This disconnect reveals a design challenge: existing approaches fail to address how practitioners need to understand and act upon AI recommendations. This paper introduces Explanatory AI as an emerging complementary paradigm where AI systems leverage generative and multimodal capabilities to serve as explanatory partners for human understanding. Unlike traditional XAI, which answers 'How did the algorithm decide?' for validation purposes, Explanatory AI addresses 'Why does this make sense?' for practitioners making informed decisions. Through theory-informed design, we synthesise multidisciplinary perspectives from cognitive science, communication research, and education with empirical evidence from healthcare and AI expert interviews. Our analysis identifies five dimensions distinguishing Explanatory AI from XAI: explanatory purpose, communication mode, epistemic stance, adaptivity, and cognitive design. We derive five meta-requirements and formulate ten design principles prescribing how to build such systems, illustrated through five UI mockups exemplifying implementation. Explanatory AI operates standalone or combined with XAI, serving distinct purposes in creating systems that genuinely support human reasoning and decision-making.
Explainable AI (XAI) aims to improve user understanding and decisions when using AI models. However, despite innovations in XAI, recent user evaluations reveal that this goal remains elusive. Understanding human cognition can help explain why users struggle to effectively use AI explanations. Focusing on reasoning on structured (tabular) data, we examined various reasoning strategies for different XAI methods (none, feature importance, feature attribution) in the decision task of anticipating AI decisions (i.e., forward simulation). We i) elicited reasoning strategies from a formative user study, and ii) collected decisions from a summative user study. Using cognitive modeling, we implemented the processes underlying each reasoning strategy and evaluated their alignment with human decision-making. We found that our models better fit human decisions than baseline machine learning proxies, providing insights into which reasoning strategies are (in)effective. We then demonstrate how the fitted model can be used to form hypotheses and investigate research questions that are costly to study with real human participants. This work contributes to debugging human understanding of XAI, informing the future development of more usable and interpretable AI explanations.
Explainable Artificial Intelligence (XAI) has become popular in the last few years. The Artificial Intelligence (AI) community in general, and the Machine Learning (ML) community in particular, is coming to the realisation that in many applications, for AI to be trusted, it must not only demonstrate good performance in its decisionmaking, but it also must explain these decisions and convince us that it is making the decisions for the right reasons. However, different applications have different requirements on the information required of the underlying AI system in order to convince us that it is worthy of our trust. How do we define these requirements? In this paper, we present three dimensions for categorising the explanatory requirements of different applications. These are Source, Depth and Scope. We focus on the problem of matching up the explanatory requirements of different applications with the capabilities of underlying ML techniques to provide them. We deliberately avoid including aspects of explanation that are already well-covered by the existing literature and we focus our discussion on ML although the principles apply to AI more broadly.
Raymond Sheh, Isaac Monteath
(formerly) Department of Computing, Curtin University, Bentley WA 6102, Australia
Explainable AI (XAI) research has produced a plethora of explanation techniques, yet user studies repeatedly show that available explanations are not effective in practice. We argue that, given the siloed nature of conventional XAI, users are struggling to select the appropriate XAI technique. Viewing XAI through a philosophical lens, we offer a formalization of what we call the selection problem: the systematic failure of XAI interfaces to bridge the gap between a user's natural-language uncertainty and the explanation technique that resolves it. Following a logical premise-conclusion format, we show that conventional interfaces require users to translate their uncertainty into a technique selection, a challenging prerequisite to meet. We also propose a structural solution: a multi-agent LLM orchestration tool that translates the user's query to the proper XAI explanation technique. We provide an example of how this structural solution could be instantiated to address the selection problem.