A User-Centric Analysis of Explainability in AI-Based Medical Image Diagnosis

Julia Wagner; Tim Schlippe

cs.HCMar 31, 2026

A User-Centric Analysis of Explainability in AI-Based Medical Image Diagnosis

Authors: Julia Wagner, Tim Schlippe

Organizations: IU International University of Applied Sciences, Germany

Abstract

In recent years, AI systems in the medical domain have advanced significantly. However, despite outperforming humans, they are rarely used in practice since it is often not clear how they make their decisions. Optimal explanation and visualization of the decision process are often lacking. Therefore, we conducted a comparative user-centric analysis of the latest state-of-the-art textual, visual and multimodal explainable artificial intelligence (XAI) methods for medical image diagnosis. Our survey of 33 physicians showed that 88% agree that it is important that AI explains the diagnosis -- 64% even strongly agree. A combination of bounding box and report is rated better than the other tested XAI methods in the evaluated aspects understandability, completeness, speed, and applicability. We even tested the potential negative impact of false AI-based medical image diagnoses and found that 50% of the participants trusted false AI diagnoses over all tested XAI methods.

Figures & tables

Panel of Figure 1 — Figure 1: Visual and Textual XAI Methods.

Panel of Figure 2 — Figure 2: Multimodal XAI Methods.

Panel of Figure 3 — Figure 3: Visual XAI Methods.

Panel of Figure 4 — Figure 4: Textual XAI Methods.

Panel of Figure 5 — Figure 5: Multimodal XAI Methods.

Panel of Figure 6 — Figure 6: Correct, Partly Correct and False Diagnoses.

Explore similar work

SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis
Sep 4, 2026Yuqing Yang, Alexander Schmatz, Zhaozhao Ma +3Multimodal Model Interpretability Medical Diagnosis
Independent Researcher · Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands · Quantitative Data Analytics Group, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands +3
Med-CAM: Minimal Evidence for Explaining Medical Decision Making
Apr 15, 2026Pirzada Suhail, Aditya Anand, Amit SethiExplainable Image Classification Explainable Medical Image Analysis
IIT Bombay
Architecture-aware Robustness Evaluation of Explainable Deep Learning for Breast Cancer Diagnosis
Sep 18, 2026Balenthira Thanusanth, Selvarajah Thuseethan, Roshan G. Ragel +3Explainable Medical Image Analysis Explainability Evaluation
University of Peradeniya, Sri Lanka · Charles Darwin University, Australia · Aneurin Bevan University Health Board, United Kingdom

Sep 4, 2026J/K move · Enter open · S save

SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis

Yuqing Yang, Alexander Schmatz, Zhaozhao Ma, Changkyu Choi+2

Independent Researcher · Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands · Quantitative Data Analytics Group, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands+3

Multimodal Model Interpretability Medical Diagnosis Multimodal Learning Multimodal Classification

Open paper AskPDF arXiv

Explainability is increasingly seen as a crucial requirement in AI-based medical diagnosis, particularly in safety-critical clinical decision-making. Most existing explainability methods in healthcare operate in a post-hoc manner and are predominantly designed for unimodal data, which limits their applicability in increasingly prevalent multimodal diagnostic settings. This paper addresses the problem of self-explainable multimodal diagnosis by formulating it within the information bottleneck (IB) framework. We propose a unified learning paradigm that jointly optimizes predictive performance and modality-specific explainability by identifying the most informative elements inside each modality that contribute to diagnostic decisions. To enable tractable and stable optimization, we employ a matrix-based Renyi's

α

-order entropy functional under the assumption of sufficiently expressive encoders. Extensive experiments on representative medical datasets spanning heterogeneous modalities demonstrate that the proposed method consistently achieves strong diagnostic performance, including an absolute accuracy improvement of 9.1 percentage points on the iCTCF dataset. Moreover, the learned explanations provide transparent and modality-aware insights into feature relevance, thereby improving both the explainability and generalization.

Abstract

Explore similar work

SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosis

Med-CAM: Minimal Evidence for Explaining Medical Decision Making

Architecture-aware Robustness Evaluation of Explainable Deep Learning for Breast Cancer Diagnosis