cs.AINov 6, 2026

RAG-Enhanced LLMs for Interactive Explainability in Clinical Decision Support Systems

Authors: Lorenzo Mannocci, Francesca Naretto, Lucia Passaro, Anna Monreale

Organizations: University of Pisa, Italy

Abstract

Artificial Intelligence (AI) is increasingly embedded in high-risks decision-making systems across domains such as healthcare, finance, and the judicial system. As reliance on predictive models grows, so does the demand for Explainable AI (XAI) to ensure transparency, trust, and interpretability. However, existing XAI approaches often fail to present explanations in a format accessible to domain experts, limiting their practical utility. Large Language Models (LLMs) have recently emerged as a bridge between complex models and end-users, offering natural language explanations. Yet, their use introduces new risks, including hallucinated outputs and lack of source traceability. To address these challenges, we propose a Clinical Decision Support System (CDSS) that not only combines standard predictive models with XAI techniques but also incorporates a Retrieval-Augmented Generation module as an interactive enhancer. This module grounds explanations in verified medical knowledge, reformulates technical outputs into a human-like format, and enables users-such as clinicians-to actively query the system. By fostering an interactive, human-in-the-loop environment, our approach empowers domain experts to explore model decisions, contextualize explanations, and build trust in AI-assisted diagnostics. The study concludes with preliminary experiments validating the proposed methodology.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Medical Heuristic Learning: An LLM-Driven Framework for Interpretable and Auditable Clinical Decision Rules

    Jun 15, 2026Wei Xu, Ke Yang, Gang Luo +4Clinical Decision SupportInterpretable ML

  2. Uncertainty Reasoning with Large Language Models for Explainable Disease Diagnosis

    May 25, 2026Xiaoyang Fan, Yufan Cai, Zhe Hou +1Clinical Decision SupportClinical Reasoning

  3. BoolXLLM: LLM-Assisted Explainability for Boolean Models

    May 12, 2026Du Cheng, Serdar Kadioglu, Xin WangExplainable Artificial IntelligenceBoolean Function Learning