RAG-Enhanced LLMs for Interactive Explainability in Clinical Decision Support Systems
Organizations: University of Pisa, Italy
Abstract
Artificial Intelligence (AI) is increasingly embedded in high-risks decision-making systems across domains such as healthcare, finance, and the judicial system. As reliance on predictive models grows, so does the demand for Explainable AI (XAI) to ensure transparency, trust, and interpretability. However, existing XAI approaches often fail to present explanations in a format accessible to domain experts, limiting their practical utility. Large Language Models (LLMs) have recently emerged as a bridge between complex models and end-users, offering natural language explanations. Yet, their use introduces new risks, including hallucinated outputs and lack of source traceability. To address these challenges, we propose a Clinical Decision Support System (CDSS) that not only combines standard predictive models with XAI techniques but also incorporates a Retrieval-Augmented Generation module as an interactive enhancer. This module grounds explanations in verified medical knowledge, reformulates technical outputs into a human-like format, and enables users-such as clinicians-to actively query the system. By fostering an interactive, human-in-the-loop environment, our approach empowers domain experts to explore model decisions, contextualize explanations, and build trust in AI-assisted diagnostics. The study concludes with preliminary experiments validating the proposed methodology.
Figures & tables
| model | diabetes | precision | recall | f1-score | accuracy | agreement |
|---|---|---|---|---|---|---|
| Random Forest | no | 0.82 | 0.79 | 0.80 | 0.75 | 0.82 |
| yes | 0.64 | 0.69 | 0.66 | |||
| MedRag | no | 0.77 | 0.80 | 0.78 | 0.71 | |
| yes | 0.60 | 0.56 | 0.58 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Clinical Decision Support
Clinical Decision Support uses data, medical knowledge, and AI to help clinicians assess patients and choose tests, diagnoses, and treatments.
See all 254 papers