Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation
Authors: Marco Salmè, Federico Siciliano, Fabrizio Silvestri, Paolo Soda, Rosa Sicilia, Valerio Guarrasi
Organizations: Department of Engineering, Research Unit of Artificial Intelligence and Computer Systems, Università Campus Bio-Medico of Roma, Rome, Italy. · Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy. · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden. · UniCamillus-Saint Camillus International University of Health Sciences, Rome, Italy.
Radiology Report Generation (RRG) through Vision-Language Models (VLMs) promises to reduce documentation burden, improve reporting consistency, and accelerate clinical workflows. However, their clinical adoption remains limited by the lack of interpretability and the tendency to hallucinate findings misaligned with imaging evidence. Existing research typically treats interpretability and accuracy as separate objectives, with concept-based explainability techniques focusing primarily on transparency, while Retrieval-Augmented Generation (RAG) methods targeting factual grounding through external retrieval. We present Concept-Enhanced Multimodal RAG (CEMRAG), a unified framework that decomposes visual representations into interpretable clinical concepts and integrates them with multimodal RAG. This approach exploits enriched contextual prompts for RRG, improving both interpretability and factual accuracy. Experiments on MIMIC-CXR and IU X-Ray across multiple VLM architectures, training regimes, and retrieval configurations demonstrate consistent improvements over both conventional RAG and concept-only baselines on clinical accuracy metrics and standard NLP measures. These results challenge the assumed trade-off between interpretability and performance, showing that transparent visual concepts can enhance rather than compromise diagnostic accuracy in medical VLMs. Our modular design decomposes interpretability into visual transparency and structured language model conditioning, providing a principled pathway toward clinically trustworthy AI-assisted radiology.
Figures & tables
Figure 1: CEMRAG overall framework, combining interpretable concept extraction with RAG for transparent and accurate radiology reporting.
Table 1: Experimental configurations for radiology report generation, comparing input modalities, prompt structures, interpretability mechanisms, and factual grounding strategies. Notation: v denotes visual embeddings, Ω extracted concepts and R retrieved reports. Instruction : “Provide a description of the findings in the radiology image”; Task : “Write the report of the radiology image taking information from similar FINDINGS. Consider as more relevant sentences that contain any of the KEYWORDS in the FINDINGS”; Final Instruction : “Write a paragraph with only the report relying in detail on the FINDINGS”.
Model
Method
Micro-F1 14
Micro-F1 5
Macro-F1 14
Macro-F1 5
F1-RG
B-1
B-4
R-L
Zero-Shot
LLaVA-Med
Image-Only
0.255
0.237
0.113
0.134
0.052
18.81
0.71
0.131
+ Concepts
0.401
0.412
0.238
0.355
0.073
21.47
1.29
0.146
+ RAG
0.498
0.528
0.314
0.447
0.184
27.08
4.60
0.187
+ CEMRAG
0.502
0.529
0.319
0.449
0.185
29.34
4.64
0.189
LLaVA with CXR-CLIP
Image-Only
0.213
0.174
0.101
0.123
0.151
19.20
4.69
0.185
Table 2: Quantitative results on the MIMIC-CXR test set for two model configurations (LLaVA-Med and LLaVA with CXR-CLIP) and two training regimes ( Zero-Shot and Supervised Fine-Tuning ). We report F1-RadGraph (F1-RG), and NLP metrics (BLEU-1 as B-1, BLEU-4 as B-4, ROUGE-L as R-L). CheXbert-based label metrics are reported over 14 labels (Micro-F1 14 , Macro-F1 14 ) and over the 5 most prevalent findings (Micro-F1 5 , Macro-F1 5 ). Retrieval is performed from the MIMIC-CXR training set.
Model
Method
Micro-F1 14
Micro-F1 5
Macro-F1 14
Macro-F1 5
F1-RG
B-1
B-4
R-L
Zero-shot
LLaVA-Med
Image-Only
0.063
0.042
0.047
0.019
0.074
17.36
1.08
0.125
+ Concepts
0.122
0.162
0.092
0.134
0.064
13.79
0.85
0.111
+ RAG
0.377
0.344
0.220
0.271
0.228
21.04
3.60
0.177
+ CEMRAG
0.387
0.397
0.252
0.315
0.234
24.34
4.10
0.191
LLaVA with CXR-CLIP
Image-Only
0.305
0.082
0.038
0.043
0.188
5.56
1.26
0.168
Table 3: Quantitative results on the IU X-Ray test set for two model configurations (LLaVA-Med and LLaVA with CXR-CLIP) and two training regimes ( Zero-Shot and Supervised Fine-Tuning ). We report F1-RadGraph (F1-RG), and NLP metrics (BLEU-1 as B-1, BLEU-4 as B-4, ROUGE-L as R-L). CheXbert-based label metrics are reported over 14 labels (Micro-F1 14 , Macro-F1 14 ) and over the 5 most prevalent findings (Micro-F1 5 , Macro-F1 5 ). Retrieval is performed cross-domain from the MIMIC-CXR training set.
Zero-Shot Setting
Chest X-Ray Image
Ground Truth Report
Endotracheal tube terminates 2 cm above the carina. Orogastric tube terminates in the stomach. Right internal jugular catheter terminates in the mid SVC. Lungs are low in volume with stable right upper lung opacities which are better assessed on the recent chest CT but suspicious for pneumonia. There is no pneumothorax or pleural effusion. Heart is normal in size. Normal cardiomediastinal silhouette.
The chest X-ray image shows bilateral diffuse infiltrates, which are abnormal substances or collections of cells that have entered the lung tissue . These infiltrates can be seen in both lungs and are distributed throughout the lung fields.
+ Concepts
The chest X-ray shows overt pulmonary edema , which is a condition where there is an abnormal accumulation of fluid in the lungs. Additionally, there is a right upper lobe opacity , which could be due to various reasons such as infection, inflammation, or other lung abnormalities. The internal jugular vein is visible, and an endotracheal tube is in place , which is a tube that is inserted through the mouth or nose and into the trachea to help maintain an open airway and assist with breathing . Lastly, the pulmonary vascularity appears to be normal.
+ RAG
The chest X-ray findings are similar to those in the earlier study . The main differences observed are the placement of a left IJ catheter and the presence of an indistinct area of opacification in the right upper zone medially , which could represent a developing consolidation. Additionally, there is a right IJ central venous catheter with its tip at the lower SVC near the cavoatrial junction, and the endotracheal tube has been slightly advanced. No pneumothorax is noted. Pulmonary vascular congestion is observed .
Table 4: Qualitative Comparison in Zero-Shot Setting for LLaVA-Med. Highlighting indicates: incorrect/hallucinated findings , incomplete descriptions , superfluous details , accurate content .
Supervised Fine-Tuned Setting
Chest X-Ray Image
Ground Truth Report
Frontal and lateral views of the chest were obtained. There are low lung volumes which accentuate the bronchovascular markings. Bibasilar opacities are seen, which most likely represent atelectasis, although aspiration or infection are not excluded in the appropriate clinical setting. No pleural effusion or pneumothorax is seen. The cardiac and mediastinal silhouettes are stable and unremarkable.
The heart size is normal. The hilar and mediastinal contours are normal. There is a new left lower lobe opacity, concerning for pneumonia . No pleural effusion or pneumothorax. The visualized osseous structures are unremarkable.
+ Concepts
In comparison with the study of ___, there are lower lung volumes . Cardiac silhouette is within normal limits. Opacification at the left base is consistent with atelectatic changes and possible small effusion . In the appropriate clinical setting, superimposed pneumonia would have to be considered .
+ RAG
The lungs are clear without focal consolidation . No pleural effusion or pneumothorax is seen . The cardiac and mediastinal silhouettes are unremarkable .
Table 5: Qualitative Comparison in Supervised Fine-Tuning Setting or LLaVA-Med. Highlighting indicates: incorrect/hallucinated findings , incomplete descriptions , superfluous details , accurate content .
Ground Truth
Generated Report
Endotracheal tube terminates 2 cm above the carina. Orogastric tube terminates in the stomach. Right internal jugular catheter terminates in the mid SVC. Lungs are low in volume with stable right upper lung opacities which are better assessed on the recent chest CT but suspicious for pneumonia. There is no pneumothorax or pleural effusion. Heart is normal in size. Normal cardiomediastinal silhouette.
The chest X-ray shows several findings. In the right upper zone, there is an indistinct area of opacification, which could represent a developing consolidation. Additionally, there is a right internal jugular central venous catheter with its tip at the lower superior vena cava near the cavoatrial junction. The endotracheal tube has been slightly advanced, with its tip 2.2 cm above the carina. No pneumothorax is noted, and the pulmonary vascular congestion is present. The cardiomediastinal silhouette remains unchanged.
Original Image Grad-ECLIP Activation Maps Endotracheal Tube Right Upper
Ground Truth
Generated Report
The patient is status post median sternotomy and aortic valve replacement. Heart size is moderately enlarged but unchanged. The aorta is tortuous and calcified. There is mild interstitial pulmonary edema, relatively unchanged. At least small bilateral pleural effusions are present. Bibasilar airspace opacities may reflect compressive atelectasis. There is no pneumothorax. There is evidence of prior vertebroplasty at the thoracolumbar junction.
The patient is status post median sternotomy and CABG. The heart size remains mildly enlarged. The aorta is tortuous and diffusely calcified. There is mild pulmonary edema, worse in the interval, with small bilateral pleural effusions. Bibasilar airspace opacities likely reflect atelectasis. No pneumothorax is identified. There are no acute osseous abnormalities.
Original Image Grad-ECLIP Activation Maps Median Sternotomy Bilateral Pleural
Table 6: Qualitative Examples with Concept-Specific Visual Explanations. Grad-ECLIP heat maps highlight image regions that influence the LLaVA-Med encoder’s matching between input images and SpLiCE concepts (highlighted in blue). The visualizations show which spatial features contribute to concept identification in the generated reports. Top row: Zero-Shot setting; bottom row: supervised fine-tuning setting.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: SpLiCE performance across vocabulary sizes and λ values. The figure reports three complementary metrics: Left : precision of extracted concepts; Center : cosine similarity between the original CLIP embedding and its sparse reconstruction; Right : average number of active concepts (sparsity). The results highlight the trade-off between fidelity, interpretability, and terminological precision.
Figure 3: Precision as a function of the number of selected concepts τ∈{3,5,7} for both the 100- and 200-bigram vocabularies. Precision decreases monotonically with increasing τ , reflecting the progressive inclusion of lower-ranked concepts with weaker activation strengths. The 100-bigram vocabulary maintains a consistent advantage across all τ values, indicating that the highest-confidence concepts are highly informative for guiding retrieval-augmented report generation.
Automated radiology report generation holds immense potential to alleviate the heavy workload of radiologists. Despite the formidable vision-language capabilities of recent Multimodal Large Language Models (MLLMs), their clinical deployment is severely constrained by inherent limitations: their "black-box" decision-making renders the generated reports untraceable due to the lack of explicit visual evidence to support the diagnosis, and they struggle to access external domain knowledge. To address these challenges, we propose the Evidence-driven Radiology Report Generation Agent (EviAgent). Unlike opaque end-to-end paradigms, EviAgent coordinates a transparent reasoning trajectory by breaking down the complex generation process into granular operational units. We integrate multi-dimensional visual experts and retrieval mechanisms as external support modules, endowing the system with explicit visual evidence and high-quality clinical priors. Extensive experiments on MIMIC-CXR, CheXpert Plus, and IU-Xray datasets demonstrate that EviAgent outperforms both large-scale generalist models and specialized medical models, providing a robust and trustworthy solution for automated radiology report generation.
Tuoshi Qi, Shenshen Bu, Yingfei Xiang +1
School of Computer Science and Engineering, Sun Yat-sen University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Sangfor Technologies, Shenzhen, Guangdong, China
Despite the remarkable generalization capabilities of Large Language Models, generating clinically accurate and structured radiology reports from medical images remains a significant challenge. Current end-to-end approaches function as black boxes, prone to factual hallucinations and generating ungrounded findings. This lack of reliability and failure to adhere to domain-specific structuring protocols, fundamentally limits their clinical utility. We present MedPAO-VL, a multimodal agent that shifts the paradigm from monolithic generation to explainable, iterative, protocol-compliant structured report generation. Unlike deterministic models, our plan-act-observe loop functions as a goal-oriented agent: it first perceives the Chest X-Ray through an explainable vision module designed to capture inherent diagnostic uncertainty, then iteratively refines the generated report by observing semantic discrepancies and self-correcting its generation for better alignment with visually detected clinical findings. Evaluation of our framework demonstrates that this agentic reasoning improves clinical correctness and completeness, achieving an expert-rated accuracy score of 3.5/5 and a structural score of 4.3/5. By prioritizing pathology specific explainability and iterative refinement over opaque end-to-end mapping, MedPAO-VL offers a transparent and clinically viable alternative for automated structured radiology report generation.
Department of Data Science and Artificial Intelligence, Indian Institute of Technology Madras, Chennai, India · TANUH – The AI Centre of Excellence in Healthcare, Indian Institute of Science, Bengaluru, India · Department of Radiology (Division of Interventional Radiology), Ramaiah Medical College and Memorial hospital, Bengaluru, India
Radiology report generation (RRG) has attracted significant attention due to its potential to reduce the workload of radiologists. The performance of current RRG approaches remains unsatisfactory against clinical standards. This paper introduces a novel RRG method, MLLM-RRG, that integrates multimodal large language models (MLLMs) with various types of clinical knowledge to generate accurate and comprehensive chest X-ray reports. Our method first designs a referring anatomical feature extractor that leverages anatomical knowledge to analyze different regions of the chest X-ray image and extract visual features without explicitly detecting regions. Next, based on the MLLM's decoder, we develop a multimodal report generator that leverages multimodal prompts constructed from dedicated visual features and textual instructions to produce the radiology report in an auto-regressive way. Finally, we introduce a disease-oriented clinical classification and alignment scheme in a multi-task learning manner to leverage disease knowledge to better preserve the clinical relevance among the generated reports. Once the model is trained, we also introduce a novel clinical quality reinforcement learning strategy to enhance the MLLM with report knowledge, further refining the tones of the generated reports towards radiologists. Extensive experiments on the MIMIC-CXR and IU X-Ray datasets demonstrate the superiority of our method over the state of the art. Our codes will be available at https://github.com/viscom-tongji/MLLM-RRG.
Miaojing Shi, Tianyu Cen, Zijie Yue +3
College of Electronic and Information Engineering, Tongji University, Shanghai, China