cs.CVFeb 17, 2026
SaveConcept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation
Organizations: Department of Engineering, Research Unit of Artificial Intelligence and Computer Systems, Università Campus Bio-Medico of Roma, Rome, Italy. · Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy. · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden. · UniCamillus-Saint Camillus International University of Health Sciences, Rome, Italy.
Abstract
Radiology Report Generation (RRG) through Vision-Language Models (VLMs) promises to reduce documentation burden, improve reporting consistency, and accelerate clinical workflows. However, their clinical adoption remains limited by the lack of interpretability and the tendency to hallucinate findings misaligned with imaging evidence. Existing research typically treats interpretability and accuracy as separate objectives, with concept-based explainability techniques focusing primarily on transparency, while Retrieval-Augmented Generation (RAG) methods targeting factual grounding through external retrieval. We present Concept-Enhanced Multimodal RAG (CEMRAG), a unified framework that decomposes visual representations into interpretable clinical concepts and integrates them with multimodal RAG. This approach exploits enriched contextual prompts for RRG, improving both interpretability and factual accuracy. Experiments on MIMIC-CXR and IU X-Ray across multiple VLM architectures, training regimes, and retrieval configurations demonstrate consistent improvements over both conventional RAG and concept-only baselines on clinical accuracy metrics and standard NLP measures. These results challenge the assumed trade-off between interpretability and performance, showing that transparent visual concepts can enhance rather than compromise diagnostic accuracy in medical VLMs. Our modular design decomposes interpretability into visual transparency and structured language model conditioning, providing a principled pathway toward clinically trustworthy AI-assisted radiology.
Figures & tables
Figure 1: CEMRAG overall framework, combining interpretable concept extraction with RAG for transparent and accurate radiology reporting.
| Strategy | Inputs | Prompt Structure | Interpretability | Factual Grounding |
| Image-Only | <Instruction> | black-box | Visual encoder and LLM priors only | |
| Concepts | , | <Instruction> + <Keywords> | Concept-level via visual-semantic alignment | Sparse visual concept decomposition |
| RAG | , | <Instruction> + <Similar Findings> | Indirect via retrieved context | Visual similarity-based nearest neighbours |
| CEMRAG | , , | <Task> + <Keywords> + <Similar Findings> + <Final Instruction> | Dual: concept annotations and similar cases | Concept-guided focus to salient findings |
Table 1: Experimental configurations for radiology report generation, comparing input modalities, prompt structures, interpretability mechanisms, and factual grounding strategies. Notation: denotes visual embeddings, extracted concepts and retrieved reports. Instruction : “Provide a description of the findings in the radiology image”; Task : “Write the report of the radiology image taking information from similar FINDINGS. Consider as more relevant sentences that contain any of the KEYWORDS in the FINDINGS”; Final Instruction : “Write a paragraph with only the report relying in detail on the FINDINGS”.
| Model | Method | Micro-F1 14 | Micro-F1 5 | Macro-F1 14 | Macro-F1 5 | F1-RG | B-1 | B-4 | R-L |
|---|---|---|---|---|---|---|---|---|---|
| Zero-Shot | |||||||||
| LLaVA-Med | Image-Only | 0.255 | 0.237 | 0.113 | 0.134 | 0.052 | 18.81 | 0.71 | 0.131 |
| + Concepts | 0.401 | 0.412 | 0.238 | 0.355 | 0.073 | 21.47 | 1.29 | 0.146 | |
| + RAG | 0.498 | 0.528 | 0.314 | 0.447 | 0.184 | 27.08 | 4.60 | 0.187 | |
| + CEMRAG | 0.502 | 0.529 | 0.319 | 0.449 | 0.185 | 29.34 | 4.64 | 0.189 | |
| LLaVA with CXR-CLIP | Image-Only | 0.213 | 0.174 | 0.101 | 0.123 | 0.151 | 19.20 | 4.69 | 0.185 |
Table 2: Quantitative results on the MIMIC-CXR test set for two model configurations (LLaVA-Med and LLaVA with CXR-CLIP) and two training regimes ( Zero-Shot and Supervised Fine-Tuning ). We report F1-RadGraph (F1-RG), and NLP metrics (BLEU-1 as B-1, BLEU-4 as B-4, ROUGE-L as R-L). CheXbert-based label metrics are reported over 14 labels (Micro-F1 14 , Macro-F1 14 ) and over the 5 most prevalent findings (Micro-F1 5 , Macro-F1 5 ). Retrieval is performed from the MIMIC-CXR training set.
| Model | Method | Micro-F1 14 | Micro-F1 5 | Macro-F1 14 | Macro-F1 5 | F1-RG | B-1 | B-4 | R-L |
|---|---|---|---|---|---|---|---|---|---|
| Zero-shot | |||||||||
| LLaVA-Med | Image-Only | 0.063 | 0.042 | 0.047 | 0.019 | 0.074 | 17.36 | 1.08 | 0.125 |
| + Concepts | 0.122 | 0.162 | 0.092 | 0.134 | 0.064 | 13.79 | 0.85 | 0.111 | |
| + RAG | 0.377 | 0.344 | 0.220 | 0.271 | 0.228 | 21.04 | 3.60 | 0.177 | |
| + CEMRAG | 0.387 | 0.397 | 0.252 | 0.315 | 0.234 | 24.34 | 4.10 | 0.191 | |
| LLaVA with CXR-CLIP | Image-Only | 0.305 | 0.082 | 0.038 | 0.043 | 0.188 | 5.56 | 1.26 | 0.168 |
Table 3: Quantitative results on the IU X-Ray test set for two model configurations (LLaVA-Med and LLaVA with CXR-CLIP) and two training regimes ( Zero-Shot and Supervised Fine-Tuning ). We report F1-RadGraph (F1-RG), and NLP metrics (BLEU-1 as B-1, BLEU-4 as B-4, ROUGE-L as R-L). CheXbert-based label metrics are reported over 14 labels (Micro-F1 14 , Macro-F1 14 ) and over the 5 most prevalent findings (Micro-F1 5 , Macro-F1 5 ). Retrieval is performed cross-domain from the MIMIC-CXR training set.
| Zero-Shot Setting | |
|---|---|
| Chest X-Ray Image | Ground Truth Report |
| Endotracheal tube terminates 2 cm above the carina. Orogastric tube terminates in the stomach. Right internal jugular catheter terminates in the mid SVC. Lungs are low in volume with stable right upper lung opacities which are better assessed on the recent chest CT but suspicious for pneumonia. There is no pneumothorax or pleural effusion. Heart is normal in size. Normal cardiomediastinal silhouette. | |
| SpLiCE Concepts: pulmonary vascularity, overt pulmonary, right upper, endotracheal tube, internal jugular | |
| Image-Only | The chest X-ray image shows bilateral diffuse infiltrates, which are abnormal substances or collections of cells that have entered the lung tissue . These infiltrates can be seen in both lungs and are distributed throughout the lung fields. |
| + Concepts | The chest X-ray shows overt pulmonary edema , which is a condition where there is an abnormal accumulation of fluid in the lungs. Additionally, there is a right upper lobe opacity , which could be due to various reasons such as infection, inflammation, or other lung abnormalities. The internal jugular vein is visible, and an endotracheal tube is in place , which is a tube that is inserted through the mouth or nose and into the trachea to help maintain an open airway and assist with breathing . Lastly, the pulmonary vascularity appears to be normal. |
| + RAG | The chest X-ray findings are similar to those in the earlier study . The main differences observed are the placement of a left IJ catheter and the presence of an indistinct area of opacification in the right upper zone medially , which could represent a developing consolidation. Additionally, there is a right IJ central venous catheter with its tip at the lower SVC near the cavoatrial junction, and the endotracheal tube has been slightly advanced. No pneumothorax is noted. Pulmonary vascular congestion is observed . |
Table 4: Qualitative Comparison in Zero-Shot Setting for LLaVA-Med. Highlighting indicates: incorrect/hallucinated findings , incomplete descriptions , superfluous details , accurate content .
| Supervised Fine-Tuned Setting | |
|---|---|
| Chest X-Ray Image | Ground Truth Report |
| Frontal and lateral views of the chest were obtained. There are low lung volumes which accentuate the bronchovascular markings. Bibasilar opacities are seen, which most likely represent atelectasis, although aspiration or infection are not excluded in the appropriate clinical setting. No pleural effusion or pneumothorax is seen. The cardiac and mediastinal silhouettes are stable and unremarkable. | |
| SpLiCE Concepts: lateral view, lung base, volume low, left hemidiaphragm, appropriate clinical | |
| Image-Only | The heart size is normal. The hilar and mediastinal contours are normal. There is a new left lower lobe opacity, concerning for pneumonia . No pleural effusion or pneumothorax. The visualized osseous structures are unremarkable. |
| + Concepts | In comparison with the study of ___, there are lower lung volumes . Cardiac silhouette is within normal limits. Opacification at the left base is consistent with atelectatic changes and possible small effusion . In the appropriate clinical setting, superimposed pneumonia would have to be considered . |
| + RAG | The lungs are clear without focal consolidation . No pleural effusion or pneumothorax is seen . The cardiac and mediastinal silhouettes are unremarkable . |
Table 5: Qualitative Comparison in Supervised Fine-Tuning Setting or LLaVA-Med. Highlighting indicates: incorrect/hallucinated findings , incomplete descriptions , superfluous details , accurate content .
| Ground Truth | Generated Report |
| Endotracheal tube terminates 2 cm above the carina. Orogastric tube terminates in the stomach. Right internal jugular catheter terminates in the mid SVC. Lungs are low in volume with stable right upper lung opacities which are better assessed on the recent chest CT but suspicious for pneumonia. There is no pneumothorax or pleural effusion. Heart is normal in size. Normal cardiomediastinal silhouette. | The chest X-ray shows several findings. In the right upper zone, there is an indistinct area of opacification, which could represent a developing consolidation. Additionally, there is a right internal jugular central venous catheter with its tip at the lower superior vena cava near the cavoatrial junction. The endotracheal tube has been slightly advanced, with its tip 2.2 cm above the carina. No pneumothorax is noted, and the pulmonary vascular congestion is present. The cardiomediastinal silhouette remains unchanged. |
| Original Image Grad-ECLIP Activation Maps Endotracheal Tube Right Upper | |
| Ground Truth | Generated Report |
| The patient is status post median sternotomy and aortic valve replacement. Heart size is moderately enlarged but unchanged. The aorta is tortuous and calcified. There is mild interstitial pulmonary edema, relatively unchanged. At least small bilateral pleural effusions are present. Bibasilar airspace opacities may reflect compressive atelectasis. There is no pneumothorax. There is evidence of prior vertebroplasty at the thoracolumbar junction. | The patient is status post median sternotomy and CABG. The heart size remains mildly enlarged. The aorta is tortuous and diffusely calcified. There is mild pulmonary edema, worse in the interval, with small bilateral pleural effusions. Bibasilar airspace opacities likely reflect atelectasis. No pneumothorax is identified. There are no acute osseous abnormalities. |
| Original Image Grad-ECLIP Activation Maps Median Sternotomy Bilateral Pleural | |
Table 6: Qualitative Examples with Concept-Specific Visual Explanations. Grad-ECLIP heat maps highlight image regions that influence the LLaVA-Med encoder’s matching between input images and SpLiCE concepts (highlighted in blue). The visualizations show which spatial features contribute to concept identification in the generated reports. Top row: Zero-Shot setting; bottom row: supervised fine-tuning setting.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: SpLiCE performance across vocabulary sizes and values. The figure reports three complementary metrics: Left : precision of extracted concepts; Center : cosine similarity between the original CLIP embedding and its sparse reconstruction; Right : average number of active concepts (sparsity). The results highlight the trade-off between fidelity, interpretability, and terminological precision.
Figure 3: Precision as a function of the number of selected concepts for both the 100- and 200-bigram vocabularies. Precision decreases monotonically with increasing , reflecting the progressive inclusion of lower-ranked concepts with weaker activation strengths. The 100-bigram vocabulary maintains a consistent advantage across all values, indicating that the highest-confidence concepts are highly informative for guiding retrieval-augmented report generation.
More in this topic
Radiology Report Generation
Radiology report generation uses medical images and clinical context to produce structured, clinically accurate findings and impressions.
See all 159 papers