MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation

Raiyan Jahangir; Nafiz Imtiaz Khan; Amritanand Sudheerkumar; Vladimir Filkov

cs.CVFeb 25, 2026

MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation

Authors: Raiyan Jahangir, Nafiz Imtiaz Khan, Amritanand Sudheerkumar, Vladimir Filkov

Organizations: University of California, Davis, CA, USA

Abstract

Screening mammography is high volume, time sensitive, and documentation heavy. Radiologists must translate subtle visual findings into consistent BI-RADS assessments, breast density categories, and structured narrative reports. While recent Vision Language Models (VLMs) enable image-to-text reporting, many rely on closed cloud systems or tightly coupled architectures that limit privacy, reproducibility, and adaptability. We present MammoWise, a local multi-model pipeline that transforms open source VLMs into mammogram report generators and multi-task classifiers. MammoWise supports any Ollama-hosted VLM and mammography dataset, and enables zero-shot, few-shot, and Chain-of-Thought prompting, with optional multimodal Retrieval Augmented Generation (RAG) using a vector database for case-specific context. We evaluate MedGemma, LLaVA-Med, and Qwen2.5-VL on VinDr-Mammo and DMID datasets, assessing report quality (BERTScore, ROUGE-L), BI-RADS classification, breast density, and key findings. Report generation is consistently strong and improves with few-shot prompting and RAG. Classification is feasible but sensitive to model and dataset choice. Parameter-efficient fine-tuning (QLoRA) of MedGemma improves reliability, achieving BI-RADS accuracy of 0.7545, density accuracy of 0.8840, and calcification accuracy of 0.9341 while preserving report quality. MammoWise provides a practical and extensible framework for deploying local VLMs for mammography reporting within a unified and reproducible workflow.

Figures & tables

Figure 1: Overall methodology of the study

Figure 2: RAG pipeline used in this study

BI-RADS	Original	After	Action
1	3331	500	Downsample (random)
2	1167	500	Downsample (random)
3	242	500	Augment (flip + scale)
4	205	500	Augment (flip + scale)
5	55	200	Augment (flip + scale + translation; capped)
Total	5000	2200

Table 1: BI-RADS class counts before and after rebalancing for fine-tuning.

Task	Evaluation Parameters	Without RAG			With RAG
Task	Evaluation Parameters	Best Result	Prompt	Model	Result	Model
BI-RADS	Accuracy	0.2909	Fewshot	LLAVA-Med	0.319	MedGemma
	Precision	0.4121	Fewshot	Qwen2.5VL	0.4508	Qwen2.5VL
	Recall	0.2909	Fewshot	LLAVA-Med	0.319	MedGemma
	F1-score	0.2217	Fewshot	LLAVA-Med	0.2756	Qwen2.5VL
	Specificity	0.8165	Fewshot	LLAVA-Med	0.8265	MedGemma

Table 2: Performance of the models on VinDr-Mammo dataset without and with RAG

Task	Evaluation Parameters	Without RAG			With RAG
Task	Evaluation Parameters	Best Result	Prompt	Model	Best Result	Model
BI-RADS	Accuracy	0.57	CoT	Qwen2.5VL	0.895	MedGemma
	Precision	0.8139	Fewshot	LLAVA-Med	0.9757	MedGemma
	Recall	0.57	CoT	Qwen2.5VL	0.895	MedGemma
	F1-score	0.59	CoT	Qwen2.5VL	0.93002	MedGemma
	Specificity	0.8602	Fewshot	LLAVA-Med	0.9807	MedGemma

Table 3: Performance of the models on DMID dataset without and with RAG

Task	Evaluation Parameter	Zeroshot	Fewshot	CoT	RAG-Fewshot	Finetune
						Generate All Tasks Together				Generate 1 Task at a Time
						3 Epochs	6 Epochs	10 Epochs	15 Epochs	6 Epochs	10 Epochs
BI-RADS	Accuracy	0.2218	0.1019	0.2493	0.319	0.3564	0.4698	0.6355	0.5139	0.4976	0.7545
	Precision	0.2722	0.1515	0.1846	0.3663	0.3608	0.5734	0.6544	0.5444	0.484	0.7356
	Recall	0.2218	0.1019	0.2493	0.319	0.3564	0.4698	0.6355	0.5139	0.4591	0.7516
	F1-score	0.1237	0.0376	0.1412	0.2698	0.3489	0.4186	0.6261	0.4911	0.4152	0.7404

Table 4: Performance comparison of MedGemma model

Task	Best Prompt	Best RAG	$\boldsymbol{\Delta}$ (RAG – Prompt)	Best QLoRA	$\boldsymbol{\Delta}$ (QLoRA – RAG)
BI-RADS	0.1412	0.2698	$+$ 0.1286	0.7404	$+$ 0.4706
Density	0.6900	0.7529	$+$ 0.0629	0.8894	$+$ 0.1365
Calcification	0.6973	0.7694	$+$ 0.0721	0.9384	$+$ 0.1690
Mass	0.5016	0.5404	$+$ 0.0388	0.8777	$+$ 0.3373
Asymmetry	0.7875	0.7787	$-$ 0.0088	0.8261	$+$ 0.0474
Suspicion	0.3369	0.3271	$-$ 0.0098	0.7951	$+$ 0.4680

Table 5: Gain summary of adding RAG and fine-tuning for MedGemma on VinDr-Mammo (macro F1-score).

Task	Our Work	MammoCLIP	LLaVA-Mammo	LLaVA-MultiMammo	PubMedCLIP
BI-RADS	0.7545	–	–	–	0.5325
Breast Density	0.884	0.88	0.766	0.806	–
Mass	0.8740	0.8	–	–	–
Calcification	0.9341	0.98	–	–	–

Table 6: Comparison of classification accuracy on several tasks of MammoWise models with other SOTA

Task	Evaluation Parameter	Models
Task	Evaluation Parameter	MedGemma	Qwen2.5VL	LLAVA-Med
BI-RADS	Accuracy	0.2218	0.1957	0.2273
	Precision	0.2722	0.0855	0.0517
	Recall	0.2218	0.1957	0.2273
	F1-score	0.1237	0.0913	0.0842
	Specificity	0.8006	0.8008	0.8

Table 7: Classification performance of different models on VinDr-Mammo dataset on zero-shot

Task	Evaluation Parameter	Models
Task	Evaluation Parameter	MedGemma	Qwen2.5VL	LLAVA-Med
BI-RADS	Accuracy	0.1019	0.2582	0.2909
	Precision	0.1515	0.4121	0.3493
	Recall	0.1019	0.2582	0.2909
	F1-score	0.0376	0.1876	0.2217
	Specificity	0.8004	0.808	0.8165

Table 8: Classification performance of different models on the VinDr-Mammo dataset on few-shot

Task	Evaluation Parameter	Models
Task	Evaluation Parameter	MedGemma	Qwen2.5VL	LLAVA-Med
BI-RADS	Accuracy	0.2493	0.2291	0.2488
	Precision	0.1846	0.0526	0.1898
	Recall	0.2493	0.2291	0.2488
	F1-score	0.1412	0.0855	0.2059
	Specificity	0.8059	0.8	0.8046

Table 9: Classification performance of different models on VinDr-Mammo dataset on chain-of-thought

Task	Evaluation Parameters	MedGemma	LLAVA-Med	Qwen2.5VL
BI-RADS	Accuracy	0.2878	0.2	0.2918
	Precision	0.2861	0.0824	0.6562
	Recall	0.2878	0.2	0.2918
	F1-score	0.2602	0.1167	0.247
	Specificity	0.8371	0.8	0.8268
	BERTScore	0.8662	0.8322	0.8557

Table 10: Performance of different models on DMID dataset with zero-shot prompting

Task	Evaluation Parameters	MedGemma	LLAVA-Med	Qwen2.5VL
BI-RADS	Accuracy	0.3531	0.3998	0.201
	Precision	0.3717	0.8139	0.2483
	Recall	0.3531	0.3998	0.201
	F1-score	0.3392	0.3419	0.0797
	Specificity	0.8542	0.8602	0.8005
	BERTScore	0.9983	0.9508	0.9999

Table 11: Performance of different models on DMID dataset with few-shot prompting

Task	Evaluation Parameters	MedGemma	LLAVA-Med	Qwen2.5VL
BI-RADS	Accuracy	0.2277	0.183	0.57
	Precision	0.3108	0.336	0.65
	Recall	0.2277	0.183	0.57
	F1-score	0.1462	0.1288	0.59
	Specificity	0.81	0.8358	0.77
	BERTScore	0.8657	0.9939	0.8448

Table 12: Performance of different models on DMID dataset with Chain-of-Thought prompting

Explore similar work

Cross-Modal Clinical Knowledge Integration for Mammography Report Generation
May 29, 2026Jiayi Zhu, Fuxiang Huang, Yu Xie +7Radiology Report Generation Medical Imaging
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China · Lingnan University, Hong Kong, China · The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Peking University Cancer Hospital Yunnan, Kunming, Yunnan, China +4
DobicVLM: Aligning Chest X-Ray Report Generation with Clinically-Grounded Programmatic Rewards via Group Relative Policy Optimization
Jul 21, 2026Thanni Adewuyi, Angelica Obayi, Andem Aniekan +10Radiology Report Generation Radiology VLMs
Dobic Health · HelpMum Africa · University of Ibadan
U-VLM: Hierarchical Vision Language Modeling for Report Generation
Feb 28, 2026Pengcheng Shi, Minghui Zhang, Kehan Song +3Radiology Report Generation Vision-Language Models
Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China

May 29, 2026J/K move · Enter open · S save

Cross-Modal Clinical Knowledge Integration for Mammography Report Generation

Jiayi Zhu, Fuxiang Huang, Yu Xie, Xi Wang+6

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China · Lingnan University, Hong Kong, China · The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Peking University Cancer Hospital Yunnan, Kunming, Yunnan, China+4

Radiology Report Generation Medical Imaging Medical Report Generation Mammography

Open paper AskPDF arXiv

Breast cancer is a major global health concern, and mammography screening plays a central role in early detection. The large volume of screening examinations creates a substantial workload for radiologists, making accurate and consistent report generation a critical clinical challenge. Existing automated mammography report generation methods primarily focus on direct visual-to-text mapping, while overlooking the structured clinical reasoning process followed by radiologists in real-world practice. To address this limitation, we propose MammoRG, a mammography report generation framework that explicitly simulates the clinical reporting workflow by following the BI-RADS guideline and incorporating prior clinical knowledge to produce diagnostic reports. Specifically, MammoRG adopts a two-stage training framework. In the first stage, the model learns to integrate clinically relevant prior knowledge from a patient's four-view mammograms through classification-based supervision. In the second stage, a terminology-aware supervised fine-tuning strategy is introduced to model mammography-specific clinical terms as atomic semantic units, enabling the generation of high-quality reports with improved clinical consistency. To facilitate clinical efficacy evaluation of generated reports, we further develop MammoRGTool, a dedicated mammography report parsing tool that extracts structured clinical information from free-text reports. Extensive experiments demonstrate that MammoRG consistently outperforms existing methods across multiple clinical efficacy metrics, particularly in diagnosis-related BI-RADS F1, where it surpasses the second-best model by 2.73%, 2.04%, 1.90%, and 3.27% on the internal, external 1, external 2, and VinDr-Mammo datasets, respectively.

Abstract

Explore similar work

Cross-Modal Clinical Knowledge Integration for Mammography Report Generation

DobicVLM: Aligning Chest X-Ray Report Generation with Clinically-Grounded Programmatic Rewards via Group Relative Policy Optimization

U-VLM: Hierarchical Vision Language Modeling for Report Generation