MammoSeek: A Mammography Agent with selective zooming for grounded reports

Haoyan Ding; Qian Wang; Lixia Wang; Debiao Li

cs.CVNov 4, 2026

MammoSeek: A Mammography Agent with selective zooming for grounded reports

Authors: Haoyan Ding, Qian Wang, Lixia Wang, Debiao Li

Organizations: University of California, Los Angeles, Los Angeles, CA, USA · Biomedical Imaging Research Institute, Cedars-Sinai Medical Center, Los Angeles, CA, USA

Abstract

Radiologists read a screening mammogram by first forming a global impression of the four views and then selectively magnifying suspicious regions before assigning a BI-RADS assessment. MLLMs, in contrast, see the whole image at a fixed, reduced resolution, which makes sub-millimeter calcifications invisible, and they perform close to chance on mammography. Therefore, we present MammoSeek, a mammography agent that reproduces this reading process. After forming an impression of each breast, the agent performs selective zooming: it decides by itself which detector candidates deserve a closer look and inspects only those on full-resolution views. Guided by a BI-RADS assessment skill, it then assigns a category and generates a localized, lexicon-structured report for each breast; its assessment is combined with image-model scores in a decision layer to decide recall. On an internal (EMBED) and an external (VinDr-Mammo) dataset, \ours reaches exam-level AUCs of 0.824 and 0.818, far above other MLLMs, mammography agent and CNN classifiers. Selective zooming also outperforms both no zooming and forced zooming in localization, structured-report F1 and recall decisions at low cost.

Figures & tables

Figure 1: MammoSeek follows the radiologist’s reading process. The MLLM agent (blue) calls tools (grey) at each step. Steps 1–3 form a tool-calling loop whose length and zoom targets are chosen by the agent. The recall decision combines the agent’s BI-RADS assessment with image-model scores; recalled breasts are reported with the agent’s findings.

Figure 2: Selective zooming. (a) Fraction of breasts the agent zooms into (EMBED: negative vs. abnormal; VinDr: BI-RADS category). (b) Localization and report F1 with no zooming, forced zooming and selective zooming decided by the agent. (c) A cluster of suspicious calcifications as the agent sees it in the whole view, in a full-resolution crop with standard windowing, and in the calcification view.

	EMBED (internal)					VinDr-Mammo (external)
Method	AUC	Sens/Spec	$\kappa$	Loc	Rep	AUC	Sens/Spec	$\kappa$	Loc	Rep
Qwen2.5-VL-7B [ 2 ]	.535	.06/.98	.02	.02	.15	.519	.02/1.0	.02	.00	.14
Qwen3-VL-32B [ 3 ]	.630	.36/.89	.20	.18	.12	.609	.14/.97	.16	.08	.12
Lingshu-7B [ 18 ] ‡	.491	1.0/.00	.00	–	.08	.511	1.0/.00	.00	–	.25
NYU BI-RADS [ 6 ] §	.615	.00/1.0	.09	–	–	.606	.00/1.0	.01	–	–
NYU cancer classifier [ 17 ]	.796	.45/.90	.33 ∗	–	–	.797	.24/.99	.25 ∗	–	–

Table 1: Diagnosis, BI-RADS agreement, localization and structured reporting on EMBED (internal) and VinDr-Mammo (external). AUC: exam level. Sens/Spec: breast-level recall. Rep: structured-report F1.

Configuration	AUC	Youden	$\kappa$	Loc	Rep	Zoom
w/o agent (decision layer)	.811 / .828	.379 / .240	.32 / .25 ∗	.35 / .25	.22 / .12	–
w/o assessment skill	.805 / .820	.397 / .352	.30 / .39	.34 / .34	.24 / .29	1.9 / 1.4
No zoom	.822 / .823	.381 / .302	.30 / .41	.30 / .32	.17 / .11	–
Forced zoom	.823 / .820	.378 / .337	.31 / .38	.32 / .35	.23 / .30	2.2 / 1.9
MammoSeek (selective zoom)	.824 / .818	.403 / .397	.33 / .43	.35 / .43	.24 / .38	1.0 / 0.8

Table 2: Ablations of the agent, the assessment skill and the zooming policy. Zoom: number of zooms per exam. Values are EMBED / VinDr.

Mammography

Mammography uses breast X-rays to screen for cancer and guide diagnosis, while AI analyzes lesions, breast density, and cancer risk.

See all 45 papers

MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation
Feb 25, 2026Raiyan Jahangir, Nafiz Imtiaz Khan, Amritanand Sudheerkumar +1Radiology Report Generation Mammography
University of California, Davis, CA, USA
From Density to Biopsy Decisions and Malignancy Prediction: A Benchmark Study of Multimodal Large Language Models Against Radiologists in Digital and Contrast-Enhanced Mammography
Sep 13, 2026Ali Abbasian Ardakani, Afshin Mohammadi, Taha Yusuf Kuzan +11Medical VLMs Mammography Image Classification
Department of Medicine, Faculty of Medicine and Dentistry, Danube Private University, Krems an der Donau, Austria · Department of Radiology, Faculty of Medicine, Urmia University of Medical Science, Urmia, Iran · Department of Radiology, Yeditepe University, Istanbul, Türkiye +10
Cross-Modal Clinical Knowledge Integration for Mammography Report Generation
May 29, 2026Jiayi Zhu, Fuxiang Huang, Yu Xie +7Radiology Report Generation Medical Imaging
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China · Lingnan University, Hong Kong, China · The Third Affiliated Hospital of Kunming Medical University, Yunnan Cancer Hospital, Peking University Cancer Hospital Yunnan, Kunming, Yunnan, China +4

Feb 25, 2026J/K move · Enter open · S save

MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation

Raiyan Jahangir, Nafiz Imtiaz Khan, Amritanand Sudheerkumar, Vladimir Filkov

University of California, Davis, CA, USA

Radiology Report Generation Mammography Medical Image Classification Multimodal RAG

Open paper AskPDF arXiv

Screening mammography is high volume, time sensitive, and documentation heavy. Radiologists must translate subtle visual findings into consistent BI-RADS assessments, breast density categories, and structured narrative reports. While recent Vision Language Models (VLMs) enable image-to-text reporting, many rely on closed cloud systems or tightly coupled architectures that limit privacy, reproducibility, and adaptability. We present MammoWise, a local multi-model pipeline that transforms open source VLMs into mammogram report generators and multi-task classifiers. MammoWise supports any Ollama-hosted VLM and mammography dataset, and enables zero-shot, few-shot, and Chain-of-Thought prompting, with optional multimodal Retrieval Augmented Generation (RAG) using a vector database for case-specific context. We evaluate MedGemma, LLaVA-Med, and Qwen2.5-VL on VinDr-Mammo and DMID datasets, assessing report quality (BERTScore, ROUGE-L), BI-RADS classification, breast density, and key findings. Report generation is consistently strong and improves with few-shot prompting and RAG. Classification is feasible but sensitive to model and dataset choice. Parameter-efficient fine-tuning (QLoRA) of MedGemma improves reliability, achieving BI-RADS accuracy of 0.7545, density accuracy of 0.8840, and calcification accuracy of 0.9341 while preserving report quality. MammoWise provides a practical and extensible framework for deploying local VLMs for mammography reporting within a unified and reproducible workflow.

Abstract

Mammography

Explore similar work

MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation

From Density to Biopsy Decisions and Malignancy Prediction: A Benchmark Study of Multimodal Large Language Models Against Radiologists in Digital and Contrast-Enhanced Mammography

Cross-Modal Clinical Knowledge Integration for Mammography Report Generation