Lesion-Aware Attention for Mammography Segmentation with Heterogeneous Supervision

Di Dai; Bo Liu; Liwei Wang

cs.CVOct 11, 2026

Lesion-Aware Attention for Mammography Segmentation with Heterogeneous Supervision

Authors: Di Dai, Bo Liu, Liwei Wang

Organizations: State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, China

Abstract

Accurate lesion segmentation in mammography is important for computer-aided breast cancer diagnosis, but dense pixel-level annotations remain costly and limited. Existing public segmentation datasets usually cover only a small set of lesion types with manually delineated masks, whereas larger mammography datasets often provide only image-level diagnostic labels. This annotation mismatch makes it difficult to train segmentation models that can generalize to rare or weakly annotated lesion categories. To address this problem, we study open-vocabulary mammographic lesion segmentation under a mixed-supervision setting, where pixel-level masks are available for base lesion categories and image-level labels are available for a broader set of categories. We propose a two-branch framework that jointly learns from segmentation data and classification data. The segmentation branch learns lesion boundaries from annotated masks, while the classification branch introduces additional image-level semantic supervision. To better transfer image-level supervision to lesion-relevant regions, we further introduce a Lesion-wise Attentive Classifier that maintains category-specific feature prototypes and generates lesion-aware attention maps for foreground feature pooling. This design reduces the influence of background tissue and encourages more compact lesion representations. Experiments on multiple mammography datasets show that the proposed method consistently improves open-vocabulary segmentation baselines on both base and novel lesion categories, while reducing the gap to fully supervised models that require dense target-domain masks. Qualitative results further indicate that our method produces more complete lesion localization for small, low-contrast, and visually subtle mammographic findings.

Figures & tables

Fig. 1: Overview of the proposed framework for mammography images. The simple baseline jointly learns from segmentation and image-level classification data via pixel-level and image-level objectives ( $\mathcal{L}_{\text{seg}}$ , $\mathcal{L}_{\text{cls}}$ ). The proposed Lesion-wise Attentive Classifier (LA-Classifier) maintains lesion-wise features in a memory bank to highlight foreground lesion pixels and suppress background. The attentively pooled score map is supervised by an auxiliary image-level classification loss ( $\mathcal{L}_{\text{aux}}$ ). LA-Map denotes the lesion-wise attention map.

Model	Training Data	Test Data	mIoU
DenseCLIP	ITPair	Mammo-bench $-$ CBIS-DDSM	13.6
Basic Segmenter	CBIS-DDSM	Mammo-bench $-$ CBIS-DDSM	73.6
Basic Segmenter	Mammo-bench	Mammo-bench $-$ CBIS-DDSM	76.2
DenseCLIP	ITPair	Mammo-bench $-$ INBreast	10.8
Basic Segmenter	INBreast	Mammo-bench $-$ INBreast	68.6
Basic Segmenter	Mammo-bench	Mammo-bench $-$ INBreast	73.4

TABLE I: Results of the exploratory experiment. For the second row, the evaluation categories are not in the training vocabulary. For the third row, the evaluation categories are within the training vocabulary. CLIP-RN50x64 with VILD prompt engineering is used to generate DenseCLIP [ 36 ] results.

Dataset	Year	Images	Annotation	BI-RADS
MIAS [ 37 ]	1994	322	Circle around the finding	No
CBIS-DDSM [ 13 ]	2017	3,103	Contour enclosing the findings	Yes
INBreast [ 14 ]	2012	410	Contour enclosing the finding	Yes
CMMD [ 15 ]	2021	5,202	Pathological labels	No
VinDr-Mammo [ 16 ]	2022	20,000	Rectangular region of interest	Yes
Mammo-Bench [ 38 ]	2025	19,731	Region of interest mask	Yes

TABLE II: Existing Public Mammography Datasets

Model	Backbone	Training Data	MIAS-mIoU			Training Data	INBreast-mIoU
Model	Backbone	Training Data	All	Base	Novel	Training Data	All	Base	Novel
DenseCLIP [ 36 ]	RN50x64 [ 40 ]	ITPairs	12.4	16.3	10.2	ITPairs	5.3	9.6	4.8
ReCo [ 41 ]	RN50x64	ITPairs	13.8	16.7	11.4	ITPairs	5.9	9.8	5.1
ZSBaseline [ 42 ]	ViT-B/16	CBIS-DDSM	61.9	66.4	58.7	CBIS-DDSM	60.5	65.1	57.9
LSeg [ 43 ]	ViT-B/16	CBIS-DDSM	63.2	68.1	59.6	CBIS-DDSM	61.7	66.3	58.8
OpenSeg [ 44 ]	EfficientNet-B7	CBIS-DDSM	65.4	-	-	CBIS-DDSM	63.8	-	-

TABLE III: Comparison of baselines and our proposed models and supervised learning models on mammography datasets. The supervised learning results are shown in gray . CBIS-DDSM and INBreast are used as segmentation data; CMMD and VinDr-Mammo are used as classification data. ITPair stands for the web-scale image-text pairs to train a CLIP/ALIGN model. All models are evaluated without multi-scale testing and Dense CRF post-processing.

LAC-CMMD	LAC-VinDr-Mammo	All	Base	Novel
-	-	71.3	78.7	66.3
-	$\checkmark$	72.4	79.1	67.8
$\checkmark$	-	73.6	80.2	69.5
$\checkmark$	$\checkmark$	74.2	80.9	70.6

TABLE IV: Ablation study on applying the LA-Classifier to different classification datasets. LAC-CMMD and LAC-VinDr-Mammo denote using LA-Classifier on CMMD and VinDr-Mammo respectively.

LA-Classifier Design	All	Base	Novel
Single-Image	70.1	78.6	64.8
Cross-Image	72.4	81.3	67.8

TABLE V: Comparison of different LA-Classifier designs. Single-image selects foreground regions from classification score maps, while Cross-image leverages cross-image semantic information (our method).

Size	All	Base	Novel	Top-K	All	Base	Novel
10	72.6	82.8	76.2	10	71.6	80.3	76.2
20	72.0	80.5	76.7	20	72.4	81.3	76.8
40	72.2	82.0	76.1	40	72.2	82.0	76.1
80	71.6	80.9	75.7	80	71.4	80.0	75.9

TABLE VI: Ablation on memory bank size

M

(left) and top-

K

region selection (right). We select the hyper-parameters with the best performance on novel categories.

Fig. 2: Error analysis of open-vocabulary mammographic lesion segmentation. (a) Decomposition of pixel-level errors into false-negative, false-positive, boundary, and category-confusion errors. (b) Failure distribution with respect to lesion size and local contrast. Small and low-contrast lesions are the dominant source of remaining errors, and colors indicate the dominant error type for each lesion component.

Fig. 3: Generalization and reliability visualization. (a) Cross-dataset qualitative results on MIAS, INBreast, and Mammo-Bench. Each block shows the mammogram crop, prediction, and uncertainty map. (b) Visualization of the supervision transfer process from global image-level activation to lesion-wise attention and final segmentation. Compared with the global average pooling CAM, the proposed LA-Map focuses more accurately on lesion-relevant regions. (c) Reliability analysis with representative successful, partial, and failure cases. Compact lesions are localized reliably, while fuzzy boundaries and extremely subtle calcifications remain difficult.

Explore similar work

Region-Grounded Vision-Language Learning for Detection-Guided Mammographic Lesion Classification
Jul 17, 2026Zhengbo Zhou, Jiren Li, Dooman Arefan +2Radiology VLMs Vision-Language Models
Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA · Department of Radiology, University of Pittsburgh, Pittsburgh, PA, USA · Department of Biomedical Informatics and Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA
DSU-Net: An Attention-Enhanced Dense Skip U-Net for Breast Lesion Segmentation in Mammographic Images
Jun 3, 2026Reza Bozorgpour, Mohammadreza Soltany SadrabadiMedical Imaging U-Net
Department of Biomedical Engineering, University of Wisconsin-Milwaukee, Milwaukee, WI, USA · Department of Mechanical Engineering, Northern Arizona University, Flagstaff, AZ, USA
Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms
Apr 21, 2026Samyak Sanghvi, Piyush Miglani, Sarvesh Shashikumar +3Medical Imaging Vision Foundation Model Adaptation
Department of Computer Science and Engineering, IIT Delhi, New Delhi, India · Yardi School of AI, IIT Delhi, New Delhi, India · Department of Radiodiagnosis, PGIMER Chandigarh, Chandigarh, India

Jul 17, 2026J/K move · Enter open · S save

Region-Grounded Vision-Language Learning for Detection-Guided Mammographic Lesion Classification

Zhengbo Zhou, Jiren Li, Dooman Arefan, Margarita Zuley+1

Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA · Department of Radiology, University of Pittsburgh, Pittsburgh, PA, USA · Department of Biomedical Informatics and Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA

Radiology VLMs Vision-Language Models Mammography Multimodal Contrastive Learning Medical Image Localization

Open paper AskPDF arXiv

Vision-language models trained with contrastive objectives have shown promise in medical image analysis. However, conventional global image-text alignment is ill-suited for mammography, where diagnostically relevant lesions are spatially localized and occupy only a small fraction of the image. Subtle morphological cues critical for malignancy assessment can be diluted when representations are learned at the whole-image level. In this work, we propose a novel region-grounded vision-language learning method for detection-guided mammographic lesion classification. The method mirrors radiologists' diagnostic paradigm. First, a region-text contrastive pretraining stage aligns lesion-specific features with structured clinical descriptors derived from radiology metadata. To mitigate semantic collapse and background bias in low-vocabulary settings, we introduce a multi-component objective incorporating positive alignment, fine-grained semantic hard negatives, and background suppression. Second, an auxiliary lesion detection head is jointly optimized with contrastive classification to preserve spatial sensitivity and enable localization-aware malignancy classification. Extensive experiments on two independent datasets, CBIS-DDSM and VinDr-Mammo, show superior performance of our method compared to related methods under in-domain, cross-dataset, and transfer learning settings.

Abstract

Explore similar work

Region-Grounded Vision-Language Learning for Detection-Guided Mammographic Lesion Classification

DSU-Net: An Attention-Enhanced Dense Skip U-Net for Breast Lesion Segmentation in Mammographic Images

Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms