Lesion-Aware Attention for Mammography Segmentation with Heterogeneous Supervision

Di Dai; Bo Liu; Liwei Wang

cs.CVOct 11, 2026

Lesion-Aware Attention for Mammography Segmentation with Heterogeneous Supervision

Authors: Di Dai, Bo Liu, Liwei Wang

Organizations: State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing, China

Abstract

Accurate lesion segmentation in mammography is important for computer-aided breast cancer diagnosis, but dense pixel-level annotations remain costly and limited. Existing public segmentation datasets usually cover only a small set of lesion types with manually delineated masks, whereas larger mammography datasets often provide only image-level diagnostic labels. This annotation mismatch makes it difficult to train segmentation models that can generalize to rare or weakly annotated lesion categories. To address this problem, we study open-vocabulary mammographic lesion segmentation under a mixed-supervision setting, where pixel-level masks are available for base lesion categories and image-level labels are available for a broader set of categories. We propose a two-branch framework that jointly learns from segmentation data and classification data. The segmentation branch learns lesion boundaries from annotated masks, while the classification branch introduces additional image-level semantic supervision. To better transfer image-level supervision to lesion-relevant regions, we further introduce a Lesion-wise Attentive Classifier that maintains category-specific feature prototypes and generates lesion-aware attention maps for foreground feature pooling. This design reduces the influence of background tissue and encourages more compact lesion representations. Experiments on multiple mammography datasets show that the proposed method consistently improves open-vocabulary segmentation baselines on both base and novel lesion categories, while reducing the gap to fully supervised models that require dense target-domain masks. Qualitative results further indicate that our method produces more complete lesion localization for small, low-contrast, and visually subtle mammographic findings.

Figures & tables

Fig. 1: Overview of the proposed framework for mammography images. The simple baseline jointly learns from segmentation and image-level classification data via pixel-level and image-level objectives ( $\mathcal{L}_{\text{seg}}$ , $\mathcal{L}_{\text{cls}}$ ). The proposed Lesion-wise Attentive Classifier (LA-Classifier) maintains lesion-wise features in a memory bank to highlight foreground lesion pixels and suppress background. The attentively pooled score map is supervised by an auxiliary image-level classification loss ( $\mathcal{L}_{\text{aux}}$ ). LA-Map denotes the lesion-wise attention map.

Model	Training Data	Test Data	mIoU
DenseCLIP	ITPair	Mammo-bench $-$ CBIS-DDSM	13.6
Basic Segmenter	CBIS-DDSM	Mammo-bench $-$ CBIS-DDSM	73.6
Basic Segmenter	Mammo-bench	Mammo-bench $-$ CBIS-DDSM	76.2
DenseCLIP	ITPair	Mammo-bench $-$ INBreast	10.8
Basic Segmenter	INBreast	Mammo-bench $-$ INBreast	68.6
Basic Segmenter	Mammo-bench	Mammo-bench $-$ INBreast	73.4

TABLE I: Results of the exploratory experiment. For the second row, the evaluation categories are not in the training vocabulary. For the third row, the evaluation categories are within the training vocabulary. CLIP-RN50x64 with VILD prompt engineering is used to generate DenseCLIP [ 36 ] results.

Dataset	Year	Images	Annotation	BI-RADS
MIAS [ 37 ]	1994	322	Circle around the finding	No
CBIS-DDSM [ 13 ]	2017	3,103	Contour enclosing the findings	Yes
INBreast [ 14 ]	2012	410	Contour enclosing the finding	Yes
CMMD [ 15 ]	2021	5,202	Pathological labels	No
VinDr-Mammo [ 16 ]	2022	20,000	Rectangular region of interest	Yes
Mammo-Bench [ 38 ]	2025	19,731	Region of interest mask	Yes

TABLE II: Existing Public Mammography Datasets

Model	Backbone	Training Data	MIAS-mIoU			Training Data	INBreast-mIoU
Model	Backbone	Training Data	All	Base	Novel	Training Data	All	Base	Novel
DenseCLIP [ 36 ]	RN50x64 [ 40 ]	ITPairs	12.4	16.3	10.2	ITPairs	5.3	9.6	4.8
ReCo [ 41 ]	RN50x64	ITPairs	13.8	16.7	11.4	ITPairs	5.9	9.8	5.1
ZSBaseline [ 42 ]	ViT-B/16	CBIS-DDSM	61.9	66.4	58.7	CBIS-DDSM	60.5	65.1	57.9
LSeg [ 43 ]	ViT-B/16	CBIS-DDSM	63.2	68.1	59.6	CBIS-DDSM	61.7	66.3	58.8
OpenSeg [ 44 ]	EfficientNet-B7	CBIS-DDSM	65.4	-	-	CBIS-DDSM	63.8	-	-

TABLE III: Comparison of baselines and our proposed models and supervised learning models on mammography datasets. The supervised learning results are shown in gray . CBIS-DDSM and INBreast are used as segmentation data; CMMD and VinDr-Mammo are used as classification data. ITPair stands for the web-scale image-text pairs to train a CLIP/ALIGN model. All models are evaluated without multi-scale testing and Dense CRF post-processing.

LAC-CMMD	LAC-VinDr-Mammo	All	Base	Novel
-	-	71.3	78.7	66.3
-	$\checkmark$	72.4	79.1	67.8
$\checkmark$	-	73.6	80.2	69.5
$\checkmark$	$\checkmark$	74.2	80.9	70.6

TABLE IV: Ablation study on applying the LA-Classifier to different classification datasets. LAC-CMMD and LAC-VinDr-Mammo denote using LA-Classifier on CMMD and VinDr-Mammo respectively.

LA-Classifier Design	All	Base	Novel
Single-Image	70.1	78.6	64.8
Cross-Image	72.4	81.3	67.8

TABLE V: Comparison of different LA-Classifier designs. Single-image selects foreground regions from classification score maps, while Cross-image leverages cross-image semantic information (our method).

Size	All	Base	Novel	Top-K	All	Base	Novel
10	72.6	82.8	76.2	10	71.6	80.3	76.2
20	72.0	80.5	76.7	20	72.4	81.3	76.8
40	72.2	82.0	76.1	40	72.2	82.0	76.1
80	71.6	80.9	75.7	80	71.4	80.0	75.9

TABLE VI: Ablation on memory bank size

M

(left) and top-

K

region selection (right). We select the hyper-parameters with the best performance on novel categories.

Fig. 2: Error analysis of open-vocabulary mammographic lesion segmentation. (a) Decomposition of pixel-level errors into false-negative, false-positive, boundary, and category-confusion errors. (b) Failure distribution with respect to lesion size and local contrast. Small and low-contrast lesions are the dominant source of remaining errors, and colors indicate the dominant error type for each lesion component.

Fig. 3: Generalization and reliability visualization. (a) Cross-dataset qualitative results on MIAS, INBreast, and Mammo-Bench. Each block shows the mammogram crop, prediction, and uncertainty map. (b) Visualization of the supervision transfer process from global image-level activation to lesion-wise attention and final segmentation. Compared with the global average pooling CAM, the proposed LA-Map focuses more accurately on lesion-relevant regions. (c) Reliability analysis with representative successful, partial, and failure cases. Compact lesions are localized reliably, while fuzzy boundaries and extremely subtle calcifications remain difficult.

Explore similar work

Jul 17, 2026

Region-Grounded Vision-Language Learning for Detection-Guided Mammographic Lesion Classification

Vision-language models trained with contrastive objectives have shown promise in medical image analysis. However, conventional global image-text alignment is ill-suited for mammography, where diagnostically relevant lesions are spatially localized and occupy only a small fraction of the image. Subtle morphological cues critical for malignancy assessment can be diluted when representations are learned at the whole-image level. In this work, we propose a novel region-grounded vision-language learning method for detection-guided mammographic lesion classification. The method mirrors radiologists' diagnostic paradigm. First, a region-text contrastive pretraining stage aligns lesion-specific features with structured clinical descriptors derived from radiology metadata. To mitigate semantic collapse and background bias in low-vocabulary settings, we introduce a multi-component objective incorporating positive alignment, fine-grained semantic hard negatives, and background suppression. Second, an auxiliary lesion detection head is jointly optimized with contrastive classification to preserve spatial sensitivity and enable localization-aware malignancy classification. Extensive experiments on two independent datasets, CBIS-DDSM and VinDr-Mammo, show superior performance of our method compared to related methods under in-domain, cross-dataset, and transfer learning settings.

Zhengbo Zhou, Jiren Li, Dooman Arefan +2

Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA · Department of Radiology, University of Pittsburgh, Pittsburgh, PA, USA · Department of Biomedical Informatics and Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA

Radiology VLMs Vision-Language Models Mammography Multimodal Contrastive Learning Medical Image Localization

Ask PDF

Jun 3, 2026

DSU-Net: An Attention-Enhanced Dense Skip U-Net for Breast Lesion Segmentation in Mammographic Images

Breast cancer remains one of the leading causes of cancer-related mortality among women worldwide, making early detection essential for effective treatment. Mammography is the primary screening modality; however, accurate delineation of suspicious lesions remains challenging and subject to inter-observer variability. Automated segmentation methods can assist radiologists by providing consistent and efficient lesion localization. This study presents DSU-Net, an attention-enhanced Dense Skip U-Net architecture for automated breast lesion segmentation in mammographic images. The proposed framework integrates dense skip connections and attention mechanisms to improve feature propagation, preserve spatial information, and enhance lesion boundary delineation. Experiments were conducted using the Curated Breast Imaging Subset of the Digital Database for Screening Mammography (CBIS-DDSM). To address severe foreground-background imbalance, a composite loss function combining Dice loss, focal loss, and binary cross-entropy loss was employed during training. The proposed model achieved a Dice Similarity Coefficient of 0.9421, an Intersection over Union of 0.8905, an accuracy of 0.9711, and an AUC-ROC of 0.9878 on the validation dataset. Qualitative evaluation demonstrated accurate delineation of lesions with varying sizes and morphologies, while quantitative results confirmed robust discrimination between lesion and background regions. These findings demonstrate that DSU-Net provides accurate and reliable breast lesion segmentation in mammographic images and highlights the potential of attention-guided deep learning for computer-aided breast cancer screening and diagnosis.

Reza Bozorgpour, Mohammadreza Soltany Sadrabadi

Department of Biomedical Engineering, University of Wisconsin-Milwaukee, Milwaukee, WI, USA · Department of Mechanical Engineering, Northern Arizona University, Flagstaff, AZ, USA

Medical Imaging U-Net Medical Image Segmentation Mammography

Ask PDF

Apr 21, 2026

Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms

Vision Transformers

(\texttt{ViT})

have become the architecture of choice for many computer vision tasks, yet their performance in computer-aided diagnostics remains limited. Focusing on breast cancer detection from mammograms, we identify two main causes for this shortfall. First, medical images are high-resolution with small abnormalities, leading to an excessive number of tokens and making it difficult for the softmax-based attention to localize and attend to relevant regions. Second, medical image classification is inherently fine-grained, with low inter-class and high intra-class variability, where standard cross-entropy training is insufficient. To overcome these challenges, we propose a framework with three key components: (1) Region of interest

(\texttt{RoI})

based token reduction using an object detection model to guide attention; (2) contrastive learning between selected

\texttt{RoI}

to enhance fine-grained discrimination through hard-negative based training; and (3) a

\texttt{DINOv2}

pretrained

\texttt{ViT}

that captures localization-aware, fine-grained features instead of global

\texttt{CLIP}

representations. Experiments on public mammography datasets demonstrate that our method achieves superior performance over existing baselines, establishing its effectiveness and potential clinical utility for large-scale breast cancer screening. Our code is available for reproducibility here: https://aih-iitd.github.io/publications/attend-what-matters

Samyak Sanghvi, Piyush Miglani, Sarvesh Shashikumar +3

Department of Computer Science and Engineering, IIT Delhi, New Delhi, India · Yardi School of AI, IIT Delhi, New Delhi, India · Department of Radiodiagnosis, PGIMER Chandigarh, Chandigarh, India

Medical Imaging Vision Foundation Model Adaptation Medical Image Classification Mammography Oncology

Ask PDF