Evaluating Federated Vision-Language and Vision-Only Models for Privacy-Aware Breast Cancer Imaging

Malithi Mithsara Wanniarachchi Kankanamge; Ning Yang; Sree Bash Debnath; Md Zobaer Islam; Abdur R. Shahid

cs.CVOct 26, 2026

Evaluating Federated Vision-Language and Vision-Only Models for Privacy-Aware Breast Cancer Imaging

Authors: Malithi Mithsara Wanniarachchi Kankanamge, Ning Yang, Sree Bash Debnath, Md Zobaer Islam, Abdur R. Shahid

Organizations: School of Computing, Southern Illinois University, 1230 Lincoln Drive, Carbondale, 62901, IL, USA. · Department of Radiation Oncology, Stritch School of Medicine, Loyola University, 1032 W. Sheridan Rd, Chicago, 60660, IL, USA. · Department of Radiology, University of Texas Southwestern Medical Center, Dallas, 75390, TX, USA.

Abstract

Artificial intelligence (AI) shows promise for breast imaging, but limited labeled data, institutional data fragmentation, and privacy concerns hinder centralized training. We investigate a privacy-aware federated learning framework combining pretrained vision-language models (VLMs) with client-level differential privacy. We compare general-purpose CLIP models, biomedical BiomedCLIP, and an ImageNet-pretrained ViT-B/16 baseline on two public breast imaging datasets, with and without differential privacy. We also evaluate a CLIP ViT-B/16 ablation that removes the text pathway. Evaluation includes patient-level ROC-AUC with bootstrap 95% confidence intervals, baseline comparisons, threshold sensitivity, inter-model agreement, and communication cost. Under federated learning, all five models achieve high normal-versus-sick discrimination (ROC-AUC: 0.940-0.990). However, acquisition year alone achieves 0.974 on the same test partition, raising concerns about confounding rather than genuine pathology recognition. Benign-versus-malignant discrimination remains uncertain (ROC-AUC: 0.504-0.637), with all confidence intervals including 0.50. Removing the text pathway preserves classification accuracy while halving communication cost. Differential privacy reduces ROC-AUC by 0.375, attributable to perturbing the full model state rather than only trainable parameters. These findings highlight the importance of evaluating confounding, predictive reliability, privacy implementation, and deployment efficiency when developing federated AI systems for breast imaging.

Figures & tables

Figure 1: Overview of the evaluated federated breast cancer classification framework, including data preparation, model training, optional client-level differential privacy, privacy-preserving aggregation, validation, and final evaluation.

Figure 2: Federated training workflows for standard FedAvg and differential privacy with update clipping and Gaussian noise.

Panel of Figure 3 — Figure 3: Data partitioning. (a, c) Class composition of the subject-level splits, before and after minority-class oversampling. (b, d) Distribution of the training partition across 20 clients under Dirichlet allocation ( $\alpha=0.5$ for BCD, $\alpha=1.0$ for breast thermography).

Panel of Figure 4 — Figure 4: Federated sampling and privacy accounting. (a) Probability of selecting at least three clients per round. (b) Accounted $\varepsilon$ against noise multiplier $\sigma$ at $\delta=10^{-5}$ .

\bar{\Delta}w_{t,k}=\Delta w_{t,k}\min\left(1,\frac{C}{\|\Delta w_{t,k}\|_{2}}\right);

Algorithm 1 Federated Small-VLM Training with Optional DP

Panel of Figure 5 — Figure 5: Patient-level ROC curves for the five evaluated backbones under federated learning (FL) and federated learning with client-level differential privacy (FL+DP). Panels (a) and (b) show the BCD normal-versus-sick task under FL and FL+DP, respectively. Panels (c) and (d) show the corresponding results for the Breast Thermography benign-versus-malignant task. Values reported in the legends denote patient-level ROC-AUC.

Panel of Figure 6 — Figure 6: Patient-level discrimination and the effect of client-level differential privacy. Panels (a) and (b) show ROC-AUC estimates with bootstrap 95% confidence intervals for BCD and Breast Thermography, respectively. The dashed vertical line denotes a chance-level AUC of 0.50. Panel (c) shows the change in patient-level ROC-AUC after introducing client-level DP with $C=1.0$ , $\sigma=1.0$ , and $\delta=10^{-5}$ . Solid lines represent BCD, and dashed lines represent breast thermography.

Panel of Figure 7 — Figure 7: Diagnostic analysis of model behavior. Panel (a) reports pairwise Spearman agreement between the predictions of the five evaluated backbones. Panel (b) shows the sensitivity of image-level macro F1-score to the decision threshold for the BCD federated models; the dashed vertical line indicates the default threshold of 0.5. Panels (c) and (d) show the distributions of predicted sick probabilities for normal and sick BCD cases under federated learning (FL) and federated learning with client-level differential privacy (FL+DP), respectively.

Figure 8: Patient-level ROC-AUC versus client-to-server communication traffic for federated configurations with available communication measurements. Marker color identifies the model backbone, while marker shape distinguishes the BCD and Breast Thermography datasets.

Figure 9: Representative BCD predictions under FL and FL+DP across the five evaluated backbones.

Figure 10: Representative breast thermography predictions under FL and FL+DP across the five evaluated backbones.

Setting	Model	AUC	95% CI	$p$	F1 0.5	F1 ${}_{t^{\ast}}$	Sens.	Spec.
FL	OpenAI CLIP ViT-B/16	0.979	[0.94, 1.00]	$<$ 0.001	0.902	0.868	0.79	1.00
	OpenAI CLIP ViT-B/32	0.940	[0.87, 0.99]	$<$ 0.001	0.385	0.867	1.00	0.07
	BiomedCLIP ViT-B/16	0.990	[0.97, 1.00]	$<$ 0.001	0.882	0.887	0.75	1.00
	LAION CLIP ViT-B/32	0.987	[0.96, 1.00]	$<$ 0.001	0.943	0.923	0.96	0.93
	ViT-B/16 (vision only)	0.970	[0.93, 1.00]	$<$ 0.001	0.882	0.849	0.75	1.00
FL+DP	OpenAI CLIP ViT-B/16	0.658	[0.50, 0.81]	0.027	0.589	0.566	0.46	0.72

Table 1: Performance of the federated small VLMs on the BCD dataset (normal vs. sick; 53 test images, 52 patients; seed 42). AUC is patient-level with bootstrap 95% confidence intervals;

p

from a one-sided Mann-Whitney test. Macro F1 is reported at the default 0.5 boundary and at the validation-selected threshold

t^{\ast}

; sensitivity and specificity are reported at the 0.5 boundary.

Setting	Model	AUC	95% CI	$p$	F1 0.5	F1 ${}_{t^{\ast}}$	Sens.	Spec.
FL	OpenAI CLIP ViT-B/16	0.504	[0.24, 0.78]	0.500	0.490	0.490	0.15	0.91
	OpenAI CLIP ViT-B/32	0.526	[0.29, 0.78]	0.429	0.485	0.485	0.59	0.42
	BiomedCLIP ViT-B/16	0.570	[0.33, 0.81]	0.296	0.486	0.486	0.11	0.98
	LAION CLIP ViT-B/32	0.563	[0.30, 0.81]	0.317	0.499	0.538	0.26	0.76
	ViT-B/16 (vision only)	0.637	[0.39, 0.87]	0.142	0.637	0.519	0.59	0.69
FL+DP	OpenAI CLIP ViT-B/16	0.422	[0.19, 0.67]	0.744	0.428	0.444	0.11	0.82

Table 2: Performance on the Breast Thermography dataset (benign vs. malignant; 72 test images, 24 patients, three views per patient; seed 42). AUC is patient-level with bootstrap 95% confidence intervals;

p

from a one-sided Mann-Whitney test. Macro F1 is reported at the default 0.5 boundary and at the validation-selected threshold

t^{\ast}

; sensitivity and specificity are reported at the 0.5 boundary. All confidence intervals include 0.50, indicating that the available test cohort does not provide statistically conclusive evidence of discrimination above chance for any evaluated configuration.

Dataset	Model	AUC FL	AUC FL+DP	$\Delta$	$\rho_{\mathrm{FL,DP}}$
BCD	OpenAI CLIP ViT-B/16	0.979	0.658	$-0.321$	$+0.403$
	OpenAI CLIP ViT-B/32	0.940	0.486	$-0.454$	$-0.004$
	BiomedCLIP ViT-B/16	0.990	0.507	$-0.483$	$+0.091$
	LAION CLIP ViT-B/32	0.987	0.556	$-0.430$	$-0.019$
	ViT-B/16 (ImageNet)	0.970	0.786	$-0.184$	$+0.486$
	Mean			$\mathbf{-0.401}$

Table 3: Change in patient-level ROC-AUC after adding client-level differential privacy (

C=1.0

\sigma=1.0

\delta=10^{-5}

\rho_{\mathrm{FL,DP}}

denotes the Spearman correlation between the predictions of the corresponding FL and FL+DP models, and the mean is taken over the five backbones. On BCD, the evaluated DP configuration substantially reduces ranking performance. On breast thermography, the observed AUC changes are small relative to the uncertainty of the patient-level evaluation.

Dataset	Setting	Mean $\rho$	Min	Max	Pairs $\rho>0$
BCD	Federated	0.860	$+0.80$	$+0.92$	10/10
BCD	Federated + DP	0.018	$-0.28$	$+0.34$	5/10
Thermography	Federated	0.397	$+0.17$	$+0.56$	10/10
Thermography	Federated + DP	0.002	$-0.14$	$+0.16$	5/10

Table 4: Pairwise Spearman correlation between the predictions of the five architectures (10 pairs per group). Under federated training the models converge on a common ranking; under DP at

C=1.0

that agreement disappears on both datasets.

Model	AUC	Traffic (GB)	Time (s)	AUC	Traffic (GB)	Time (s)
	BCD			Thermography
OpenAI CLIP ViT-B/16	0.979	301	389	0.504	362	510
OpenAI CLIP ViT-B/32	0.940	304	371	0.526	366	519
BiomedCLIP ViT-B/16	0.990	394	484	0.570	474	633
LAION CLIP ViT-B/32	0.987	304	376	0.563	366	474
ViT-B/16 (vision only)	0.970	173	290	0.637	208	379

Table 5: Communication and computation cost of the federated configurations, with patient-level ROC-AUC reported for reference.

Dataset	Configuration	AUC	95% CI	Traffic (GB)
BCD	CLIP ViT-B/16	0.979	[0.94, 1.00]	301
BCD	without text encoder	0.993	[0.97, 1.00]	173
Thermography	CLIP ViT-B/16	0.504	[0.24, 0.78]	362
Thermography	without text encoder	0.585	[0.33, 0.83]	209

Table 6: Effect of removing the text encoder from CLIP ViT-B/16. Both configurations use the same pretrained image encoder and visual projection; the vision-only variant replaces the image–text similarity term with a linear classification head. AUC is patient-level with bootstrap 95% confidence intervals.

Medical Imaging

Medical imaging uses computational methods to analyze, reconstruct, and interpret images for diagnosis, treatment planning, and monitoring.

See all 251 papers

Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms
Apr 21, 2026Samyak Sanghvi, Piyush Miglani, Sarvesh Shashikumar +3Medical Imaging Vision Foundation Model Adaptation
Department of Computer Science and Engineering, IIT Delhi, New Delhi, India · Yardi School of AI, IIT Delhi, New Delhi, India · Department of Radiodiagnosis, PGIMER Chandigarh, Chandigarh, India
Similarity Weighted Aggregation with Global Differential Privacy for Federated Brain Lesion Segmentation
Aug 1, 2026Muhammad Irfan Khan, Eero Lehtonen, Joni Obradovic +4Federated Learning Aggregation Non-IID Federated Learning
Turku University of Applied Sciences, Turku 20520, Finland
OmniMed-FL: A Robust Multimodal Federated Learning Framework for Clinical Diagnosis
Sep 9, 2026Ayush Debnath, Ruelia Saha, Sudip MisraMultimodal Federated Learning Medical Diagnosis
Indian Institute of Technology Kharagpur, India · KTH Royal Institute of Technology, Sweden · SRM University-AP, India +1

Apr 21, 2026J/K move · Enter open · S save

Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms

Samyak Sanghvi, Piyush Miglani, Sarvesh Shashikumar, Kaustubh R Borgavi+2

Department of Computer Science and Engineering, IIT Delhi, New Delhi, India · Yardi School of AI, IIT Delhi, New Delhi, India · Department of Radiodiagnosis, PGIMER Chandigarh, Chandigarh, India

Medical Imaging Vision Foundation Model Adaptation Medical Image Classification Mammography Oncology Mammography Image Classification

Open paper AskPDF arXiv

Vision Transformers

(\texttt{ViT})

have become the architecture of choice for many computer vision tasks, yet their performance in computer-aided diagnostics remains limited. Focusing on breast cancer detection from mammograms, we identify two main causes for this shortfall. First, medical images are high-resolution with small abnormalities, leading to an excessive number of tokens and making it difficult for the softmax-based attention to localize and attend to relevant regions. Second, medical image classification is inherently fine-grained, with low inter-class and high intra-class variability, where standard cross-entropy training is insufficient. To overcome these challenges, we propose a framework with three key components: (1) Region of interest

(\texttt{RoI})

based token reduction using an object detection model to guide attention; (2) contrastive learning between selected

\texttt{RoI}

to enhance fine-grained discrimination through hard-negative based training; and (3) a

\texttt{DINOv2}

pretrained

\texttt{ViT}

that captures localization-aware, fine-grained features instead of global

\texttt{CLIP}

representations. Experiments on public mammography datasets demonstrate that our method achieves superior performance over existing baselines, establishing its effectiveness and potential clinical utility for large-scale breast cancer screening. Our code is available for reproducibility here: https://aih-iitd.github.io/publications/attend-what-matters

Abstract

Medical Imaging

Explore similar work

Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms

Similarity Weighted Aggregation with Global Differential Privacy for Federated Brain Lesion Segmentation

OmniMed-FL: A Robust Multimodal Federated Learning Framework for Clinical Diagnosis