BAPLe++: Cross-Model Transferability and Prompt-Level Defense of Backdoors in Medical Foundation Models
Organizations: Computing and Mathematical Sciences Division, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
Abstract
Medical foundation models (Med-FMs) are increasingly adapted to clinical tasks through prompt learning, which tunes a small set of prompt tokens while the backbone remains frozen. Recent work has shown that this efficiency introduces a vulnerability: jointly optimizing learnable prompts and an imperceptible noise trigger during few-shot adaptation implants a backdoor that leaves clean performance intact but steers triggered inputs to an attacker-chosen class. Since prompts are compact and readily shared, this raises two questions: whether a backdoored prompt remains effective on a different Med-FM, and whether it can be repaired without retraining the backbone. We address both in BAPLe++. First, we present the first study of cross-model transferability of prompt-based backdoors, quantified by two proposed metrics, Clean Accuracy Difference (CAD) and Backdoor Accuracy Difference (BAD). Second, we propose DePLe (Defense via Prompt Learning), which uses the infected prompt as a fixed anchor and learns a new prompt on a small clean set that preserves the clean task while diverging from the compromised embedding region. Experiments on four Med-FMs and six chest X-ray and histopathology datasets show that backdoors transfer across models, but asymmetrically and in a dataset-dependent manner. Compared with four training-free input transformations, DePLe reduces backdoor accuracy more consistently while preserving, and in several settings improving, clean accuracy. Our code can be accessed on https://github.com/Ufaqkhan/Baple-Plus-Plus
Figures & tables
| MedCLIP | BioMedCLIP | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| COVID | RSNA18 | MIMIC | COVID | RSNA18 | MIMIC | ||||||||
| Method | CA | BA | CA | BA | CA | BA | CA | BA | CA | BA | CA | BA | |
| Clean | 0.822 | – | 0.603 | – | 0.585 | – | 0.843 | – | 0.582 | – | 0.351 | – | |
| BadNets | 0.820 | 0.510 | 0.619 | 0.373 | 0.559 | 0.284 | 0.845 | 0.975 | 0.632 | 0.942 | 0.373 | 1.000 | |
| WaNet | 0.831 | 0.470 | 0.612 | 0.319 | 0.587 | 0.266 | 0.839 | 0.599 | 0.587 | 0.510 | 0.334 | 0.599 | |
| FIBA | 0.820 | 0.511 | 0.623 | 0.360 | 0.562 | 0.292 | 0.856 | 0.729 | 0.630 | 0.614 | 0.373 | 0.722 | |
| MedCLIP | BioMedCLIP | ||||||||||||
| COVID | RSNA18 | MIMIC | COVID | RSNA18 | MIMIC | ||||||||
| Defense | CA | BA | CA | BA | CA | BA | CA | BA | CA | BA | CA | BA | |
| Without Defense (BAPLe) | 0.813 | 0.996 | 0.610 | 0.995 | 0.472 | 0.991 | 0.841 | 1.000 | 0.636 | 0.999 | 0.368 | 0.996 | |
| Gaussian Blur | 0.820 0.007 | 0.599 0.397 | 0.599 0.011 | 0.518 0.477 | 0.470 0.002 | 0.556 0.435 | 0.819 0.022 | 0.914 0.086 | 0.600 0.036 | 0.733 0.266 | 0.353 0.015 | 0.821 0.175 | |
| JPEG-30 | 0.796 0.017 | 0.551 0.445 | 0.590 0.020 | 0.460 0.535 | 0.460 0.012 | 0.503 0.488 | 0.738 0.103 | 0.884 0.116 | 0.576 0.060 | 0.794 0.205 | 0.328 0.040 | 0.837 0.159 | |
| Feature Squeeze (MF-3) | 0.789 0.024 | 0.522 0.474 | 0.603 0.007 | 0.657 0.338 | 0.462 0.010 | 0.587 0.404 | 0.849 0.008 | 0.973 0.027 | 0.631 0.005 | 0.964 0.035 | 0.368 0.000 | 0.965 0.031 | |
| PLIP | QuiltNet | ||||||||||||
| Kather | PanNuke | DigestPath | Kather | PanNuke | DigestPath | ||||||||
| Defense | CA | BA | CA | BA | CA | BA | CA | BA | CA | BA | CA | BA | |
| Without Defense (BAPLe) | 0.878 | 0.982 | 0.834 | 0.964 | 0.902 | 0.968 | 0.893 | 0.955 | 0.810 | 0.905 | 0.903 | 0.820 | |
| Gaussian Blur | 0.859 0.019 | 0.662 0.320 | 0.825 0.009 | 0.864 0.100 | 0.892 0.010 | 0.887 0.081 | 0.882 0.011 | 0.543 0.412 | 0.810 0.000 | 0.833 0.072 | 0.908 0.005 | 0.711 0.109 | |
| JPEG-30 | 0.846 0.032 | 0.555 0.427 | 0.827 0.007 | 0.742 0.222 | 0.749 0.153 | 0.874 0.094 | 0.775 0.118 | 0.453 0.502 | 0.723 0.087 | 0.687 0.218 | 0.920 0.017 | 0.602 0.218 | |
| Feature Squeeze (MF-3) | 0.857 0.021 | 0.819 0.163 | 0.829 0.005 | 0.918 0.046 | 0.882 0.020 | 0.937 0.031 | 0.893 0.000 | 0.775 0.180 | 0.791 0.019 | 0.865 0.040 | 0.894 0.009 | 0.736 0.084 | |
Backdoor Attacks
Backdoor attacks implant hidden, trigger-dependent behaviors in models while aiming to preserve normal performance.
See all 141 papers