The MAMA-MIA Challenge: Advancing Generalizability and Fairness in Breast MRI Tumor Segmentation and Treatment Response Prediction
Authors: Lidia Garrucho, Smriti Joshi, Kaisar Kushibar, Richard Osuala, Maciej Bobowicz, Xavier Bargalló, Paulius Jaruševičius, Kai Geissler, Raphael Schäfer, Muhammad Alberb, Tony Xu, Anne Martel, Daniel Sleiman, Navchetan Awasthi, Hadeel Awwad, Joan C. Vilanova, Robert Martí, Daan Schouten, Jeong Hoon Lee, Mirabela Rusu, Eleonora Poeta, Luisa Vargas, Eliana Pastor, Maria A. Zuluaga, Jessica Kächele, Dimitrios Bounias, Alexandra Ertl, Katarzyna Gwoździewicz, Maria-Laura Cosaka, Pasant M. Abo-Elhoda, Sara W. Tantawy, Shorouq S. Sakrana, Norhan O. Shawky-Abdelfatah, Amr Muhammad Abdo-Salem, Androniki Kozana, Eugen Divjak, Gordana Ivanac, Katerina Nikiforaki, Michail E. Klontzas, Rosa García-Dosdá, Meltem Gulsun-Akpinar, Oğuz Lafcı, Carlos Martín-Isla, Oliver Díaz, Laura Igual, Karim Lekadir
Breast cancer is the most frequently diagnosed malignancy among women worldwide and a leading cause of cancer-related mortality. Dynamic contrast-enhanced magnetic resonance imaging plays a central role in tumor characterization and treatment monitoring, particularly in patients receiving neoadjuvant chemotherapy. However, existing artificial intelligence models for breast magnetic resonance imaging are typically developed and evaluated using heterogeneous datasets, study populations, and assessment protocols, making direct comparison difficult and limiting understanding of model robustness across institutions and clinically relevant patient subgroups. The MAMA-MIA Challenge was designed to address these challenges by providing a standardized benchmark for the joint evaluation of primary tumor segmentation and prediction of pathologic complete response using pre-treatment magnetic resonance imaging only. The training cohort comprised 1,506 patients from multiple institutions in the United States, while evaluation was conducted on an external test set of 574 patients from three independent European centers to assess cross-continental and cross-institutional generalization. A unified scoring framework combined predictive performance with subgroup consistency across age, menopausal status, and breast density. Twenty-six international teams participated in the final evaluation phase. Results demonstrate substantial performance variability under a common external evaluation framework and reveal trade-offs between overall accuracy and subgroup fairness. The challenge provides standardized datasets, evaluation protocols, and public resources to promote the development of robust and equitable artificial intelligence systems for breast cancer imaging.
Figures & tables
MAMA-MIA Dataset
Training
Validation / test
Total
Collection
NACT
ISPY1
ISPY2
DUKE
Sum
GUM
KAU
HCB
Sum
Train+Val+Test
Country
USA
USA
USA
USA
Poland
Lithuania
Spain
Year
2002–06
2010–16
2000–14
1995–02
2010–20
2010–20
2010–20
Clinical variables
pCR Yes / No / N.A.
11 / 53 / 0
49 / 118 / 4
316 / 664 / 0
64 / 216 / 11
440 / 1051 / 15
11 / 19 / –
71 / 161 / –
78 / 234 / –
160 / 414 / –
600 / 1465 / 15
Tumor subtype Lum / TN / HER2enr HER2pure / N.A.
21 / 11 / 8 6 / 18
67 / 45 / 25 29 / 5
536 / 358 / 86 0 / 0
123 / 85 / 50 30 / 3
747 / 499 / 169 65 / 26
11 / 9 / 3 7 / –
95 / 71 / 39 27 / –
209 / 43 / 49 11 / –
315 / 123 / 91 45 / –
1062 / 622 / 260 110 / 26
TABLE I : Clinical and imaging variables across the MAMA-MIA benchmark datasets [ 31 ] . Four cohorts are used for training (NACT, ISPY1, ISPY2, DUKE) and three for validation/test (GUM, KAU, HCB). Data cells with more than 3 items are split into two rows to maximize readability. Best viewed in color.
❶ MIC
❷ FME
❸ ViCOROB
❹ Martel Lab
❺ AIH-MAMA
Preprocessing Strategy
resample
third-order spline 1×1×1
nnUNet pre 0.7×0.7×2.0
b-spline 1×1×1
trilinear 1×1×1
bilinear 1×1×1
crop or pad
×
×
breast bbox
breast bbox
non-zero voxels 320×320×128
reorient
×
×
×
RAS
RAS
intensity clipping
×
×
percentile [0.1,99.9]
percentile [0.05,99.95]
×
normalize
z-score
z-score
z-score
min-max
z-score (non-zero pixels only)
TABLE II : Technical summary of top-performing primary tumor segmentation methods (Task 1). Abbreviations used in the table- nnUNet pre: default nnUNet [ 13 ] preprocessing strategy; RAS: Right-Anterior-Superior orientation; p0/p1/p2: pre-contrast, first post-contrast and second post-contrast phases; px: last available post-contrast phase; s1: subtraction of first post-contrast phase with pre-contrast phase; bbox: Bounding Box; CE: Cross Entropy; CV: Cross Validation; n/a: not applicable/available.
① pimed-lab
② FME
③ AI Strollers
Preprocessing
resample
1×1×1
varying per model
nnUNet pre [ 13 ] 0.7×0.7×2.0
crop / pad
lesion bbox
lesion bbox
×
reorient
×
×
×
clip
×
×
×
normalize
z-score
z-score
z-score
TABLE III : Technical summary of top-performing pathologic complete response (pCR) prediction methods (Task 2). Abbreviations used in the table - nnUNet pre: default nnUNet [ 13 ] preprocessing strategy; bbox: Bounding Box; p0/p1/p2: pre-contrast, first post-contrast and second post-contrast phases; px: last available post-contrast phase; BCE: Binary Cross Entropy; CE: Cross Entropy; CV: Cross Validation; n/a: not applicable/available.
Fig. 1 : Fairness plotted against performance across the two MAMA-MIA benchmark tasks. Teams with fairness-performance tradeoffs within the area indicated in green improve upon the baseline tradeoff. Note that the axis ranges are restricted to improve readability; consequently, teams falling outside these bounds are not shown.
Fig. 2 : Qualitative results for challenging cases. The columns correspond to the MRI image, the zoomed-in tumor region (Tumor Zoom), and the top-5 team predictions (predicted segmentations) overlaid with the expert annotated tumor segmentation (ground truth). Results are shown for: (a) clip artifact (successful), (b) breast implant (failure), (c) low contrast (failure), (d) small tumor (high variance between teams), and (e) non-mass enhancement (high variance between teams). Artifacts, low discriminability, and atypical morphology remain key challenges.
Rank
Team
Combined Score [95% CI] ↑
Fairness Score [95% CI] ↑
Performance Score [95% CI] ↑
DSC [95% CI] ↑
NormHD [95% CI] ↓
1
MIC
0.8858 [0.8577, 0.8969]
0.9531 [0.9051, 0.9651]
0.8185 [0.8018, 0.8350]
0.7360 [0.7160, 0.7539]
0.0990 [0.0816, 0.1172]
2
FME
0.8820 [0.8464, 0.8913]
0.9574 [0.8976, 0.9659]
0.8066 [0.7873, 0.8248]
0.7125 [0.6895, 0.7330]
0.0993 [0.0815, 0.1182]
3
ViCOROB
0.8782 [0.8485, 0.8891]
0.9482 [0.8963, 0.9643]
0.8083 [0.7913, 0.8248]
0.7182 [0.6976, 0.7381]
0.1017 [0.0859, 0.1189]
4
Martel Lab
0.8735 [0.8471, 0.8856]
0.9449 [0.8999, 0.9619]
0.8021 [0.7863, 0.8182]
0.7121 [0.6918, 0.7315]
0.1078 [0.0915, 0.1244]
5
AIH-Mama
0.8677 [0.8379, 0.8781]
0.9532 [0.8984, 0.9661]
0.7823 [0.7634, 0.8012]
0.6914 [0.6701, 0.7139] ∗
0.1268 [0.1081, 0.1452]
6
HWT@YCH
0.8655 [0.8365, 0.8789]
0.9339 [0.8843, 0.9514]
0.7971 [0.7798, 0.8143]
0.7080 [0.6872, 0.7273]
0.1138 [0.0967, 0.1324]
TABLE IV : Final leaderboard for Task 1: Primary Tumor Segmentation. Confidence intervals (95%) for all Task 1 metrics were estimated using a non-parametric percentile bootstrap ( B=1000 , seed=42). DSC, NormHD, and Performance Score CIs were obtained by resampling patients with replacement and recomputing the mean metric on each bootstrap sample. Fairness Score CIs were computed using patient-level bootstrapping with repeated group-level disparity estimation across demographic groups. Combined Score CIs were derived jointly from the same bootstrap resamples for both performance and fairness components.
Rank
Team
Combined Score [95% CI] ↑
Fairness Score [95% CI] ↓
Performance Score [95% CI] ↑
AUC [95% CI] ↑
1
pimed-lab
0.6907 [0.6364, 0.7272]
0.1150 [0.0399, 0.2292]
0.4964 [0.4853, 0.5102]
0.4556 [0.4035, 0.5121]
2
FME
0.6642 [0.4893, 0.6602]
0.2121 [0.2232, 0.5561]
0.5405 [0.4992, 0.5841] +
0.5722 [0.5195, 0.6305]
3
AI Strollers
0.6625 [0.4778, 0.6569]
0.2013 [0.2148, 0.5663]
0.5263 [0.4870, 0.5658]
0.5218 [0.4629, 0.5771]
4
Baseline
0.6431 [0.4600, 0.6360]
0.2179 [0.2418, 0.5701]
0.5042 [0.4551, 0.5498]
0.5031 [0.4455, 0.5551]
5
BWS-KNU
0.6355 [0.4792, 0.6325]
0.2177 [0.2205, 0.5324]
0.4886 [0.4478, 0.5314]
0.5075 [0.4513, 0.5656]
6
Martel Lab
0.6310 [0.4802, 0.6316]
0.2272 [0.2397, 0.5164]
0.4892 [0.4477, 0.5316]
0.4740 [0.4195, 0.5330]
TABLE V : Final leaderboard for Task 2: Treatment Response Prediction to Neoadjuvant Chemotherapy. Note that AUC is reported as an additional result and is excluded from the ranking scheme. Confidence intervals (95% CI) were estimated using a non-parametric percentile bootstrap ( B=1000 , seed=42). Balanced Accuracy and AUC were bootstrapped from prediction–label pairs, while Fairness and Combined Scores used patient-level resampling with repeated equalized-odds disparity computation across demographic groups. The Combined Score CI was obtained by repeating the resampling jointly for performance and fairness components.
Fig. 3 : Impact of tumor size on segmentation performance. (a) Histogram of tumor volumes ( mm3 ) on a logarithmic scale, with dashed lines indicating the 20th and 80th percentile thresholds used to categorize tumors as Small, Medium, or Large, (b) Percentage distribution of tumor size categories across the three participating centers (GUM, KAU, HCB), with boxed values showing the average Dice Similarity Coefficient (DSC) for the top 5 teams in each category. (c) Comparison of DSC distributions between the top 5 and bottom 5 performing teams across tumor size categories, with gap bars ( Δ ) indicating the difference in mean DSC. Brackets give two-sided Wilcoxon signed-rank p-values comparing the Top 5 against the Bottom 5 teams. For each patient, the mean DSC across the five top-ranked teams was compared with the mean DSC across the five bottom-ranked teams, so patients are the unit of analysis and the two arms are paired on identical cases.
Fig. 4 : Distribution of DSC across the three participating centers (GUM, KAU, HCB) for the top 5 performing teams in Task 1. Each boxplot represents the per-case DSC distribution for a team, in order of their ranking. Horizontal dashed lines indicate the average DSC across all top 5 teams for each center.
Fig. 5 : Subgroup analysis of segmentation performance. Boxplots show DSC distributions for the top 5 (blue) and bottom 5 (red) performing teams, stratified by (a) breast density, (b) age group, (c) menopausal status, and (d) Tumor Subtype. Brackets give two-sided Wilcoxon signed-rank p-values comparing the Top 5 against the Bottom 5 teams. For each patient, the mean DSC across the five top-ranked teams was compared with the mean DSC across the five bottom-ranked teams, so patients are the unit of analysis and the two arms are paired on identical cases. The performance of high-ranking methods consistently improve in comparison to the low-ranking methods. There is no substantial differences between subgroups in the challenge fairness categories. Across tumor subtype categories, the triple_negative subtype has the highest overall median DSC, while luminal category is especially challenging for the weaker models.
Fig. 6 : Fairness-versus-performance evaluation tradeoffs: Effect of λ on the ranking across tasks and participating teams.
Fig. 7 : Recall comparison between pCR (minority) and not pCR (majority) classes across participating teams. Δ denotes the recall gap, reflecting class-wise prediction bias.
Fig. 8 : Reliability diagrams for top 3 teams in pCR prediction. Predicted probabilities are grouped into quantile bins; the fraction of true positives per bin is plotted against mean predicted probability. Dashed line: perfect calibration; dotted line: class prevalence. Annotated values indicate bin sample sizes.
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
We describe the submission of team FME to the MAMA-MIA Challenge, which evaluated primary tumor segmentation and prediction of pathological complete response (pCR) from pretreatment dynamic contrast-enhanced breast MRI on an external multi-country cohort. For segmentation, we trained a five-fold residual-encoder nnU-Net ensemble using only the first post-contrast minus pre-contrast image, combined with mirroring test-time augmentation and largest-connected-component filtering. For pCR prediction, we ensembled 25 pretrained 3D video classifiers trained on lesion-centred crops from the pre-contrast and first two post-contrast volumes. FME ranked second in both tasks. The segmentation method achieved a combined performance-fairness score of 0.882, with Dice 0.713 and normalized Hausdorff distance 0.099. The pCR method achieved a combined score of 0.664, balanced accuracy of 0.541, and equalized-odds disparity of 0.212. The results indicate that subtraction-based input and ensembling support robust tumor segmentation under cross-site domain shift, whereas pCR prediction from baseline DCE-MRI alone remains limited. For the submission repository, see https://github.com/FraunhoferMEVIS/MAMA-MIA-Challenge-FME
Kai Geissler, Raphael Schäfer
Fraunhofer Institute for Digital Medicine MEVIS Bremen, Germany
Artificial intelligence (AI) has achieved remarkable success in medical imaging, but it is widely recognized that these models often perform inconsistently across real-world clinical settings. Such inconsistencies occur when patient demographics and imaging protocols vary, for example, in detecting small tumors, analyzing scans from different contrast phases, or evaluating patients of different ages or sexes. To quantify these inconsistencies, we develop a large-scale, open benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models across tumor size, location, patient subgroup, and imaging protocol. We leverage large language models (LLMs) to extract and organize subgroup information from clinical data, which makes the analysis both scalable and reproducible. Our benchmark reveals that current state-of-the-art AI models, optimized for average accuracy, perform poorly in rare or underrepresented subgroups, such as young, female African Americans. However, collecting sufficient annotated data for these rare cases is often impractical. The benchmark provides a foundation for building more reliable and robust AI models for tumor detection and highlighting the need for rigorous, subgroup-level evaluation in medical imaging and computer vision. Datasets, code
Qi Chen, Wenxuan Li, Pedro R. A. S. Bassi +14
Johns Hopkins University · German Cancer Research Center · University Hospital Basel +10
Breast MRI is highly sensitive for detecting breast tumors, but exams contain many slices and require substantial reading time. Deep learning models often perform well on internal splits but can fail across institutions because of domain shift and dataset-origin bias. We study this failure mode for binary breast MRI tumor classification. EfficientNet-B3 and WaveViT-Small are trained using Duke Breast Cancer MRI and fastMRI, and evaluated only on the independent multi-center MAMA-MIA cohort. In a deliberately confounded setup, where label is perfectly correlated with dataset origin, external accuracy is near chance (0.5048--0.5265), despite very high recall. We then construct a mixed training set in which each class contains samples from both Duke and fastMRI, while preserving patient-level splitting, augmentation, and leakage controls. On MAMA-MIA, dataset mixing improves accuracy/F1 to 0.8463/0.8625 for WaveViT-Small and 0.8884/0.8994 for EfficientNet-B3. These results show that controlling dataset-origin bias is important for reliable breast MRI classification.
Mohammad Ali Dadrast, Hamid Usefi
Data Science Program Memorial University of Newfoundland St. John’s, Canada · Department of Mathematics and Statistics Memorial University of Newfoundland St. John’s, Canada