Label-Space Mismatch Distorts Cross-Cohort Evaluation in Breast Cancer Molecular Subtyping
Authors: Suxing Liu, Byungwon Min, Zhijian Zheng, Jiahao Wang
Organizations: Department of IT Engineering, Mokwon University, Daejeon 35349, Republic of Korea · School of Digital Media Technology, Jiangxi Arts & Ceramics Technology Institute, Jingdezhen 333001, China · University of Alabama, Tuscaloosa, AL 35487, USA · School of Computer Sciences, Universiti Sains Malaysia, Minden Heights, Penang 11800, Malaysia
Cross-cohort evaluation of breast cancer subtype classifiers is unreliable when source and target cohorts use different annotation standards: METABRIC labels six classes, PAM50 defines five, and a claudin-low prediction cannot be correct under a five-class standard. A classifier transferred from 1,369 METABRIC patients (train n=958) to TCGA-BRCA (n=987) and GSE21653 (n=266) separates three evaluation operations routinely conflated in cross-site comparisons. (1) Label-space restriction - an evaluation-rule change, not a retraining gain - is the dominant correction (73.76%->79.74% on TCGA-BRCA; 63.16%->68.42% on GSE21653); (2) EM prior correction on the reconciled space is an optional calibration step (Brier fidelity up, macro F1 down; to 80.65% and 74.06%); and (3) the target label space is not recoverable from unlabelled predictions (EM retains prior 0.0751 on the absent class; calibrated detectors fail), so compliance must come from cohort documentation. The protocol exposes why internal validation overstates cross-cohort value: multimodal fusion gains do not transfer (no configuration beats mRNA alone, n=694; McNemar p=0.13-1.00), and the genomic panel adds nothing to pathological complete response prediction over clinical variables (DeltaAUC = +0.0045). Clinically, without label-space compliance, cross-site accuracy comparisons misattribute annotation-standard differences to model skill; Luminal A/B and claudin-low boundary errors map to distinct treatment decisions; and subtype accuracy is a weaker portable endpoint than survival discrimination (external C-index 0.577-0.584). Reporting the cohort's allowed label set and applying label-space compliance are therefore prerequisites before any cross-cohort performance claim.
Figures & tables
Modality Combination
Dim
Val Acc (%)
5-fold CV (%)
Clinical only
69
63.9
64.1 ± 2.7
mRNA only
412
74.6
73.8 ± 1.6
Methylation only
232
56.6
54.7 ± 2.4
Clinical + mRNA
481
75.6
74.5 ± 1.9
Clinical + Methylation
301
68.3
65.7 ± 1.9
mRNA + Methylation
644
74.6
73.1 ± 2.6
TABLE I: Modality-combination ablation on the METABRIC validation set (n=205). Five-fold cross-validation results use the training subset (n=958). The clinical + mRNA + methylation model exceeds the all-modality model by 3.4 percentage points on the validation split.
Model
CV Accuracy (%)
Logistic Regression ( C=0.01 )
75.3
Logistic Regression ( C=0.1 )
74.1
Logistic Regression ( C=1.0 )
72.9
Calibrated LogReg (sigmoid)
74.4
Random Forest (300 trees)
72.4
XGBoost
72.0
TABLE II: Model comparison using five-fold cross-validation on the METABRIC training subset (n=958). Models use the selected clinical + mRNA + methylation feature set (713 dimensions), unless otherwise stated.
Subtype
Precision
Recall
F1-score
Support
Luminal A
0.700
0.851
0.768
74
Luminal B
0.812
0.780
0.796
50
HER2-enriched
0.741
0.769
0.755
26
Claudin-low
0.733
0.579
0.647
19
Basal-like
0.778
0.667
0.718
21
Normal-like
0.625
0.312
0.417
16
TABLE III: Per-class classification metrics on the METABRIC test set (n=206).
Fig. 1 : Confusion matrix for the six-class PAM50 prediction task on the held-out METABRIC test set (n=206) for the 3-modality logistic regression (Clinical + mRNA + Methylation, 713 dimensions; accuracy 73.8%). Cell values are predicted counts with row percentages. Luminal A/Luminal B confusion dominates the error budget.
Fig. 2 : External validation on TCGA-BRCA (n=987). The within-platform rank-normalized mRNA model reaches 73.8% accuracy and a survival C-index of 0.577 ( p =0.047).
Configuration
Dims
METABRIC CV
METABRIC test
TCGA
TCGA macro F1
p
mRNA
412
72.91
72.82
80.55
0.768
—
mRNA + clinical
422
73.43
75.24
80.40
0.774
1.00
mRNA + methylation
644
72.91
72.33
78.67
0.747
0.13
clinical + mRNA + methylation
654
73.17
74.76
79.11
0.760
0.30
original flagship
713
75.92
73.79
n/a
n/a
—
TABLE IV: Modality ablation in the harmonized three-layer space shared by METABRIC and TCGA-BRCA. CV is five-fold cross-validated accuracy on pooled METABRIC train+val (n=1,163). TCGA accuracy is on n=694 patients annotated with one of the five PAM50 classes present in that cohort. The McNemar column is a paired test against mRNA alone. The final row is the original 713-dimension model, shown for reference: its clinical block includes METABRIC-specific fields and has no external counterpart.
Fig. 3 : The multimodal advantage does not transfer. Panel A: internal cross-validated and held-out METABRIC accuracy for each configuration, with the original 713-dimension flagship as a reference line. Panel B: TCGA-BRCA accuracy over the five PAM50 classes present (n=694) with bootstrap 95% confidence intervals. Every configuration’s interval overlaps the mRNA-only interval; the harmonized clinical block helps internally and is negligible externally.
TCGA-BRCA (n=987)
GSE21653 (n=266)
Estimator
Acc (%)
Macro F1
Acc (%)
Macro F1
base (no reconciliation)
73.76
0.585
63.16
0.504
mask_only
79.74
0.757
68.42
0.626
em_pam50
80.65
0.737
74.06
0.683
em_full (no mask)
75.99
0.586
68.42
0.541
TABLE V: Decomposition of the cross-cohort correction. Within each cohort, all rows share one classifier and feature set, using logistic regression ( C =0.01) and within-platform rank normalization. TCGA-BRCA uses 412 genes; GSE21653 uses the 369 genes mapped to GPL570, with a source model refitted on those genes. Rows differ only in label-space handling. mask_only uses the annotation-standard fact that claudin-low is not a PAM50 subtype; em_pam50 adds EM prior correction on the reconciled space; em_full runs EM without label-space knowledge. Neither the mask nor any row uses target labels.
Fig. 4 : Accuracy and macro F1 for the same mRNA-only estimator across METABRIC validation, METABRIC test, and TCGA-BRCA. TCGA accuracy is similar to the internal estimates, while the six-class macro F1 is affected by the absent claudin-low class.
Estimator
CV acc
Base acc
Mask acc
Δ
Logistic Regression ( C =0.01)
72.9%
73.34%
80.55 %
+7.20
Random Forest (300 trees)
70.4%
75.36%
78.82 %
+3.46
XGBoost (300 trees)
72.2%
72.05%
77.95 %
+5.91
TABLE VI: Mask-only label-space correction is effective across estimator families on TCGA-BRCA (harmonised mRNA, n =694). Base: argmax over the full six-class space. Mask: the absent claudin-low class is dropped from the candidate set and the remaining five re-argmaxed. All estimators are trained on METABRIC train+val ( n =1,163).
Detector
Micro P
Micro R
F1
Exact
Null FP
Uncalibrated EM ( em_orig )
0.920
0.685
0.786
0.550
17/120
Calibrated EM ( em_cal )
0.577
0.876
0.696
0.362
73/120
Importance-weighted EM ( em_cal_iw )
0.540
0.544
0.542
0.242
56/120
Probability frequency ( pfreq )
0.548
0.902
0.682
0.372
87/120
Geometric coherence ( geo )
0.467
0.154
0.232
0.193
27/120
Never predicted ( naive )
1.000
0.006
0.012
0.167
0/120
TABLE VII: Absent-class detectors scored with the standard empty-set convention (20 seeds × 6 absent sets × 3 covariate-shift levels × 2 prior-shift settings = 720 cohorts). Null false positives count cohorts with no absent class on which the detector returned a non-empty set. The naïve exact-match rate of 0.167 is the fraction of settings in which nothing was in fact absent.
Cohort
Estimator
Acc (%)
Macro F1
ECE
Brier
TCGA-BRCA (n=987)
6-class base
73.76
0.585
0.0352
0.3756
6-class mask_only
79.74
0.757
0.0557
0.2906
6-class em_pam50
80.65
0.737
0.0460
0.2745
5-class drop (n=867)
80.04
0.774
0.0560
0.2892
5-class merge
76.90
0.695
0.0344
0.3284
TABLE VIII: Five-class training baselines against the six-class estimator with label-space reconciliation. All rows use one rank-normalised mRNA feature set and logistic regression ( C=0.01 ); rows differ only in training label space and post-hoc handling. Paired McNemar comparisons against the 6-class base are reported in the text; where p≥0.05 the difference is not statistically distinguishable and is described as such rather than as an advantage.
Fig. 5 : Label-space reconciliation and prior correction. (A) Accuracy before and after label-space reconciliation on TCGA-BRCA and GSE21653, showing the mask-only and EM-corrected stages separately. (B) The identifiability failure: unconstrained EM retains substantial prior mass on claudin-low, whose true prevalence in both cohorts is zero, and recovers only a fraction of the available gain. (C) Controlled simulation separating label-space mismatch from prior shift; the correction is near zero without prior shift and consistently large with it.
Predicted Subtype
HR
95% CI
p-value
Luminal B
1.50
[1.14, 1.97]
<0.005
HER2-enriched
1.43
[1.08, 1.89]
0.01
Basal-like
1.46
[1.07, 1.99]
0.02
Claudin-low
0.82
[0.58, 1.16]
0.25
Normal-like
1.31
[0.86, 2.00]
0.21
TABLE IX: Cox regression hazard ratios for predicted subtypes (Luminal A = reference) on METABRIC (n=411).
Fig. 6 : SHAP-based modality attribution. mRNA expression accounts for 63.3% of model decisions, methylation 26.7% and clinical features 10.0%; copy-number and mutation blocks contribute under 1% each.
Treated subgroup
n
Events
AUC (panel)
AUC (subtype)
Δ
Chemotherapy
275
120
0.624
0.571
+0.053
Hormone therapy
768
210
0.679
0.577
+0.102
Radiotherapy
767
225
0.718
0.640
+0.078
TABLE X: 5-year outcome discrimination within treatment subgroups, genomic panel versus PAM50 subtype alone. These are administered-treatment cohorts, not response assessments.
Fig. 7 : Treatment-response prediction in GSE20194 (n=278; 56 pCR, 222 residual disease). (A) Discrimination of pCR after neoadjuvant chemotherapy. The genomic panel does not improve on clinical variables: the increment is +0.0045 with a 95% CI spanning zero, and the genomic-only model (0.773) is worse than clinical-only (0.791). (B) pCR rate by predicted subtype. Despite adding no predictive value over clinical features, the predicted subtype reproduces the observed response pattern, with pCR rates of 53.6% for HER2-enriched and 42.4% for basal-like compared with 6.0% for Luminal A.
Clinical risk prediction models often underperform in real-world settings due to poor calibration, limited transportability, and subgroup disparities. These challenges are amplified in high-dimensional multimodal cancer datasets characterized by complex feature interactions and a p >> n structure. We present a fully reproducible multimodal machine learning framework for 5-year overall survival prediction in breast cancer, integrating clinical variables with high-dimensional transcriptomic and copy-number alteration (CNA) features from the METABRIC cohort. After variance- and sparsity-based filtering and dimensionality reduction, models were trained using stratified train/validation/test splits with validation-based hyperparameter tuning. Two survival approaches were compared: an elastic-net regularized Cox model (CoxNet) and a gradient-boosted survival tree model implemented using XGBoost. CoxNet provides embedded feature selection and stable estimation, whereas XGBoost captures nonlinear effects and higher-order interactions. Performance was assessed using time-dependent area under the ROC curve (AUC), average precision (AP), calibration curves, Brier score, and bootstrapped 95 percent confidence intervals. CoxNet achieved validation and test AUCs of 98.3 and 96.6, with AP values of 90.1 and 80.4. XGBoost achieved validation and test AUCs of 98.6 and 92.5, with AP values of 92.5 and 79.9. Fairness diagnostics showed stable discrimination across age groups, estrogen receptor status, molecular subtypes, and menopausal state. This work introduces a governance-oriented multimodal survival framework emphasizing calibration, fairness auditing, robustness, and reproducibility for high-dimensional clinical machine learning.
Toktam Khatibi
Faculty of Industrial and Systems Engineering, Tarbiat Modares University, Tehran, Iran
Breast MRI is highly sensitive for detecting breast tumors, but exams contain many slices and require substantial reading time. Deep learning models often perform well on internal splits but can fail across institutions because of domain shift and dataset-origin bias. We study this failure mode for binary breast MRI tumor classification. EfficientNet-B3 and WaveViT-Small are trained using Duke Breast Cancer MRI and fastMRI, and evaluated only on the independent multi-center MAMA-MIA cohort. In a deliberately confounded setup, where label is perfectly correlated with dataset origin, external accuracy is near chance (0.5048--0.5265), despite very high recall. We then construct a mixed training set in which each class contains samples from both Duke and fastMRI, while preserving patient-level splitting, augmentation, and leakage controls. On MAMA-MIA, dataset mixing improves accuracy/F1 to 0.8463/0.8625 for WaveViT-Small and 0.8884/0.8994 for EfficientNet-B3. These results show that controlling dataset-origin bias is important for reliable breast MRI classification.
Mohammad Ali Dadrast, Hamid Usefi
Data Science Program Memorial University of Newfoundland St. John’s, Canada · Department of Mathematics and Statistics Memorial University of Newfoundland St. John’s, Canada
Breast cancer is the most frequently diagnosed malignancy among women worldwide and a leading cause of cancer-related mortality. Dynamic contrast-enhanced magnetic resonance imaging plays a central role in tumor characterization and treatment monitoring, particularly in patients receiving neoadjuvant chemotherapy. However, existing artificial intelligence models for breast magnetic resonance imaging are typically developed and evaluated using heterogeneous datasets, study populations, and assessment protocols, making direct comparison difficult and limiting understanding of model robustness across institutions and clinically relevant patient subgroups. The MAMA-MIA Challenge was designed to address these challenges by providing a standardized benchmark for the joint evaluation of primary tumor segmentation and prediction of pathologic complete response using pre-treatment magnetic resonance imaging only. The training cohort comprised 1,506 patients from multiple institutions in the United States, while evaluation was conducted on an external test set of 574 patients from three independent European centers to assess cross-continental and cross-institutional generalization. A unified scoring framework combined predictive performance with subgroup consistency across age, menopausal status, and breast density. Twenty-six international teams participated in the final evaluation phase. Results demonstrate substantial performance variability under a common external evaluation framework and reveal trade-offs between overall accuracy and subgroup fairness. The challenge provides standardized datasets, evaluation protocols, and public resources to promote the development of robust and equitable artificial intelligence systems for breast cancer imaging.