Target Recoverability under Selective Labels: A Failure Mode in ECG-Based Acute Coronary Syndrome Prediction
Organizations: UC Berkeley and UCSF · Lund University · UC Berkeley
Abstract
Medical machine learning models often learn from diagnoses observed only in patients clinicians chose to test. The biases of such selective labels are well known, but less understood is what happens when a diagnosis includes cases the model's inputs cannot detect. Because testing decisions draw on information the model does not see, these cases can appear predictable among tested patients, producing relationships that fail, and can even reverse, among the untested. We formalize this mechanism and show in simulations that it worsens as testing becomes more selective or undetectable cases more common, while models trained only on detectable cases remain informative. We empirically study electrocardiogram (ECG)-based prediction of acute coronary syndrome (ACS), which is confirmed only by invasive testing and whose subtypes differ in how clearly they appear on the ECG. Three ECG foundation models fine-tuned on revascularization, our ACS proxy, performed well among catheterized patients (AUROC 0.740-0.770) but worse than chance for 30-day major adverse cardiovascular events among untested patients (AUROC 0.381-0.478). Models trained on ECG-visible subtypes retained discrimination (AUROC 0.680-0.709), whereas a model trained on unstable angina, which can occur without ECG changes, was inversely related to risk (AUROC 0.310). Under selective labels, effective learning therefore requires prediction targets that can be inferred from the model's inputs and evaluation in the intended deployment population, which often extends beyond the patients who were tested.
Figures & tables
| Model / target | Cath revasc. AUC (95% CI) | Untested MACE AUC (95% CI) | Untested trop.-MACE AUC (95% CI) | Untested death AUC (95% CI) |
|---|---|---|---|---|
| Trained in Nightingale catheterized cohort | ||||
| ST-MEM | 0.770 (0.726–0.812) | 0.478 (0.454–0.495) | 0.497 (0.466–0.520) | 0.442 (0.428–0.457) |
| ECGFounder | 0.745 (0.696–0.787) | 0.422 (0.401–0.437) | 0.437 (0.410–0.457) | 0.396 (0.383–0.410) |
| ECGFlow | 0.740 (0.694–0.782) | 0.381 (0.361–0.395) | 0.385 (0.360–0.403) | 0.387 (0.373–0.400) |
| Trained in Chongqing angiography cohort | ||||
| ACS | 0.687 (0.662–0.710) | 0.517 (0.492–0.535) | 0.515 (0.482–0.538) | 0.523 (0.511–0.535) |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Characteristic | Train | Validation | Internal test |
|---|---|---|---|
| ECGs | 14,367 | 1,796 | 1,795 |
| Patients | 13,589 | 1,717 | 1,710 |
| ACS | 7,112 (49.5%) | 889 (49.5%) | 889 (49.5%) |
| OMI | 920 (6.4%) | 116 (6.5%) | 115 (6.4%) |
| AMI | 2,136 (14.9%) | 279 (15.5%) | 264 (14.7%) |
| STEMI | 1,151 (8.0%) | 148 (8.2%) | 143 (8.0%) |
| Characteristic | Catheterized | Untested |
|---|---|---|
| ECGs | 4,818 | 102,925 |
| Encounters | 2,437 | 65,854 |
| Patients | 2,157 | 42,671 |
| Revascularization | 796 (32.7%) | – |
| 30-d MACE | 1,210 (49.7%) | 8,362 (12.7%) |
| Troponin-MACE | 1,154 (47.4%) | 6,279 (9.5%) |
| Target | Own-label AUC (95% CI) | Composite ACS AUC (95% CI) |
|---|---|---|
| ACS | 0.648 (0.623–0.676) | 0.648 (0.623–0.676) |
| OMI | 0.866 (0.835–0.896) | 0.603 (0.576–0.631) |
| AMI | 0.842 (0.815–0.868) | 0.614 (0.588–0.642) |
| STEMI | 0.908 (0.882–0.933) | 0.614 (0.588–0.641) |
| NSTEMI | 0.737 (0.692–0.779) | 0.606 (0.578–0.633) |
| UA | 0.618 (0.590–0.643) | 0.462 (0.434–0.489) |
| Training approach | AUROC (95% CI) | Brier score (95% CI) | Log loss (95% CI) |
|---|---|---|---|
| Broad-label learning | 0.378 (0.373–0.383) | 0.325 (0.322–0.329) | 0.851 (0.843–0.859) |
| Observed-input weighting | 0.378 (0.373–0.383) | 0.325 (0.322–0.329) | 0.851 (0.843–0.859) |
| Oracle weighting | 0.638 (0.632–0.644) | 0.183 (0.182–0.185) | 0.551 (0.547–0.554) |
| Recoverable-label learning | 0.638 (0.632–0.644) | 0.214 (0.212–0.215) | 0.692 (0.683–0.702) |
| Target | Catheterized AUC (95% CI) | Untested AUC (95% CI) |
|---|---|---|
| ACS | 0.573 (0.548–0.598) | 0.517 (0.492–0.535) |
| OMI | 0.633 (0.608–0.658) | 0.709 (0.702–0.716) |
| AMI | 0.629 (0.605–0.653) | 0.687 (0.676–0.696) |
| STEMI | 0.633 (0.608–0.657) | 0.693 (0.683–0.703) |
| NSTEMI | 0.613 (0.589–0.637) | 0.680 (0.672–0.688) |
| UA | 0.402 (0.376–0.425) | 0.310 (0.296–0.321) |
Selective Labels
Selective labels studies how to learn and evaluate decision systems when outcomes are observed only for selected people, cases, or actions.
See all 13 papers