Beyond AUC: Do Mammography Risk Models Predict When Cancer Occurs?
Organizations: Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University, Moscow, Russia
Abstract
Longitudinal deep learning models for mammography-based breast cancer risk prediction output either per-year cumulative risks or a discrete distribution over the year of diagnosis, from which cumulative risks F1,...,F5 are derived. These models are evaluated almost exclusively with year-wise AUCs and the concordance index. We show that these metrics are largely insensitive to when cancer occurs: they can be high even for a model that carries no information about the year of diagnosis. We re-evaluate two recent longitudinal models (OA-BreaCR, LMV-Net) on CSAW-CC with a 5-year horizon using timing metrics computed on cancer cases only. Despite C-indices up to 0.75, neither orders cancer cases by time to diagnosis significantly better than chance (c_time 0.51 and 0.55, both 95% CIs covering 0.5). Weighted absolute timing error favours a constant "year 3" prediction over both models, so we use c_time as the primary timing metric. For Mirai's additive-hazard head, also used by LMV-Net, we analyse how per-year increments passed through a ReLU can collapse under severe class imbalance. We also propose a breast-level, two-stage model that separates whether from when. It matches LMV-Net in discrimination (C-index 0.745 vs. 0.746) but orders cancer cases better (c_time 0.58 vs. 0.51, paired 95% CI of the difference 0.01-0.13), on par with Mirai's published weights (0.57).
Figures & tables
| AUC k (%) | |||||||||
| Method | C-index | 1y | 2y | 3y | 4y | 5y | c time | WMAE | WCS (%) |
| Constant (ref.) | – | – | – | – | – | – | 50.0 | 1.20 | 60.0 |
| OA-BreaCR † [ 15 ] | 69.1 (64.5–73.7) | 66.1 | 71.1 | 72.4 | 71.9 | 74.6 | 54.7 (49.6–60.1) | 1.48 (1.45–1.50) | 42.1 (40.0–45.0) |
| LMV-Net † [ 13 ] | 74.6 (70.3–78.6) | 72.3 | 76.9 | 76.2 | 76.2 | 74.5 | 50.9 (45.5–56.7) | 1.52 (1.36–1.70) | 45.9 (40.9–51.0) |
| Ours † | 74.5 (70.2–78.7) | 72.2 | 77.0 | 76.6 | 75.1 | 75.0 | 58.1 (51.9–64.2) | 1.39 (1.25–1.56) | 44.0 (38.9–49.7) |
| Stage 1 only † | 73.4 (69.4–77.5) | 69.2 | 75.2 | 75.1 | 74.4 | 75.0 | 50.0 (50.0–50.0) | 1.31 (1.31–1.31) | 40.0 (40.0–40.0) |
Event Time Prediction
Event time prediction estimates when future events will occur from event histories, time series, and contextual information.
See all 53 papers