The C-index Multiverse
Organizations: MRC Human Genetics Unit University of Edinburgh Edinburgh, UK
Abstract
Quantifying out-of-sample discrimination performance for time-to-event outcomes is a fundamental step for model evaluation and selection in the context of predictive modelling. The concordance index, or C-index, is a widely used metric for this purpose, particularly with the growing development of machine learning methods. Beyond differences between proposed C-index estimators (e.g. Harrell's, Uno's and Antolini's), we demonstrate the existence of a C-index multiverse among available R and python software, where seemingly equal implementations can yield different results. This can undermine reproducibility and complicate fair comparisons across models and studies. Key variation sources include tie handling and adjustment to censoring. Additionally, the absence of a standardised approach to summarise risk from survival distributions, result in another source of variation dependent on input types. We demonstrate the consequences of the C-index multiverse when quantifying predictive performance for several survival models (from Cox proportional hazards to recent deep learning approaches) on publicly available breast cancer data, and semi-synthetic examples. Our work emphasises the need for better reporting to improve transparency and reproducibility. This article aims to be a useful guideline, helping analysts when navigating the multiverse, providing unified documentation and highlighting potential pitfalls of existing software. All code is publicly available at: www.github.com/BBolosSierra/CindexMultiverse.
Figures & tables
| Package/Function | Language | Tied Times | Tied Predictions | IPCW |
|---|---|---|---|---|
| Implementations that estimate | ||||
| Hmisc / rcorr.cens 1 | R | ✓ | ✓ ∗ | ✗ |
| SurvMetrics / Cindex 2 | R | ✓ | ✓ | ✗ |
| lifelines / concordance_index 3 | Python | ✓ | ✓ | ✗ |
| pysurvival / concordance_index 4 | Python | ✓ | ✓ ∗ | ✓ |
| scikit-survival (sksurv) / | Python | ✓ | ✓ | ✗ |
| Covariate | Censored (N = 812) | Event (N = 1125) |
|---|---|---|
| MKI67 | 5.83 (0.35) | 5.90 (0.33) |
| EGFR | 6.26 (0.82) | 6.17 (0.88) |
| ERBB2 | 10.64 (1.35) | 10.85 (1.36) |
| PGR | 6.31 (1.06) | 6.19 (0.99) |
| Hormone Therapy | ||
| No | 319 (39%) | 413 (37%) |
| Model | Package | Rounding | C-index (ties excluded) | C-index (ties included) |
|---|---|---|---|---|
| CPH | pec | No | 0.6346 [0.6112, 0.6588] | 0.6346 [0.6113, 0.6589] |
| pec | Yes | 0.6351 [0.6114, 0.6595] | 0.6348 [0.6113, 0.6593] | |
| pycox | No | 0.6317 [0.6047, 0.6544] | 0.6318 [0.6053, 0.6548] | |
| pycox | Yes | 0.6298 [0.6018, 0.6519] | 0.6294 [0.6020, 0.6516] | |
| RSF | pec | No | 0.6395 [0.6170, 0.6643] | 0.6395 [0.6171, 0.6644] |
| pec | Yes | 0.6400 [0.6173, 0.6646] | 0.6397 [0.6172, 0.6644] |
| Hmisc | Survmetrics | lifelines | pysurvival | sksurv (cens.) | SurvivalEVAL (Harrell / Naive) | ||||||||||
| Case | Time | ( ) | Predictions | CP | CN | CP | CN | CP | CN | CP | CN | CP | CN | CP | CN |
| 1A | (1,1) | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | N/T/R/A: ✓ | N/T/R/A: 1 | ||
| 1B | (1,1) | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | N/T/R/A: ✓ | N/T/R/A: 0 | ||
| 1C | (1,1) | ✓ (1) | 0.5 | ✓ | 0.5 | ✓ | 0.5 | ✓ | 0.5 (1) | ✓ | 0.5 (*) | N/T: ✗; R/A: ✓ | N/T: –; R/A: 0.5 | ||
| 2A | (1,0) | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | N/T/R/A: ✓ | N/T/R/A: 1 | ||
| 2B | (1,0) | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | N/T/R/A: ✓ | N/T/R/A: 0 | ||
| pec | survC1 | survival (”n”) | survival (”n/G2”) | sksurv (ipcw) | SurvivalEVAL (Uno/IPCW) | ||||||||||
| Case | Time | ( ) | Rank | CP | CN | CP | CN | CP | CN | CP | CN | CP | CN | CP | CN |
| 1A | (1,1) | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | N/T/R/A: ✓ | N/T/R/A: 1 | ||
| 1B | (1,1) | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | N/T/R/A: ✓ | N/T/R/A: 0 | ||
| 1C | (1,1) | ✓ (1) | 0.5 | ✓ | 0.5 | ✓ | 0.5 | ✓ | 0.5 | ✓ | 0.5 | N/T: ✗; R/A: ✓ | N/T: –; R/A: 0.5 | ||
| 2A | (1,0) | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | ✓ | 1 | N/T/R/A: ✓ | N/T/R/A: 1 | ||
| 2B | (1,0) | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | ✓ | 0 | N/T/R/A: ✓ | N/T/R/A: 0 | ||
| pycox.Ant | pycox.Adj.Ant | SurvivalEVAL | |||||||
| Case | Time | ( ) | Rank | CP | CN | CP | CN | CP | CN |
| 1A | (1,1) | ✓ | 1 | ✓ | 1 | N/T/R/A: ✓ | N/T/R/A: 1 | ||
| 1B | (1,1) | ✓ | 0 | ✓ | 0 | N/T/R/A: ✓ | N/T/R/A: 0 | ||
| 1C | (1,1) | ✓ | 0 | ✓ | 0.5 | N/T: ✗; R/A: ✓ | N/T: –; R/A: 0.5 | ||
| 2A | (1,0) | ✓ | 1 | ✓ | 1 | N/T/R/A: ✓ | N/T/R/A: 1 | ||
| 2B | (1,0) | ✓ | 0 | ✓ | 0 | N/T/R/A: ✓ | N/T/R/A: 0 | ||
| Package / Function | IPCW | Definition of |
| Implementations that estimate | ||
| Hmisc / rcorr.cens | ✗ | - |
| SurvMetrics / Cindex | ✗ | - |
| lifelines / concordance_index | ✗ | - |
| pysurvival / concordance_index | ✓ | |
| scikit-survival (sksurv) / | ✗ | - |
| tiedOutcomeIn | tiedPredIn | tiedMatchIn | 5C (CP) | 5C (CN) | 6C (CP) | 6C (CN) |
| 1 | 1 | 1 | ✓ | 1 | ✓ | 0.5 |
| 1 | 1 | 0 | ✓ | 0.5 | ✓ | 0.5 |
| 1 | 0 | 1 | ✓ | 1 | ✗ | - |
| 1 | 0 | 0 | ✗ | - | ✗ | - |
| 0 | 1 | 1 | ✓ | 1 | ✓ | 0.5 |
| 0 | 1 | 0 | ✓ | 0 | ✓ | 0.5 |
| Hyper-parameter | DeepSurv | Cox-Time | DeepHit | RSF |
|---|---|---|---|---|
| Optimizer | adam | adam | adam | - |
| Activation | selu | relu | relu | - |
| Dense Layers | 1 | 2 | 2 | - |
| Nodes / Layer | 41 | 32, 32 | 32, 32 | 100 |
| Learning Rate | 0.0103 | 0.01 | 0.001 | - |
| Dropout | 0.1601 | 0.1 | 0.6 | - |
| Fold | Train (1550 obs) | Test (387–388 obs) | ||
|---|---|---|---|---|
| Event (%) | Censor (%) | Event (%) | Censor (%) | |
| 1 | 58.45 | 41.55 | 56.59 | 43.41 |
| 2 | 58.49 | 41.51 | 56.44 | 43.56 |
| 3 | 58.17 | 41.83 | 57.73 | 42.27 |
| 4 | 58.13 | 41.87 | 57.88 | 42.12 |
| 5 | 57.16 | 42.84 | 61.76 | 38.24 |
| No rounding | Rounded to the nearest integer | |
| Number of individuals | 1937 | 1937 |
| Number of unique times | 1711 | 303 |
| Number of tied time pairs | 246 | 7150 |
| Tied time pairs (%) | 0.0131 | 0.3813 |
| No rounding | Age rounded to the nearest integer | Age binned to 5 year windows | |
| Number of individuals | 1937 | 1937 | 1937 |
| Number of unique predictions | 1858 | 296 | 68 |
| Number of tied predictions pairs | 81 | 8139 | 39314 |
| Tied prediction pairs (%) | 0.0043 | 0.4341 | 2.0967 |
| Case | Observed time | Effective time / Rank | CP (Marginal) | CN (Marginal) | Weight | |
| 1A | (1,1) | N/T/R/A: ✓ | N/T/R/A: 1 | |||
| 1B | (1,1) | N/T/R/A: ✓ | N/T/R/A: 0 | |||
| 1C | (1,1) | N/T: ✗; R/A: ✓ | N/T: –; R/A: 0.5 | |||
| 2A | (1,0) | N/T/R/A: ✓ | N/T/R/A: 1 | |||
| 2B | (1,0) | N/T/R/A: ✓ | N/T/R/A: 0 | |||
| 2C | (1,0) | N/T: ✗; R/A: ✓ | N/T: –; R/A: 0.5 |
Survival Analysis
Survival analysis models when events such as disease progression, failure, or death occur, accounting for incomplete follow-up and censoring.
See all 112 papers