Confidence Matters: Uncertainty Quantification and Precision Assessment of Deep Learning-based CMR Biomarker Estimates Using Scan-rescan Data
Organizations: School of Biomedical Engineering & Imaging Sciences, King’s College London, London, UK · Cardio-Oncology Centre of Excellence, Royal Brompton Hospitals, London, United Kingdom · Perspectum Ltd., Oxford, UK
Abstract
The performance of deep learning (DL) methods for the analysis of cine cardiovascular magnetic resonance (CMR) is typically assessed in terms of accuracy, overlooking precision. In this work, uncertainty estimation techniques, namely deep ensemble, test-time augmentation, and Monte Carlo dropout, are applied to a state-of-the-art DL pipeline for cardiac functional biomarker estimation, and new distribution-based metrics are proposed for the assessment of biomarker precision. The model achieved high accuracy (average Dice 87%) and point estimate precision on two external validation scan-rescan CMR datasets. However, distribution-based metrics showed that the overlap between scan/rescan confidence intervals was >50% in less than 45% of the cases. Statistical similarity tests between scan and rescan biomarkers also resulted in significant differences for over 65% of the cases. We conclude that, while point estimate metrics might suggest good performance, distributional analyses reveal lower precision, highlighting the need to use more representative metrics to assess scan-rescan agreement.
Figures & tables
| Mean standard deviation of differences and CoV values ( ) | ||||||
|---|---|---|---|---|---|---|
| Roca-HC | ||||||
| DE | TTA | MC-D | ||||
| Mean std | CoV | Mean std | CoV | Mean std | CoV | |
| LVEF | 3.21 0.62 | 2.52 | 3.20 0.77 | 2.52 | 3.11 0.26 | 2.52 |
| LVM | 3.58 0.90 | 1.74 | 3.81 1.37 | 1.84 | 3.58 0.38 | 1.84 |
| RVEF | 4.01 0.99 | 3.38 | 4.62 1.99 | 3.90 | 4.15 0.46 | 3.64 |
| Confidence Precision Percentage (CPP) ( ) | ||||||
|---|---|---|---|---|---|---|
| Roca-HC | ||||||
| DE | TTA | MC-D | ||||
| A B | B A | A B | B A | A B | B A | |
| LVEF | 31.52 | 28.26 | 27.17 | 33.70 | 6.52 | 7.61 |
| LVM | 39.13 | 42.39 | 52.17 | 57.61 | 11.96 | 9.78 |
| RVEF | 28.26 | 26.09 | 35.87 | 36.96 | 7.61 | 9.78 |
| Precision Difference Percentage (PDP) ( ) | ||||||
|---|---|---|---|---|---|---|
| Roca-HC | Strain-8 | |||||
| DE | TTA | MC-D | DE | TTA | MC-D | |
| LVEF | 80.43 | 86.96 | 95.65 | 85.00 | 85.00 | 95.00 |
| LVM | 68.48 | 72.83 | 93.48 | 75.00 | 75.00 | 100.00 |
| RVEF | 73.91 | 75.00 | 95.65 | 65.00 | 90.00 | 90.00 |
Uncertainty Quantification
Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.
See all 649 papers