CalibCause: A Quality-Gated, Interpretable and Calibrated Deep Learning Framework for Diabetic Retinopathy Grading
Organizations: Department of Data Science and Artificial Intelligence National University of Computer and Emerging Sciences (FAST-NUCES) Islamabad 44000, Pakistan · Department of Computer Science National University of Computer and Emerging Sciences (FAST-NUCES) Islamabad 44000, Pakistan · Department of Computer Engineering Istinye University Istanbul 34408, Turkey · German Research Center for Artificial Intelligence (DFKI) Saarbrücken, Germany
Abstract
Most deep learning systems for diabetic retinopathy (DR) grading are evaluated on a single data source, with limited evidence of cross-site generalisation, uncertainty calibration, and interpretability. We propose CalibCause, a quality-gated mixture-of-experts framework that combines per-expert visual explanations, label-free image-quality conditioning, knowledge distillation, and calibrated confidence with deferral. We evaluated the framework using 51,310 fundus images from EyePACS, DDR, and APTOS 2019, with IDRiD and DeepDRiD for external testing. Evaluation included pooled testing, leave-one-environment-out experiments, paired bootstrap confidence intervals, and local grade-threshold adaptation. On the pooled test split, the label-free distilled student achieved quadratic weighted kappa (QWK) of 0.863 and accuracy of 0.865 with 27.8 million parameters, 52% fewer than its teacher. Across held-out environments, the label-free model achieved mean QWK of 0.741 compared with 0.743 for a plain ConvNeXt-Tiny baseline, while adversarial invariance did not improve cross-site grading accuracy. Refitting grade thresholds using 200 labelled images increased held-out EyePACS QWK from 0.533 to 0.621 for the label-free model. After vector scaling, expected calibration error ranged from 0.025 to 0.036 across CalibCause variants, and deferring the least confident 20% of images increased student accuracy from 0.865 to 0.923 on the remaining images. These findings demonstrate that quality-aware grading, interpretable expert routing, model compression, and calibrated deferral can be integrated without sacrificing cross-site performance relative to a plain baseline, although reliable deployment still requires local validation and adaptation.
Figures & tables
| Dataset | Images | Role |
|---|---|---|
| EyePACS | 35,126 (test: 7,026) | Development environment |
| DDR | 12,522 (test: 2,504) | Development environment |
| APTOS 2019 | 3,662 (test: 733) | Development environment |
| IDRiD | 516 | External test only |
| DeepDRiD | 400 | External test only |
| Model | Par. | Acc | QWK (95% CI) | AUC | ECE raw | ECE T | ECE V |
|---|---|---|---|---|---|---|---|
| Full, teacher | 58.21 | 0.829 | 0.813 (0.800–0.825) | 0.938 | 0.270 | 0.144 | 0.031 |
| Label-free, teacher | 58.21 | 0.849 | 0.839 (0.827–0.850) | 0.945 | 0.261 | 0.149 | 0.025 |
| No invariance block, teacher | – | 0.857 | 0.861 (0.851–0.870) | 0.944 | 0.223 | 0.117 | 0.030 |
| Full, student | 27.82 | 0.851 | 0.840 (0.829–0.851) | 0.968 | 0.182 | 0.057 | 0.036 |
| Label-free, student | 27.82 | 0.865 | 0.863 (0.853–0.873) | 0.969 | 0.173 | 0.055 | 0.033 |
| Model | EyePACS | DDR | APTOS | Mean | EyePACS s43 |
|---|---|---|---|---|---|
| Plain ConvNeXt-T | 0.547 (0.525–0.569) | 0.809 (0.786–0.829) | 0.874 (0.848–0.896) | 0.743 | 0.564 |
| No invariance block | 0.543 (0.522–0.565) | 0.801 (0.777–0.822) | 0.870 (0.843–0.892) | 0.738 | – |
| Label-free | 0.533 (0.512–0.555) | 0.816 (0.794–0.835) | 0.875 (0.848–0.897) | 0.741 | 0.527 |
| Full | 0.524 (0.500–0.548) | 0.787 (0.762–0.810) | 0.863 (0.834–0.888) | 0.724 | 0.521 |
| Model | AUC | Sens@90 | F1 | MAE | Within-1 | NLL |
|---|---|---|---|---|---|---|
| Held-out environment, mean over three folds | ||||||
| Plain ConvNeXt-T | 0.910 | 0.835 | 0.503 | 0.384 | 0.902 | 1.240 |
| No invariance block | 0.914 | 0.854 | 0.511 | 0.395 | 0.901 | 1.279 |
| Label-free | 0.919 | 0.853 | 0.511 | 0.380 | 0.898 | 1.265 |
| Full | 0.908 | 0.839 | 0.495 | 0.410 | 0.897 | 1.279 |
| Pooled test split | ||||||
| Setting | Test set | Acc | QWK | AUC | Sens | Spec |
|---|---|---|---|---|---|---|
| Per-source | DDR | 0.909 | 0.935 | 0.964 | 0.934 | 0.957 |
| Per-source | APTOS | 0.862 | 0.914 | 0.962 | 0.963 | 0.940 |
| Per-source | EyePACS | 0.829 | 0.761 | 0.913 | 0.741 | 0.967 |
| External | IDRiD | 0.624 | 0.764 | 0.954 | 0.926 | 0.839 |
| External | DeepDRiD | 0.688 | 0.825 | 0.973 | 0.978 | 0.818 |
| Metric | QGDR | CalibCause (full) |
|---|---|---|
| Accuracy | 0.809 | 0.909 |
| QWK | 0.823 | 0.935 |
| Macro AUC (5-class) | 0.902 | 0.943 |
| Referable AUC | 0.952 | 0.964 |
| Referable sensitivity | 0.882 | 0.934 |
| Referable specificity | 0.877 | 0.957 |
Diabetic Retinopathy Grading
Diabetic retinopathy grading uses retinal images to estimate disease severity and identify cases that need further clinical review.
See all 36 papers