Cyst-X: A Multicenter MRI Benchmark for Pancreatic Cyst Risk Stratification and Cross-Institutional AI
Organizations: Machine & Hybrid Intelligence Lab, Department of Radiology, Northwestern University, Chicago, IL, USA. · Istanbul Faculty of Medicine, Istanbul University, Istanbul, Turkey. · Department of Preventive Medicine, Northwestern University, Chicago, IL, USA. · Division of Gastroenterology and Hepatology, New York University, New York, NY, USA. ·
Abstract
Intraductal papillary mucinous neoplasms (IPMNs) are pancreatic cancer precursors, and guideline-based risk stratification both over-treats indolent lesions and misses progression. We present Cyst-X, a public benchmark of 1,461 T1-weighted (T1W) and T2-weighted (T2W) MRI volumes from 764 patients at seven centers, with expert pancreas masks, risk labels, splits and reference AI risk-prediction models. PanSegNet segmented the pancreas with Dice of 86.34% (T1W) and 89.26% (T2W) on held-out centers. For the primary task, high-risk versus low-risk IPMN, a 3D DenseNet-121 achieved cross-validated AUCs of 74.59% and 73.45%; complementary broad binary and three-class tasks reached 78.67--81.28% and 73.47--75.76%. Because patient images often cannot be pooled across institutions, we also simulated federated learning across the seven centers. On T2W MRI, FedProx was similar to centralized training in center-averaged external AUC (71.00% versus 71.18%) but not in pooled AUC (62.35% versus 77.49%). Radiologist comparisons showed cohort-dependent sensitivity--specificity tradeoffs. Cyst-X supports reproducible method development and identifies within-IPMN risk discrimination and cross-center score comparability as open challenges.
Figures & tables
| Method | Dice | Jaccard | Precision | Recall | HD95 | ASSD |
|---|---|---|---|---|---|---|
| (%) | (%) | (%) | (%) | (mm) | (mm) | |
| T1W MRI | ||||||
| PanSegNet | 86.81 7.30 | 77.32 9.74 | 86.56 8.59 | 87.84 7.80 | 3.72 8.86 | 0.74 1.73 |
| Swin-UNETR | 79.09 1.40 | 67.19 1.63 | 79.09 1.67 | 81.37 0.74 | 26.55 8.48 | 7.58 3.76 |
| +FedAvg | 71.26 2.59 | 58.30 2.74 | 69.74 3.09 | 77.13 1.34 | 43.65 8.60 | 14.77 5.01 |
| +FedProx( =0.3) | 57.24 3.41 | 44.16 2.85 | 56.85 4.33 | 62.93 2.80 | 87.27 8.34 | 31.99 6.19 |
| Method | MACs | Param. | T1W MRI | T2W MRI | ||
| AUC (%) | 95% CI (%) | AUC (%) | 95% CI (%) | |||
| IPMN-only binary: high-risk versus low-risk IPMN | ||||||
| Internal 5-fold cross-validation: T1W =488 (149/339), T2W =496 (152/344) | ||||||
| 3D Radiomics | – | – | 73.19 2.55 | [68.21, 78.30] | 73.64 2.91 | [68.93, 78.18] |
| DenseNet-121 | 18.31G | 11.24M | 74.59 3.28 | [69.38, 79.76] | 73.45 5.28 | [68.89, 78.03] |
| External leave-one-center-out evaluation: T1W =488, T2W =496 | ||||||
| Method | Input | Internal | External | ||
|---|---|---|---|---|---|
| Sens | Spec | Sens | Spec | ||
| 629 broad cases (high-risk versus non-high-risk (low-risk+no-risk)) | |||||
| Radiologist 1 | T1W+T2W | 64.08 | 89.94 | – | – |
| Radiologist 2 | T1W+T2W | 32.39 | 97.13 | – | – |
| Radiologist 3 | T1W+T2W | 41.55 | 94.66 | – | – |
| Average | T1W+T2W | 46.01 | 93.91 | – | – |
| Center | NYU | MCF | NU | AHN | MCA | IU | EMC |
| Imaging device | Siemens, GE | Siemens, GE | Siemens | N/A | Siemens, GE | Siemens, Philips | Siemens, GE |
| MRI magnet (T) | 1.5, 3 | 1.5, 3 | 1.5, 3 | N/A | 1.5, 3 | 1.5 | 1.5 |
| T1W cohort | |||||||
| Patient count | 162 | 148 | 206 | 17 | 25 | 74 | 91 |
| Female | 98 | 81 | 109 | 12 | 14 | 42 | 43 |
| Male | 64 | 67 | 97 | 5 | 11 | 32 | 48 |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| T1W Modality | T2W Modality | Paired Two Modalities | ||||||||||
| Data Centers | No | Low | High | Total | No | Low | High | Total | No | Low | High | Total |
| Center 1: NYU | 48 | 79 | 23 | 150 | 48 | 79 | 24 | 151 | 48 | 79 | 23 | 150 |
| Center 2: MCF | 29 | 42 | 63 | 134 | 25 | 42 | 63 | 130 | 25 | 42 | 63 | 130 |
| Center 3: NU | 43 | 126 | 17 | 186 | 44 | 127 | 16 | 187 | 43 | 126 | 16 | 185 |
| Center 4: AHN | 1 | 11 | 4 | 16 | 1 | 13 | 4 | 18 | 1 | 11 | 3 | 15 |
| Center 5: MCA | 0 | 10 | 14 | 24 | 0 | 7 | 16 | 23 | 0 | 6 | 9 | 15 |
| Center | Dice(%) | Jaccard(%) | Precision(%) | Recall(%) | HD95(mm) | ASSD(mm) |
| T1W Modality | ||||||
| NYU | 84.99 7.74 | 74.56 9.84 | 84.80 8.66 | 86.10 8.32 | 2.93 5.36 | 0.63 1.96 |
| MCF | 87.48 5.79 | 78.17 8.36 | 87.64 7.44 | 88.11 7.37 | 2.98 5.78 | 0.51 0.48 |
| NU | 88.33 5.89 | 79.52 7.98 | 87.49 7.69 | 89.72 5.74 | 2.94 6.22 | 0.64 0.82 |
| AHN | 81.22 6.11 | 68.79 8.16 | 84.54 8.92 | 79.15 7.85 | 4.25 3.85 | 0.56 0.52 |
| Center | Dice(%) | Jaccard(%) | Precision(%) | Recall(%) | HD95(mm) | ASSD(mm) |
| T1W Modality | ||||||
| NYU | 84.62 | 73.88 | 85.66 | 84.41 | 3.20 | 0.50 |
| MCF | 86.93 | 77.38 | 87.42 | 87.32 | 3.19 | 0.51 |
| NU | 87.98 | 79.00 | 86.54 | 90.07 | 2.82 | 0.69 |
| AHN | 80.55 | 67.85 | 85.32 | 77.44 | 3.97 | 0.54 |
| Center | Dice(%) | Jaccard(%) | Precision(%) | Recall(%) | HD95(mm) | ASSD(mm) |
| T1W Modality | ||||||
| NYU | 77.78 2.40 | 65.26 2.51 | 77.48 3.78 | 80.72 1.68 | 23.76 11.55 | 6.93 5.58 |
| MCF | 81.63 2.92 | 70.17 3.51 | 82.97 2.06 | 82.05 3.48 | 11.27 2.84 | 2.68 1.71 |
| NU | 82.89 2.06 | 72.00 2.89 | 82.21 2.55 | 84.90 1.77 | 13.77 4.98 | 3.34 0.96 |
| AHN | 65.14 11.05 | 50.90 10.48 | 74.43 6.51 | 61.04 12.47 | 22.51 15.68 | 4.23 2.94 |
| Center | Dice(%) | Jaccard(%) | Precision(%) | Recall(%) | HD95(mm) | ASSD(mm) |
| T1W Modality | ||||||
| NYU | 71.84 2.51 | 58.25 2.21 | 71.06 3.45 | 76.67 1.26 | 34.81 8.00 | 12.12 6.18 |
| MCF | 77.94 3.36 | 65.40 3.85 | 79.16 3.10 | 79.07 3.57 | 15.64 3.96 | 3.60 1.74 |
| NU | 77.61 2.78 | 65.25 3.38 | 74.06 2.82 | 83.66 1.95 | 31.89 5.17 | 8.36 1.83 |
| AHN | 51.07 10.96 | 37.61 8.85 | 61.60 4.80 | 48.41 13.08 | 34.78 12.52 | 12.68 8.57 |
| Center | Dice(%) | Jaccard(%) | Precision(%) | Recall(%) | HD95(mm) | ASSD(mm) |
| T1W Modality | ||||||
| NYU | 60.31 2.27 | 46.02 2.12 | 59.92 2.67 | 66.25 0.97 | 81.99 9.69 | 26.72 5.39 |
| MCF | 71.63 4.18 | 57.63 4.54 | 73.65 4.55 | 72.56 3.43 | 30.03 11.76 | 6.65 3.25 |
| NU | 65.07 3.77 | 51.06 3.74 | 62.81 4.73 | 70.57 2.54 | 91.69 8.79 | 24.60 3.51 |
| AHN | 36.24 10.71 | 24.91 8.01 | 37.94 9.71 | 38.25 13.66 | 101.79 4.39 | 35.16 10.74 |
| Center | Dice(%) | Jaccard(%) | Precision(%) | Recall(%) | HD95(mm) | ASSD(mm) |
| T1W Modality | ||||||
| NYU | 70.16 2.67 | 56.32 2.50 | 68.93 2.96 | 75.42 1.94 | 48.86 6.08 | 15.39 5.71 |
| MCF | 77.56 3.35 | 64.78 3.86 | 78.55 2.42 | 78.98 4.17 | 16.27 5.43 | 3.90 1.93 |
| NU | 76.02 3.38 | 63.70 4.09 | 72.70 4.04 | 82.47 2.04 | 38.92 12.04 | 10.19 2.66 |
| AHN | 47.10 10.95 | 33.87 8.76 | 57.15 7.38 | 44.02 13.92 | 48.54 16.35 | 17.19 9.42 |
| Center | Dice(%) | Jaccard(%) | Precision(%) | Recall(%) | HD95(mm) | ASSD(mm) |
| T1W Modality | ||||||
| NYU | 70.36 2.53 | 56.56 2.52 | 69.05 3.13 | 75.88 1.13 | 45.50 9.37 | 14.81 5.80 |
| MCF | 78.06 3.01 | 65.45 3.43 | 78.71 2.11 | 79.71 3.99 | 16.84 4.69 | 3.89 1.76 |
| NU | 77.12 3.06 | 64.79 3.77 | 73.90 3.36 | 83.19 2.12 | 34.39 8.18 | 8.86 2.00 |
| AHN | 49.40 10.96 | 36.07 8.83 | 58.90 8.21 | 46.13 12.58 | 44.40 25.11 | 14.65 9.00 |
| Validation-Selected Threshold | Fixed 0.5 Threshold | |||||||
| Method | AUC(%) | 95%CI(%) | ACC(%) | Sens(%) | Spec(%) | ACC(%) | Sens(%) | Spec(%) |
| Internal 5-fold test evaluation, 339 low-risk + 149 high-risk; results are mean s.d. across folds. | ||||||||
| Radiomics | 73.19 2.55 | [68.21, 78.30] | 73.96 2.79 | 44.83 9.29 | 86.63 6.82 | 73.34 3.50 | 35.04 6.80 | 90.23 2.66 |
| DenseNet-121 | 74.59 3.28 | [69.38, 79.76] | 73.35 2.74 | 43.47 6.64 | 86.33 6.11 | 70.51 3.16 | 51.26 22.49 | 79.24 12.49 |
| External leave-one-center-out evaluation: Train on six centers and test on the remaining one | ||||||||
| Center 1: New York University Langone Health (NYU), 79 low-risk + 23 high-risk | ||||||||
| Validation-Selected Threshold | Fixed 0.5 Threshold | |||||||
| Method | AUC(%) | 95%CI(%) | ACC(%) | Sens(%) | Spec(%) | ACC(%) | Sens(%) | Spec(%) |
| Internal 5-fold test evaluation, 344 low-risk + 152 high-risk; results are mean s.d. across folds. | ||||||||
| Radiomics | 73.64 2.91 | [68.93, 78.18] | 73.19 2.38 | 40.09 11.99 | 87.84 6.39 | 74.59 2.78 | 31.57 3.85 | 93.62 3.16 |
| DenseNet-121 | 73.45 5.28 | [68.89, 78.03] | 73.39 3.91 | 44.71 10.02 | 86.06 3.87 | 71.01 4.63 | 43.29 31.08 | 83.29 15.70 |
| External leave-one-center-out evaluation: Train on six centers and test on the remaining one | ||||||||
| Center 1: New York University Langone Health (NYU), 79 low-risk + 24 high-risk | ||||||||
| Validation-Selected Threshold | Fixed 0.5 Threshold | |||||||
| Method | AUC(%) | 95%CI(%) | ACC(%) | Sens(%) | Spec(%) | ACC(%) | Sens(%) | Spec(%) |
| Internal 5-fold test evaluation, 332 low-risk + 142 high-risk; results are mean s.d. across folds. | ||||||||
| Early fusion | 77.19 4.38 | [72.77, 81.36] | 74.50 2.98 | 45.17 4.91 | 87.11 4.00 | 75.16 3.99 | 43.02 20.19 | 88.86 5.11 |
| Late fusion | 73.25 5.54 | [68.52, 78.34] | 72.38 3.92 | 42.23 9.19 | 85.33 6.10 | 67.12 8.53 | 55.41 23.63 | 72.60 21.20 |
| Siamese fusion | 73.91 6.20 | [68.90, 79.07] | 71.50 4.10 | 40.04 6.57 | 84.97 7.95 | 70.47 3.22 | 39.15 24.69 | 84.14 14.82 |
| Logit fusion | 71.51 3.21 | [66.22, 76.67] | 71.08 1.67 | 40.59 11.40 | 84.03 5.56 | 72.97 1.50 | 18.14 9.58 | 96.45 2.98 |
| Validation-Selected Threshold | Fixed 0.5 Threshold | |||||||
| Method | AUC(%) | 95%CI(%) | ACC(%) | Sens(%) | Spec(%) | ACC(%) | Sens(%) | Spec(%) |
| Internal 5-fold evaluation, 503 non-high-risk + 149 high-risk; results are mean s.d. across folds. | ||||||||
| Radiomics | 78.02 0.97 | [73.45, 82.06] | 79.14 1.17 | 40.27 6.31 | 90.63 2.23 | 80.22 1.44 | 23.57 9.06 | 96.99 1.71 |
| ResNet-34 | 76.72 1.65 | [72.16, 80.89] | 77.90 1.85 | 42.76 11.05 | 88.27 2.70 | 75.03 3.38 | 35.79 24.54 | 86.77 10.76 |
| ResNet-50 | 77.83 2.50 | [73.48, 81.69] | 77.90 3.44 | 48.94 7.51 | 86.40 5.78 | 77.00 2.33 | 44.32 15.57 | 86.68 6.66 |
| EfficientNet-B0 | 76.33 5.34 | [71.69, 80.62] | 77.79 2.63 | 40.23 11.23 | 88.87 3.60 | 73.14 4.83 | 52.64 25.80 | 79.32 13.02 |
| Validation-Selected Threshold | Fixed 0.5 Threshold | |||||||
| Method | AUC(%) | 95%CI(%) | ACC(%) | Sens(%) | Spec(%) | ACC(%) | Sens(%) | Spec(%) |
| Internal 5-fold evaluation, 503 non-high-risk + 152 high-risk; results are mean s.d. across folds. | ||||||||
| Radiomics | 78.91 1.71 | [74.20, 82.86] | 80.00 1.48 | 46.13 10.15 | 90.35 3.43 | 80.15 1.98 | 27.10 9.25 | 96.27 1.98 |
| ResNet-34 | 79.44 2.73 | [75.26, 83.54] | 79.08 0.84 | 42.15 7.61 | 90.22 2.77 | 78.94 1.08 | 33.70 12.17 | 92.74 4.93 |
| ResNet-50 | 78.96 2.82 | [74.80, 82.91] | 78.17 2.36 | 42.84 8.99 | 88.90 3.83 | 78.50 1.97 | 40.15 15.14 | 90.15 6.43 |
| EfficientNet-B0 | 77.10 2.10 | [72.38, 81.06] | 77.27 1.42 | 36.92 10.35 | 89.47 3.13 | 72.28 11.96 | 37.94 26.81 | 82.53 22.87 |
| Validation-Selected Threshold | Fixed 0.5 Threshold | |||||||
| Method | AUC(%) | 95%CI(%) | ACC(%) | Sens(%) | Spec(%) | ACC(%) | Sens(%) | Spec(%) |
| Internal 5-fold evaluation, 487 non-high-risk + 142 high-risk; results are mean s.d. across folds. | ||||||||
| Early fusion | [76.39, 84.84] | |||||||
| Late fusion | [73.49, 81.99] | |||||||
| Siamese fusion | [76.76, 84.99] | |||||||
| Logit fusion | [75.63, 83.82] | |||||||
| Cohort | Method | Sensitivity [95% CI] | Specificity [95% CI] |
|---|---|---|---|
| 629 | T2W DenseNet | 57.04 [49.30, 64.79] | 88.50 [85.63, 91.38] |
| 629 | Radiologist mean | 46.01 [39.20, 52.58] | 93.91 [92.20, 95.62] |
| 474 | T2W DenseNet | 41.55 [33.80, 49.30] | 86.75 [82.83, 90.36] |
| 474 | Radiologist mean | 46.01 [39.20, 52.58] | 91.67 [89.25, 93.78] |
| 357 | T2W DenseNet | 30.28 [23.24, 38.05] | 89.77 [85.58, 93.49] |
| 357 | Radiologist mean | 46.01 [39.20, 52.58] | 87.29 [83.41, 90.70] |
| Cohort | Metric | AI minus radiologist mean | 95% CI |
|---|---|---|---|
| 629 | Sensitivity | 11.03 | [3.05, 18.78] |
| 629 | Specificity | 5.41 | [ 8.08, 2.80] |
| 474 | Sensitivity | 4.46 | [ 13.15, 3.76] |
| 474 | Specificity | 4.92 | [ 8.43, 1.41] |
| 357 | Sensitivity | 15.73 | [ 23.94, 7.51] |
| 357 | Specificity | 2.48 | [ 2.48, 7.45] |
| Evaluation | Training task | MRI | Sens | Spec | AUC [95% CI] |
|---|---|---|---|---|---|
| Internal | Broad | T1W | 48.59 | 80.00 | 67.13 [61.39, 73.19] |
| Internal | Broad | T2W | 57.04 | 78.60 | 67.26 [61.58, 73.02] |
| Internal | IPMN-only | T1W | 45.77 | 81.40 | 66.51 [60.77, 72.61] |
| Internal | IPMN-only | T2W | 41.55 | 80.93 | 62.15 [56.23, 68.60] |
| Internal | Histopathology-only | T1W | 26.76 | 86.51 | 61.11 [55.05, 67.56] |
| Internal | Histopathology-only | T2W | 30.28 | 89.77 | 62.35 [56.23, 68.90] |
| Method | MRI | Pooled int. | Pooled ext. | Center int. | Center ext. | pooled ext. [95% CI] | center ext. [95% CI] |
|---|---|---|---|---|---|---|---|
| Centralized | T1W | 74.94 | 71.35 | 68.74 | 74.22 | – | – |
| Centralized | T2W | 78.17 | 77.49 | 68.57 | 71.18 | – | – |
| FedAvg | T1W | 71.35 | 66.90 | 67.82 | 73.12 | 4.45 [ 7.77, 1.04] | 1.10 [ 7.22, 4.85] |
| FedAvg | T2W | 68.63 | 65.15 | 61.99 | 72.28 | 12.34 [ 16.15, 8.64] | 1.10 [ 4.54, 6.67] |
| FedProx (0.1) | T1W | 73.27 | 62.54 | 69.61 | 73.80 | 8.81 [ 11.96, 5.74] | 0.42 [ 4.32, 3.19] |
| FedProx (0.1) | T2W | 66.85 | 74.18 | 61.25 | 68.61 | 3.31 [ 6.54, 0.28] | 2.57 [ 8.15, 2.78] |
| Before recalibration | After recalibration | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Setting | Task | Model | MRI | Slope | Intercept | Slope | Intercept | Brier | Pooled AUC (%) |
| Int. | IPMN-only | DenseNet-121 | T1W | 0.42 [0.30, 0.56] | 0.28 [ 0.44, 0.11] | 0.59 [0.46, 0.74] | 0.19 [ 0.34, 0.05] | 0.203 / 0.189 (0.212) | 70.84 / 73.74 |
| Int. | IPMN-only | DenseNet-121 | T2W | 0.19 [0.12, 0.27] | 0.47 [0.16, 0.76] | 0.74 [0.56, 0.94] | 0.19 [ 0.29, 0.08] | 0.232 / 0.186 (0.213) | 67.86 / 73.61 |
| Int. | IPMN-only | Radiomics | T1W | 0.95 [0.73, 1.22] | 0.02 [ 0.10, 0.07] | 0.79 [0.60, 1.02] | 0.10 [ 0.20, 0.00] | 0.181 / 0.183 (0.212) | 72.96 / 73.10 |
| Int. | IPMN-only | Radiomics | T2W | 0.99 [0.72, 1.28] | 0.03 [ 0.06, 0.10] | 0.58 [0.42, 0.75] | 0.22 [ 0.37, 0.10] | 0.180 / 0.189 (0.213) | 73.55 / 72.90 |
| Int. | Broad | DenseNet-121 | T1W | 0.39 [0.30, 0.48] | 0.55 [ 0.76, 0.33] | 0.69 [0.57, 0.85] | 0.06 [ 0.20, 0.08] | 0.177 / 0.148 (0.176) | 74.94 / 78.46 |
| Setting | Task | Model | MRI | 0.10 | 0.20 | 0.30 | 0.40 |
|---|---|---|---|---|---|---|---|
| Int. | IPMN-only | DenseNet-121 | T1W | 0.230 (0.228) | 0.162 ∗ (0.132) | 0.114 ∗ (0.008) | 0.072 ∗ ( 0.158) |
| Int. | IPMN-only | DenseNet-121 | T2W | 0.232 (0.229) | 0.171 ∗ (0.133) | 0.108 ∗ (0.009) | 0.056 ∗ ( 0.156) |
| Int. | IPMN-only | Radiomics | T1W | 0.229 (0.228) | 0.161 ∗ (0.132) | 0.097 ∗ (0.008) | 0.067 ∗ ( 0.158) |
| Int. | IPMN-only | Radiomics | T2W | 0.229 (0.229) | 0.160 ∗ (0.133) | 0.099 ∗ (0.009) | 0.058 ∗ ( 0.156) |
| Int. | Broad | DenseNet-121 | T1W | 0.155 (0.143) | 0.110 ∗ (0.036) | 0.074 ∗ ( 0.102) | 0.047 ∗ ( 0.286) |
| Int. | Broad | DenseNet-121 | T2W | 0.160 (0.147) | 0.117 ∗ (0.040) | 0.085 ∗ ( 0.097) | 0.069 ∗ ( 0.280) |
Medical Image Benchmarks
Medical image benchmarks compare AI methods on curated clinical images, tasks, and evaluation protocols to measure accuracy, robustness, and clinical usefulness.
See all 367 papers