VoxForge: A Unified Foundation Model for Dense 3D Features Across MRI, CT and PET
Organizations: Amazon AGI
Abstract
A unified 3D medical foundation model promises reusable, frozen features that serve a wide variety of tasks such as segmentation, classification and registration on MRI, CT and PET as well as specialist models do. Generalist models still need separate backbones for localized and global tasks, specialist models are confined to a single modality, with the concern that spanning modalities under-emphasizes fine anatomy, and both typically rely on end-to-end fine-tuning to be competitive. We present VoxForge, a single encoder that integrates recent advances in 3D architecture, self-distillation and cross-modal curricula to learn transferable features from permissively licensed MRI and CT volumes. We evaluate it strictly frozen, the test a reusable foundation model must pass, against state-of-the-art generalist and specialist models on segmentation, classification and registration across MRI and CT, and on PET, which it never sees. VoxForge has the best or tied-best average rank in every task group, serves localized and global tasks across modalities at the level of state-of-the-art specialists, and is extremely label-efficient, matching top-performing models on TotalSegmentator CT with a tenth of the labels.
Figures & tables
| encoder | pretraining input | , | grid | |
| CT | ||||
| nnF-ViT, nnF-CNN | native spacing, crops | 0.5–5 | 0.5–1.0 | anisotropic |
| SPECTRE | resampled to , rescaled crops | 1.0–2.0 | 0.5–1.0 | anisotropic |
| VoxelFM | rescaled crops | 0.7–5 | 0.7–5 | isotropic |
| Merlin | resampled to | 3.0 | 1.5 | anisotropic |
| CT-Instruct | native spacing, crops | 0.5–5 | 0.5–1.0 | anisotropic |
| dataset | modality | target | used in |
| Prostate158 [ 1 ] | MRI | prostate zones | Seg., k-means |
| ACDC [ 10 ] | cine MRI | cardiac structures | Seg., LE, k-means |
| ATLAS 2023 [ 45 ] | MRI | liver, liver tumor | Seg., k-means |
| AMOS-MRI [ 35 ] | MRI | abdominal organs | Seg., k-means |
| MM-WHS [ 72 ] | MRI | whole heart | Seg. |
| MSD Hippocampus [ 3 ] | MRI | hippocampus | Seg., LE |
| Modality | Volumes | Subjects | Sources |
| MRI | 148,831 | 41,192 | 21 TCIA collections, 895 OpenNeuro datasets |
| CT | 86,088 | 29,777 | 49 TCIA collections, NLST among them |
| Total | 234,919 |
| Dataset | License | Volumes | Subjects | Dataset | License | Volumes | Subjects |
| TCIA, 21 collections: 80,708 volumes, 5,840 subjects | |||||||
| PROSTATEx | CC BY 3.0 | 18,743 | 346 | Breast-MRI-NACT-Pilot | CC BY 3.0 | 917 | 64 |
| ISPY2 | CC BY 4.0 | 13,731 | 398 | ReMIND | CC BY 4.0 | 670 | 114 |
| ACRIN-6698 | CC BY 4.0 | 12,803 | 385 | QIN Breast DCE-MRI | CC BY 3.0 | 632 | 10 |
| ACRIN-Contralateral-Breast-MR | CC BY 4.0 | 8,118 | 980 | Vestibular-Schwannoma-SEG | CC BY 4.0 | 476 | 242 |
| Advanced-MRI-Breast-Lesions | CC BY 4.0 | 6,039 | 632 | PROSTATE-DIAGNOSIS | CC BY 3.0 | 368 | 92 |
| Collection | License | Volumes | Subjects | Collection | License | Volumes | Subjects |
| NLST | CC BY 4.0 | 72,804 | 26,229 | RIDER Pilot | CC BY 4.0 | 127 | 8 |
| COVID-19-NY-SBU | CC BY 4.0 | 2,774 | 457 | ADRENAL-ACC-KI67-SEG | CC BY 4.0 | 119 | 53 |
| TCGA-KIRC | CC BY 3.0 | 1,153 | 229 | Pelvic-Reference-Data | CC BY 3.0 | 116 | 58 |
| VAREPOP-APOLLO | CC BY 4.0 | 932 | 38 | QIN-BREAST | CC BY 3.0 | 110 | 43 |
| CPTAC-UCEC | CC BY 4.0 | 700 | 75 | CT-VS-PET-VENTILATION-IMAGING | CC BY 4.0 | 99 | 20 |
| HCC-TACE-SEG | CC BY 4.0 | 561 | 105 | TCGA-ESCA | CC BY 3.0 | 98 | 16 |
| Stage | Data (volumes) | Crop | Passes |
| S1 | MRI 148,831 | 250 | |
| S2 | (i) MRI 138,109 / 113,970 | , then | 40 + 32 |
| (ii) MRI 138,109 / 113,970 | , alternating | ||
| S3 | (i) MRI 113,970 + CT 86,088 | 24 | |
| (ii) CT 86,088 | 18 of 24 |
| S1 | S2 | S3 | |||
| phase | MRI | (i) MRI | (ii) MRI | (i) MRI and CT | (ii) CT |
| Optimization | |||||
| optimizer steps | 36,250 | 12,536 | 8,592 | 11,376 | 3,024 |
| base learning rate | |||||
| linear warmup | 10 passes | 1 pass per crop size | pass per phase | 1 pass | 1 pass |
| learning-rate schedule | constant | constant / cosine to | one cosine across the four phases, ending at half the peak | cosine to | cosine over 24 planned passes, ending near |
| S1 MRI, | S2 S1 high-res MRI adds | S2 high-res MRI | S3 S2 CT adds | S3 MRI CT | |
| Segmentation, Dice % (UNETR) | |||||
| MRI, 7 datasets | 82.4 | 0.6 | 83.0 | 0.1 | 82.9 |
| CT, 5 datasets | 67.8 | 1.7 | 69.5 | 0.4 | 69.9 |
| ISLES’24 circle of Willis, NCCT | 49.5 | 0.8 | 50.3 | 0.1 | 50.4 |
| ISLES’24 LVO region, CTA and NCCT | 14.0 | 6.4 | 7.6 | 3.9 | 11.5 |
| FDG PET lesions (UNETR, mean SD over head runs) | |||||
| Unified models | Specialists | |||||
| task group | table | VoxForge | nnFoundation | CT | MRI | registration |
| MRI lesion and anatomy segmentation | Tab. S15 | 0.97 | 0.72 nnF-CNN | – | 0.51 Triad Swin SimMIM | – |
| MRI pathology classification | Tab. S11 | 1.00 | 0.81 nnF-ViT | – | 0.57 Triad CNN MAE | – |
| MRI CT registration | Tab. S26 | 0.90 | 0.60 nnF-ViT | – | – | 0.90 anatomix |
| CT lesion and anatomy segmentation | Tab. S16 | 0.90 | 0.29 nnF-ViT | 0.71 SPECTRE | – | – |
| CT pathology classification | Tab. S12 | 0.93 | 0.38 nnF-ViT | 0.81 SPECTRE | – | – |
| task group | plotted ( Fig. 1 of the main paper) | dataset- balanced | (first) | (first) no exposure | closest rival | standardized margin | W/T/L vs nnF-ViT |
| MRI lesion and anatomy seg. | 0.97 | 0.96 [0.91, 1.00] | 0.99 | 0.99 | nnF-CNN (+0.26) | +0.73 | 4/6/0 |
| MRI pathology classification | 1.00 | 1.00 [1.00, 1.00] | 1.00 | 1.00 | nnF-ViT (+0.19) | +0.61 | 9/0/0 |
| MRI CT registration | 0.90 | 0.90 | – | – | anatomix (+0.00) | 0.01 | 2/0/0 |
| CT lesion and anatomy seg. | 0.90 | 0.88 [0.79, 0.95] | 0.84 | 0.94 | Merlin (+0.17) | +0.04 | 6/2/0 |
| CT pathology classification | 0.93 | 0.94 [0.93, 0.95] | 1.00 | 1.00 | SPECTRE (+0.11) | +0.11 | 16/0/0 |
| CT vessel seg. (no contrast) | 0.92 | 0.92 [0.83, 1.00] | 0.75 | 0.75 | SPECTRE (+0.08) | +0.34 | 1/1/0 |
| k-NN (no training) | linear probe | multi-layer attention probe | over 9 cells | |||||||||||
| encoder | IDH | grade | LLD | IDH | grade | LLD | IDH | grade | LLD | mean rank | inv. norm. rank | |||
| Unified models | ||||||||||||||
| VoxForge | 86.1 | 85.6 | 79.7 | 94.2 [94.0, 94.3] | 95.3 [95.1, 95.5] | 90.2 [89.9, 90.5] | 92.5 [92.1, 92.9] | 95.4 [95.3, 95.5] | 93.6 [93.1, 94.0] | 1.0 | 1.00 | |||
| nnF-ViT [ 60 ] | 78.0 | 74.7 | 78.6 | 79.4 [77.6, 80.9] | 88.3 [88.1, 88.4] | 86.5 [86.2, 86.9] | 82.2 [82.0, 82.3] | 89.1 [88.9, 89.3] | 87.6 [82.4, 91.3] | 2.1 | 0.81 | |||
| nnF-CNN [ 60 ] | 58.2 | 66.7 | 66.4 | 57.4 [53.8, 62.9] | 59.7 [57.5, 66.3] | 61.5 [55.3, 62.4] | 73.6 [70.5, 74.9] | 68.7 [67.4, 71.2] | 60.7 [59.3, 63.5] | 6.0 | 0.17 | |||
| Specialists | ||||||||||||||
| (a) Linear probe | ||||||||
| STOIC | RSNA 2023 | |||||||
| encoder | COVID | severe | any | bowel | extrav. | kidney | liver | spleen |
| Unified models | ||||||||
| VoxForge | 82.8 [81.9, 83.8] | 78.1 [76.6, 79.2] | 77.3 [75.6, 78.8] | 82.3 [79.9, 83.5] | 74.5 [72.9, 76.4] | 79.9 [77.1, 81.6] | 78.6 [77.5, 81.0] | 79.1 [77.1, 80.2] |
| nnF-ViT [ 60 ] | 76.4 [73.6, 79.4] | 72.3 [69.8, 75.0] | 70.4 [69.2, 72.7] | 61.0 [58.3, 63.2] | 63.6 [62.0, 64.3] | 76.2 [74.7, 78.2] | 73.3 [70.5, 76.2] | 69.5 [67.9, 72.1] |
| nnF-CNN [ 60 ] | 57.1 [56.0, 59.6] | 60.7 [58.9, 61.1] | 57.7 [55.4, 59.1] | 47.8 [45.3, 49.9] | 52.3 [51.9, 54.0] | 62.6 [59.9, 63.6] | 58.5 [57.3, 61.2] | 53.0 [52.1, 54.9] |
| encoder | linear | attention | |
| Merlin [ 12 ] | 75.4 | 73.1 | |
| CT-Instruct [ 36 ] | 61.2 | 59.9 | |
| nnF-CNN [ 60 ] | 56.2 | 55.6 | |
| nnF-ViT [ 60 ] | 70.3 | 63.3 | |
| SPECTRE [ 17 ] | 76.6 | 81.7 | |
| VoxelFM [ 40 ] | 76.7 | 78.8 |
| dataset | train | test | target | classes, train / test |
| UCSF-PDGM | 395 | 100 | IDH | wildtype 313 / 79, mutant 82 / 21 |
| grade | grades 2–3: 79 / 20, grade 4: 316 / 80 | |||
| LLD-MMRI | 398 | 100 | lesion type | hemangioma 61 / 18, ICC 48 / 10, abscess 47 / 7, metastasis 41 / 10, cyst 44 / 9, FNH 34 / 12, HCC 123 / 34 |
| STOIC | 1,600 | 400 | COVID-19 | positive 961 / 244 |
| severe | severe 243 / 58 | |||
| RSNA 2023 | 2,512 | 628 | any injury | injured 731 / 174 |
| Unified models | Specialists | ||||||
| dataset | VoxForge | nnF-ViT [ 60 ] | nnF-CNN [ 60 ] | Triad CNN MAE [ 64 ] | Triad CNN SimMIM [ 64 ] | Triad Swin MAE [ 64 ] | Triad Swin SimMIM [ 64 ] |
| Prostate158 | 81.2 [80.1, 82.2] | 76.9 [75.6, 78.1] | 78.2 [76.7, 79.5] | 76.1 [74.6, 77.5] | 76.4 [75.1, 77.7] | 76.2 [74.7, 77.4] | 77.2 [75.9, 78.3] |
| ACDC | 91.0 [90.5, 91.6] | 88.7 [87.8, 89.5] | 89.1 [88.2, 90.0] | 88.2 [87.3, 89.1] | 88.1 [87.1, 89.1] | 86.1 [84.8, 87.2] | 86.8 [85.7, 87.8] |
| ATLAS liver | 91.4 [89.7, 92.9] | 89.5 [87.3, 91.3] | 90.5 [88.0, 92.5] | 89.5 [87.5, 91.3] | 89.6 [87.4, 91.3] | 88.6 [86.6, 90.4] | 90.1 [88.2, 91.7] |
| ATLAS tumor | 67.1 [61.1, 72.1] | 56.3 [49.3, 62.8] | 57.6 [49.7, 64.8] | 47.6 [40.5, 54.4] | 50.7 [43.5, 58.1] | 45.3 [38.7, 52.0] | 52.2 [44.7, 59.1] |
| AMOS-MRI | 85.7 [84.5, 86.6] | 75.8 [74.0, 77.6] | 79.1 [77.3, 80.9] | 81.3 [79.7, 83.0] | 81.4 [79.8, 83.0] | 78.6 [76.8, 80.3] | 81.5 [80.0, 82.9] |
| Unified models | Specialists | ||||||
| dataset | VoxForge | nnF-ViT [ 60 ] | nnF-CNN [ 60 ] | Merlin [ 12 ] | CT- Instruct [ 36 ] | SPECTRE [ 17 ] | VoxelFM [ 40 ] |
| AMOS22 | 86.9 [85.6, 88.2] | 76.7 [74.7, 78.6] | 75.5 [73.3, 77.4] | 85.1 [83.6, 86.5] | 71.4 † [69.1, 73.5] | 87.9 † [86.6, 89.0] | 85.4 [83.9, 86.7] |
| MSD Pancreas | 59.9 [58.4, 62.1] | 52.9 [48.4, 57.8] | 52.4 [48.5, 56.7] | 61.2 [57.2, 65.6] | 51.8 † [47.6, 56.3] | 60.6 † [56.7, 64.8] | 57.8 † [53.7, 62.2] |
| MSD Colon | 45.8 [35.1, 58.6] | 29.3 [20.4, 38.6] | 19.5 [12.8, 26.6] | 40.6 [28.9, 53.2] | 22.9 † [14.3, 32.0] | 46.7 † [35.5, 58.8] | 34.1 [22.5, 46.5] |
| TotalSegmentator | 91.5 [88.0, 94.2] | 89.5 [85.7, 92.6] | 89.4 [85.6, 92.6] | 88.8 [85.2, 91.9] | 82.5 † [77.6, 86.8] | 87.6 † [83.8, 92.3] | 86.9 † [83.1, 91.9] |
| AISD | 43.2 [40.1, 48.8] | 16.9 [14.9, 18.9] | 12.0 [10.5, 13.6] | 45.1 [41.3, 49.2] | 34.3 [31.0, 37.9] | 40.5 [36.8, 44.6] | 40.8 [37.2, 44.9] |
| PET in | Dice | lesion | over 3 cells | |||
| encoder | pretraining | lesion | all | F1 | mean rank | inv. norm. rank |
| Unified models | ||||||
| VoxForge | 69.7 [67.6, 71.8] | 38.3 [35.7, 40.8] | 59.1 [55.8, 61.9] | 1.0 | 1.00 | |
| nnF-ViT [ 60 ] | ✓ | 55.9 [53.5, 58.2] | 28.1 [26.0, 30.3] | 40.0 [35.4, 44.3] | 7.7 | 0.05 |
| nnF-CNN § [ 60 ] | ✓ | 66.6 [64.3, 68.9] | 35.2 [32.8, 37.7] | 39.6 [34.1, 45.2] | 4.0 | 0.57 |
| Curia-MAE § [ 21 ] | 64.0 [61.7, 66.3] | 33.6 [31.2, 35.9] | 50.5 [45.8, 54.8] | 4.0 | 0.57 | |
| vs VoxForge | sensitivity at FP/study | ||||||
| encoder | Dice lesion | lesion F1 | FP/study | CPM | 0.5 | 1 | 2 |
| Unified models | |||||||
| VoxForge | ref | ref | 5.0 | 36.1 | 23.1 | 34.7 | 47.3 |
| nnF-ViT [ 60 ] | 13.9 [ 15.4, 12.3] | 19.2 [ 21.7, 16.6] | 15.9 | 27.5 | 15.2 7.9 [ 13.1, 0.3] | 25.7 9.0 [ 15.2, 3.8] | 35.4 12.0 [ 17.0, 7.8] |
| nnF-CNN § [ 60 ] | 3.1 [ 4.1, 2.0] | 19.6 [ 23.1, 15.6] | 15.7 | 27.8 | 14.4 8.6 [ 13.4, 3.1] | 25.4 9.3 [ 15.0, 4.1] | 37.4 9.9 [ 14.8, 5.3] |
| Curia-MAE § [ 21 ] | 5.7 [ 6.9, 4.5] | 8.6 [ 11.0, 6.3] | 8.7 | 32.1 | 18.4 4.6 [ 7.1, 1.0] | 31.1 3.6 [ 6.6, 0.6] | 42.2 5.1 [ 8.0, 2.0] |
| dataset | encoder | 5% | 10% | 25% | 50% | 100% |
| ACDC [ 10 ] MRI, cases 8/16/40/80/160 | VoxForge | 74.7 4.7 | 83.9 0.8 | 88.1 0.7 | 89.5 1.0 | 90.9 0.9 |
| nnF-ViT [ 60 ] | 58.9 6.4 | 74.0 1.4 | 82.1 1.4 | 85.7 1.2 | 88.2 1.1 | |
| MSD Hippocampus [ 3 ] MRI, cases 10/21/52/104/208 | VoxForge | 86.9 0.4 | 87.9 0.4 | 88.4 0.2 | 88.8 0.6 | 89.3 0.5 |
| nnF-ViT [ 60 ] | 85.6 0.4 | 87.0 0.6 | 87.6 0.4 | 88.2 0.4 | 88.9 0.4 | |
| ISLES’24 CoW NCCT [ 47 ] CT, cases 6/12/30/59/118 | VoxForge | 37.8 4.5 | 41.9 5.3 | 46.3 3.3 | 50.5 3.0 | 54.1 3.3 |
| nnF-ViT [ 60 ] | 9.7 3.3 | 18.0 3.3 | 27.3 1.8 | 33.7 1.4 | 39.8 1.4 |
| 1% (3 cases) | 10% (29) | 100% (293) | |
| Unified models | |||
| VoxForge | 76.8 [71.2, 81.7] | 88.4 [84.8, 91.5] | 91.5 [88.0, 94.2] |
| nnF-ViT [ 60 ] | 70.9 [63.8, 77.3] | 84.5 [79.6, 88.5] | 89.5 [85.7, 92.6] |
| nnF-CNN [ 60 ] | 72.0 [66.1, 77.7] | 82.2 [76.3, 87.2] | 89.4 [85.6, 92.6] |
| Specialists | |||
| Merlin [ 12 ] | 70.4 [63.3, 76.6] | 83.8 [79.4, 87.7] | 88.8 [85.2, 91.9] |
| (a) ACDC, VoxForge vs. nnF-ViT | |||||||||||
| 5% (8 cases) | 25% (40) | 100% (160) | |||||||||
| structure | VoxForge | nnF-ViT | VoxForge | nnF-ViT | VoxForge | nnF-ViT | |||||
| RV | 67.3 | 52.5 | 85.9 | 77.9 | 89.8 | 85.1 | |||||
| Myocardium | 74.8 | 55.5 | 85.3 | 79.0 | 88.7 | 86.8 | |||||
| LV | 82.1 | 68.7 | 92.9 | 89.5 | 94.2 | 92.7 | |||||
| (b) TotalSegmentator CT, VoxForge vs. SPECTRE † | |||||||||||
| remaps on raw intensities | remaps on the normalized input | floor | CT noise | |||||||
| encoder | MRI | CT | PET | MRI | CT | PET | MRI | CT | PET | cosine |
| Unified models | ||||||||||
| VoxForge | 0.892 [0.88, 0.90] | 0.868 [0.86, 0.88] | 0.903 [0.90, 0.91] | 0.918 [0.91, 0.93] | 0.878 [0.87, 0.88] | 0.903 [0.90, 0.91] | 0.01 | 0.00 | 0.01 | 0.810 |
| nnF-ViT [ 60 ] | 0.680 [0.66, 0.70] | 0.667 [0.65, 0.68] | 0.545 [0.52, 0.57] | 0.644 [0.63, 0.66] | 0.629 [0.61, 0.65] | 0.545 [0.52, 0.57] | 0.00 | 0.01 | 0.06 | 0.606 |
| nnF-CNN [ 60 ] | 0.816 [0.78, 0.85] | 0.832 [0.81, 0.85] | 0.712 [0.69, 0.73] | 0.803 [0.77, 0.84] | 0.831 [0.81, 0.85] | 0.712 [0.69, 0.73] | 0.19 | 0.25 | 0.33 | 0.975 |
| Specialists | ||||||||||
| BraTS-Africa (MRI sequences) | ISLES’24 (NCCT vs. CTA) | ||||||
| descriptor | AUROC | top-1 | error (mm) | AUROC | top-1 | error (mm) | |
| VoxForge | 0.975 [0.973, 0.977] | 0.504 [0.491, 0.518] | 16 | 0.991 [0.988, 0.993] | 0.664 [0.610, 0.718] | 14 | |
| nnF-ViT [ 60 ] | 0.927 [0.921, 0.932] | 0.449 [0.432, 0.467] | 18 | 0.971 [0.964, 0.978] | 0.632 [0.572, 0.693] | 14 | |
| nnF-CNN [ 60 ] | 0.629 [0.622, 0.636] | 0.015 [0.014, 0.016] | 65 | 0.612 [0.598, 0.625] | 0.009 [0.008, 0.010] | 79 | |
| SPECTRE [ 17 ] | – | – | – | 0.920 [0.911, 0.929] | 0.168 [0.148, 0.190] | 29 | |
| VoxelFM [ 40 ] | – | – | – | 0.858 [0.848, 0.868] | 0.098 [0.087, 0.109] | 39 | |
| descriptor | T1–T1c | T1–T2 | T1–FLAIR | T1c–T2 | T1c–FLAIR | T2–FLAIR |
| VoxForge | 0.983 | 0.972 | 0.975 | 0.970 | 0.971 | 0.979 |
| nnF-ViT [ 60 ] | 0.958 | 0.895 | 0.957 | 0.895 | 0.942 | 0.914 |
| MIND-SSC | 0.948 | 0.950 | 0.939 | 0.916 | 0.917 | 0.953 |
| raw-intensity patch | 0.904 | 0.473 | 0.817 | 0.502 | 0.799 | 0.635 |
| NCCT | CTA | NCCT, 2 cells | ||||||
| encoder | CoW | LVO | CoW | LVO | mean rank | inv. norm. rank | ||
| Unified models | ||||||||
| VoxForge | 54.0 [51.2, 56.6] | 19.7 [14.2, 23.9] | 84.8 [82.4, 86.6] | 43.0 [35.6, 52.7] | 1.5 | 0.92 | ||
| nnF-ViT [ 60 ] | 39.8 [37.1, 42.4] | 15.2 [9.9, 20.5] | 82.0 [79.4, 84.1] | 17.5 [12.4, 23.6] | 4.0 | 0.50 | ||
| nnF-CNN [ 60 ] | 35.8 [33.1, 38.5] | 10.0 [5.5, 14.4] | 81.6 [78.6, 84.2] | 13.2 [8.6, 18.3] | 7.0 | 0.00 | ||
| Specialists | ||||||||
| (a) Landmark start | |||||
| method | Dice | MRI / CT | folding % | HD95 | |
| Unified models | |||||
| VoxForge features | 0.495 [0.440, 0.547] | 0.493 / 0.496 | 0.90 | 35.4 | – |
| VoxForge MIND-SSC | 0.547 [0.489, 0.601] | 0.555 / 0.540 | 0.19 | 34.6 | – |
| nnF-ViT [ 60 ] features | 0.491 [0.425, 0.552] | 0.503 / 0.478 | 0.46 | 36.0 | +0.4 [ 2.4, +3.2] |
| nnF-ViT [ 60 ] MIND-SSC | 0.530 [0.469, 0.586] | 0.543 / 0.517 | 0.25 | 36.3 | +1.7 ∗ [+0.4, +3.1] |
| (a) MRI, VoxForge | |||
| readout | IDH | grade | LLD |
| patch mean | 86.1 | 85.6 | 79.7 |
| class token | 92.3 | 92.8 | 78.8 |
| ISLES’22 | BraTS-Africa | TriALS | TriALS | |
| encoder | (MRI) | (MRI) | (all-train bank) | (3-ref bank) |
| VoxForge | 0.325 [0.227, 0.432] | 0.722 [0.602, 0.828] | 0.163 [0.109, 0.222] | 0.179 [0.125, 0.240] |
| nnF-ViT [ 60 ] | 0.088 ∗ [0.050, 0.130] | 0.174 ∗ [0.130, 0.221] | 0.067 ∗ [0.041, 0.097] | 0.064 ∗ [0.040, 0.092] |
| SPECTRE [ 17 ] | – | – | 0.092 ∗ [0.056, 0.134] | 0.255 ∗ [0.180, 0.330] |
| Triad CNN MAE [ 64 ] | 0.017 ∗ [0.011, 0.026] | 0.139 ∗ [0.090, 0.195] | – | – |
| VoxForge wins vs. nnF-ViT | 92% (45/49) | 97% (28/29) | 96% (70/73) | – |
| Dice (%) | ||||||
| dataset | encoder | NMI | win | |||
| Prostate158 | VoxForge | 30.9 | 39.9 | 51.9 | 0.17 | 100 |
| nnF-ViT | 10.8 | 12.3 | 14.4 | 0.16 | – | |
| Triad CNN MAE | 16.9 | 31.0 | 41.5 | 0.23 | – | |
| ACDC | VoxForge | 13.7 | 24.7 | 38.1 | 0.20 | 100 |
| nnF-ViT | 3.7 | 4.5 | 6.0 | 0.11 | – | |
Medical Imaging Foundation Models
Medical imaging foundation models learn reusable representations for tasks across clinical images, modalities, and institutions.
See all 258 papers