How Well Do Segmentation Foundation Models Generalize Across Structural and Functional Medical Imaging?
Organizations: Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China. · Human Phenome Institute, Fudan University, Shanghai, China. · Shanghai Universal Medical Imaging Diagnostic Center, Shanghai, China.
Abstract
Segmentation foundation models offer reusable tools for universal medical-image segmentation, but their development and evaluation have largely centered on structural CT and MRI imaging. Functional imaging such as PET provides complementary metabolic information yet remains less studied and validated. Moreover, existing evaluations often involve different datasets and tasks, making modality-associated variation difficult to distinguish from differences in segmentation targets and data distributions. To enable a more direct comparison, we introduce UMD, a whole-body multi-organ dataset comprising 490 PET/CT and 464 PET/MRI examinations with 13 annotated organs (approximately 675,000 2D slices and 12,000 3D organ annotations). UMD evaluates structural and functional images separately on corresponding organs within the same examination, reducing variation in patient anatomy and target selection. We benchmark the out-of-the-box segmentation capabilities of five representative 3D foundation models on UMD. Overall, most evaluated models show limited PET segmentation performance, and some configurations also yield near-zero overlap on CT or MRI under the evaluated workflows. These findings highlight the need to assess tool applicability across different imaging conditions and identify opportunities for cross-modality generalization. UMD provides a paired data resource and evaluation setting for this investigation.
Figures & tables
| Model | Setting | Reported Performance ∗ | UMD Evaluation Performance | |||||
|---|---|---|---|---|---|---|---|---|
| CT | MRI | PET | CT | PET | MRI | PET | ||
| SAM-Med3D-turbo | 1 point prompt | 0.790 | 0.754 | / | 0.404 | 0.410 | 0.433 | 0.397 |
| SegVol | 1 point prompt | 0.793 | / | / | 0.107 | 0.075 | 0.134 | 0.148 |
| nnInteractive | 1 point prompt | 0.55 | 0.55 | / | 0.481 | 0.342 | 0.511 | 0.217 |
| VISTA3D | class-id prompt | 0.711 | / | / | 0.838 | 0.010 | 0.022 | 0.008 |
| SAT | text prompt | 0.756 | 0.838 | 0.634 | 0.018 | 0.003 | 0.006 | 0.000 |
| Model | SAM-Med3D-turbo | SegVol | nnInteractive | VISTA3D | SAT |
|---|---|---|---|---|---|
| Liver | 0.7039(0.1155) | 0.0806(0.1982) | 0.4993(0.3319) | 0.9186(0.0378) | 0.0057(0.0205) |
| 0.6835(0.1100) | 0.1618(0.2539) | 0.3904(0.3769) | 0.0676(0.0475) | 0.0020(0.0451) | |
| Kidney-L | 0.4701(0.1300) | 0.1203(0.1967) | 0.7930(0.1636) | 0.8648(0.1007) | 0.0001(0.0018) |
| 0.5678(0.1411) | 0.0687(0.1329) | 0.4464(0.3436) | 0.0000(0.0000) | 0.0041(0.0638) | |
| Kidney-R | 0.4387(0.1491) | 0.1437(0.2279) | 0.7780(0.1753) | 0.8365(0.1184) | 0.0061(0.0780) |
| 0.5596(0.1843) | 0.0630(0.1178) | 0.4187(0.3413) | 0.0000(0.0000) | 0.0082(0.0900) |
| Model | SAM-Med3D-turbo | SegVol | nnInteractive | VISTA3D | SAT |
|---|---|---|---|---|---|
| Liver | 0.7719(0.0755) | 0.3986(0.2433) | 0.8209(0.2412) | 0.0384(0.0548) | 0.0000(0.0005) |
| 0.6259(0.2050) | 0.3731(0.2500) | 0.1225(0.2252) | 0.0749(0.0244) | 0.0000(0.0000) | |
| Kidney-L | 0.4857(0.1422) | 0.1090(0.1136) | 0.6346(0.2242) | 0.0005(0.0053) | 0.0000(0.0000) |
| 0.5666(0.1142) | 0.1047(0.0867) | 0.1679(0.2398) | 0.0000(0.0000) | 0.0000(0.0000) | |
| Kidney-R | 0.5521(0.1488) | 0.0917(0.1088) | 0.7040(0.1546) | 0.0017(0.0090) | 0.0000(0.0000) |
| 0.5708(0.1257) | 0.0884(0.0785) | 0.2225(0.2494) | 0.0000(0.0001) | 0.0000(0.0000) |
3D Medical Image Segmentation
3D Medical Image Segmentation identifies organs, lesions, and fine structures in volumetric scans for measurement and clinical planning.
See all 238 papers