REFORM-3D: A Representation-Centric Evaluation Framework for 3D Medical Vision Foundation Models
Organizations: Department of Computer Science, Khalifa University, Abu Dhabi, UAE · University of Western Australia
Abstract
Evaluation of 3D medical foundation models is typically dominated by downstream task scores, which entangle representation quality with adaptation choices. This is especially critical in volumetric imaging, where spacing and anisotropy can substantially alter image statistics while the anatomy itself remains unchanged. We therefore introduce REFORM-3D, a Representation-centric Evaluation FramewORk for benchmarking frozen 3D medical vision foundation Models across three questions: Acquisition Robustness, asking whether features remain stable when spacing varies while anatomy is fixed; Cross-Modal Anatomical Alignment, asking whether labeled anatomical structures remain recoverable across modalities under balanced bidirectional retrieval; and Anatomical Transfer, asking whether organ identity remains recoverable from limited labeled support under case-disjoint evaluation. These are also pretraining questions, because what a 3D encoder learns depends on how volumes are geometrically presented during self-supervised pretraining. We further introduce Med3DINO encoder family, pretrained on K unlabeled MR, CT, PET, CTA, and CBCT volumes under three explicit geometric assumptions: isotropic normalization, voxel-relative sampling, and spacing-aware resampling. Across Med3DINO and heterogeneous external encoders, REFORM-3D reveals module-specific strengths rather than a single model ordering. Within the controlled Med3DINO comparison, voxel-relative sampling yields lower embedding drift, whereas spacing-aware resampling favors anatomical transfer; semantic-transfer advantages depend on the dataset and feature type. Project link: REFORM-3D.
Figures & tables
| Pretraining exposure | Dataset | Evaluation role |
| Seen in pretraining | AbdomenAtlas [ 37 ] ; ImageCAS [ 80 ] ; JHU-Stroke Brain MR [ 42 ] ; TotalSegmentator-CT [ 72 ] ; HECKTOR-2025 [ 58 ] ; | Acquisition Robustness; CT contribution to CT–MR alignment and anatomical transfer; CT–PET lesion-category retrieval |
| Unseen in pretraining | KiTS23 [ 25 ] ; TotalSegmentator-MR [ 1 ] ; CHAOS [ 33 ] ; AMOS22 [ 29 ] ; MMWHS [ 85 ] | Acquisition Robustness; CT–MR alignment; anatomical transfer |
| Model | Encoder | Evaluated feature(s) | Modalities | Volumes | Pretraining objective | Pretraining geometry |
| External Encoders | ||||||
| BrainIAC [ 66 ] | ViT-B | First patch token | MR | 32.0K | SimCLR | Isotropic normalization |
| OpenMind [ 69 ] | Primus-M | GAP | MR/PET † | 113.9K | Volume contrastive | Isotropic normalization |
| VoCo [ 73 ] | SwinUNETR | Multiscale GAP | CT | 160K | Volume contrastive | Region-specific resampling |
| 3DINO [ 75 ] | ViT-L | CLS / GAP | CT/MR/PET † | 98.8K | DINOv2 | Voxel-relative sampling |
| SAM-Med3D [ 70 ] | ViT-B | Post-neck GAP | CT/MR | 22K | Supervised Dice + CE | Isotropic normalization |
| Method | AbdomenAtlas [ 37 ] | KiTS23 [ 25 ] | ImageCAS [ 80 ] | TS-MR [ 1 ] | ||||||||
| Drift | CKA | Trans | Drift | CKA | Trans | Drift | CKA | Trans | Drift | CKA | Trans | |
| External encoders | ||||||||||||
| BrainIAC [ 66 ] | 0.30 | 97.60 | 67.20 | 0.30 | 96.70 | 46.60 | 0.90 | 93.90 | 52.50 | 0.50 | 99.10 | 55.70 |
| OpenMind [ 69 ] | 0.10 | 77.70 | 57.10 | 1.20 | 80.70 | 46.70 | 0.20 | 53.00 | 50.10 | 0.20 | 92.60 | 49.30 |
| VoCo [ 73 ] | 0.80 | 86.70 | 74.60 | 0.70 | 81.30 | 46.10 | 2.30 | 42.60 | 52.00 | 0.10 | 95.94 | 71.10 |
| 3DINO (CLS) [ 75 ] | 16.30 | 87.50 | 75.10 | 15.40 | 77.30 | 45.50 | 32.10 | 83.40 | 55.90 | 6.00 | 92.40 | 74.60 |
| Method | AbdomenAtlas [ 37 ] | KiTS23 [ 25 ] | TotalSegmentator-MR [ 1 ] | |||
| CKA | Trans | CKA | Trans | CKA | Trans | |
| Model-Specific Features (CLS for 3DINO and Med3DINO) | ||||||
| BrainIAC [ 66 ] | 0.50 | 61.20 | 2.70 | 45.90 | 2.70 | 51.60 |
| OpenMind [ 69 ] | 0.30 | 52.30 | 3.70 | 51.00 | 4.00 | 48.60 |
| VoCo [ 73 ] | 0.60 | 66.10 | 6.10 | 49.90 | 6.60 | 59.70 |
| 3DINO [ 75 ] | 14.10 | 66.40 | 52.00 | 49.80 | 49.20 | 65.90 |
| Method | TotalSeg CT–MR [ 1 , 72 ] | MMWHS [ 85 ] | AMOS22 [ 29 ] | CHAOS [ 33 ] (liver-only) | ||||||||
| mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | mAP | Top-1 | Top-5 | |
| External encoders | ||||||||||||
| BrainIAC [ 66 ] | 9.90 | 11.70 | 32.50 | 21.60 | 19.30 | 38.90 | 10.50 | 12.60 | 31.90 | 38.60 | 18.80 | 62.50 |
| OpenMind [ 69 ] | 10.50 | 9.60 | 34.30 | 17.60 | 15.00 | 43.90 | 10.80 | 12.30 | 39.70 | 32.70 | 12.50 | 56.20 |
| VoCo [ 73 ] | 11.40 | 18.90 | 49.90 | 19.30 | 13.20 | 25.40 | 16.30 | 25.70 | 50.30 | 32.40 | 6.20 | 68.80 |
| 3DINO (CLS) [ 75 ] | 17.60 | 32.20 | 60.60 | 29.40 | 40.70 | 77.10 | 21.10 | 27.40 | 57.10 | 24.60 | 6.20 | 50.00 |
| Method | TotalSeg CT–MR [ 1 , 72 ] | MMWHS [ 85 ] | AMOS22 [ 29 ] | ||||||
| BA centroid | BA probe | BA centroid | BA probe | BA centroid | BA probe | ||||
| External encoders | |||||||||
| BrainIAC [ 66 ] | 17.00 | 20.40 | 3.40 | 35.30 | 42.50 | 7.10 | 16.70 | 21.60 | 4.90 |
| OpenMind [ 69 ] | 14.50 | 14.30 | -0.20 | 23.40 | 24.60 | 1.20 | 18.80 | 18.50 | -0.20 |
| VoCo [ 73 ] | 25.80 | 36.10 | 10.20 | 29.20 | 47.60 | 18.40 | 37.10 | 52.70 | 15.60 |
| 3DINO (CLS) [ 75 ] | 47.20 | 47.70 | 0.50 | 57.40 | 61.60 | 4.20 | 57.60 | 60.40 | 2.80 |
| Method | Seen datasets | Unseen datasets | Unseen modality | |||||||||||||||||
| BTCV [ 35 ] | BraTS-GLI [ 4 ] | ISLES22 [ 26 ] | WORD [ 45 ] | TS-CT [ 72 ] | AMOS-CT [ 29 ] | AMOS-MR [ 29 ] | KiTS23 [ 25 ] | LA-SEG [ 74 ] | TDSC-ABUS [ 44 ] | |||||||||||
| Dice | HD95 | Dice | HD95 | Dice | HD95 | Dice | HD95 | Dice | HD95 | Dice | HD95 | Dice | HD95 | Dice | HD95 | Dice | HD95 | Dice | HD95 | |
| MedDINOv3 [ 38 ] | 78.79 | – | – | – | – | – | – | – | – | – | 87.38 | – | – | – | 70.68 | – | – | – | – | – |
| 3DINO [ 75 ] | 82.10 | 9.4 | 91.80 | 3.90 | 63.39 | 11.08 | 85.46 | 6.83 | 86.78 | 5.83 | 85.56 | 5.01 | 80.69 | 5.54 | 80.75 | 13.87 | 92.40 | 5.20 | 47.70 | 89.10 |
| Med3DINO-ISO | 64.71 | 15.56 | 91.70 | 3.46 | 64.98 | 9.87 | 86.10 | 7.50 | 86.94 | 4.96 | 83.35 | 4.99 | 82.12 | 4.69 | 88.00 | 19.61 | 95.69 | 1.96 | 72.78 | 75.61 |
| Med3DINO-REL | 65.50 | 7.94 | 91.77 | 3.54 | 67.76 | 9.37 | 86.65 | 6.30 | 88.79 | 3.83 | 85.46 | 4.28 | 82.07 | 5.58 | 88.11 | 14.57 | 95.17 | 1.97 | 74.94 | 68.72 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Modality | Volumes | Share (%) |
| CT | 34,052 | 54.4 |
| MR | 23,245 | 37.1 |
| PET | 3,368 | 5.4 |
| CTA | 1,070 | 1.7 |
| CBCT | 903 | 1.4 |
| Total Corpus | 62,638 | 100% |
| Component | Controlled setting |
| Shared data and model | 62,638-volume corpus; initialization policy; ViT-L/16. |
| Shared optimization | DINO/iBOT/KoLeo; non-geometric augmentations; crop counts and output sizes; batch size; AdamW schedule; 125K updates. |
| ISO geometry | Resampling to mm followed by voxel-relative crops. |
| REL geometry | Retained anisotropy with voxel-relative crops. |
| SA geometry | Retained anisotropy with physical through-plane crop extents and anisotropy-aware rotation handling. |
| Acquisition Robustness | |||
| Dataset | Full-manifest scans | Sampled scans | Controlled scans |
| AbdomenAtlas [ 37 ] | 9,208 | 3,000 | 1,000 |
| KiTS23 [ 25 ] | 489 | 489 | 69 |
| ImageCAS [ 80 ] | 1,000 | 1,000 | 708 |
| TotalSegmentator-MR [ 1 ] | 616 | 616 | 131 |
| JHU-Stroke [ 42 ] | 1,853 | 1,138 | 131 |
| Dataset | Target definition |
| AbdomenAtlas | Binary presence of eligible structures from the annotation-label mapping. |
| KiTS23 | Kidney, tumor, and cyst presence candidates; only cyst presence is informative in the stored controlled cohort. |
| ImageCAS | One-versus-rest indicators for coronary-mask foreground-volume tertiles. |
| TS-MR | Structure presence from the individual segmentation masks. |
| JHU-Stroke | One-versus-rest indicators for lesion-mask foreground-volume tertiles. |
| Method | JHU-Stroke [ 42 ] | ||
| Drift | CKA | Semantic Transfer | |
| External encoders | |||
| BrainIAC [ 66 ] | 1.30 | 99.80 | 59.90 |
| OpenMind [ 69 ] | 0.00 | 97.70 | 54.41 |
| VoCo [ 73 ] | 0.10 | 85.72 | 66.40 |
| 3DINO [ 75 ] (CLS) | 1.43 | 95.30 | 70.73 |
| Comparison | AbdomenAtlas [ 37 ] | KiTS23 [ 25 ] | ImageCAS [ 80 ] | TotalSegmentator-MR [ 1 ] | JHU-Stroke [ 42 ] |
| Drift — CLS | |||||
| REL vs. ISO | +7.31 [6.96, 7.65] | +4.00 [3.03, 4.98] | -3.81 [-4.16, -3.47] | +0.33 [-0.19, 0.85] | +1.45 [1.26, 1.64] |
| SA vs. ISO | +0.09 [-0.25, 0.42] | -4.22 [-5.26, -3.17] | -18.45 [-18.90, -18.01] | +0.28 [-0.08, 0.65] | -3.43 [-3.69, -3.18] |
| SA vs. REL | -7.22 [-7.41, -7.02] | -8.22 [-8.83, -7.61] | -14.64 [-15.09, -14.20] | -0.05 [-0.54, 0.44] | -4.88 [-5.05, -4.72] |
| CKA — CLS | |||||
| REL vs. ISO | +1.61 [1.39, 1.82] | -0.21 [-0.54, 0.13] | -10.36 [-11.11, -9.61] | -0.04 [-0.19, 0.11] | +0.68 [0.58, 0.78] |
| Dataset | Feature | Comparison | Logistic (pp) [CI] | Nearest centroid (pp) [CI] |
| Semantic transfer | ||||
| AbdomenAtlas | CLS | REL–ISO | ||
| SA–ISO | ||||
| SA–REL | ||||
| AbdomenAtlas | GAP | REL–ISO | ||
| SA–ISO | ||||
| Support | 3DINO/REL/SA probe BA | SA highest mean | Query cases: TS/MMWHS/AMOS22 |
| 1 | 54.8/55.3/56.4 | 1/3 | 85/19/54 |
| 2 | 64.7/66.1/68.7 | 3/3 | 84/18/53 |
| 4 | 71.2/75.1/77.5 | 3/3 | 82/16/51 |
| 8 | 76.9/80.0/81.5 | 3/3 | 77–78/12/47 |
| Surface | mAP: ISO/REL/SA | SA–REL | Top-1: ISO/REL/SA | SA–REL |
| M5 + context | 15.3/33.2/42.7 | 14.8/44.5/60.5 | ||
| M0 + context | 15.5/31.5/40.4 | 12.9/38.7/52.1 | ||
| M5 + organ-only | 16.7/24.0/26.4 | 18.6/27.9/31.8 | ||
| M0 + organ-only | 17.8/22.2/23.7 | 18.9/31.3/32.5 |
| Controlled change | Mean Kendall | Reference leader retained |
| Disable anti-aliasing | 0.82 | 51/54 (94%) |
| Trilinear nearest | 0.73 | 48/54 (89%) |
| Centered fixed physical FOV | 0.70 | 46/54 (85%) |
| Fine-organ organ-family labels | 0.78 | 5/6 (83%) |
| Organ-only M5 1-mm mask erosion | 0.89 | 5/6 (83%) |
| Acquisition: controlled spacing severity | |||
| Condition | Drift | CKA | Transfer: signed SA–REL effect |
| Moderate | REL | REL | AbdomenAtlas ; KiTS23 ; TS-MR ; ImageCAS ; JHU |
| Severe | REL | REL | AbdomenAtlas ; KiTS23 ; TS-MR ; ImageCAS ; JHU |
| Retrieval: modality direction and organ-level counts | |||
| Dataset | CT MR mAP | MR CT mAP | |
| AMOS22 | SA ; SA higher on 8/13 organs | SA ; SA higher on 10/13 organs | |
| AMOS22 | MMWHS | ||||
| Organ | CT MR | MR CT | Organ | CT MR | MR CT |
| Adrenal gland left | Ascending aorta | ||||
| Adrenal gland right | Left atrium | ||||
| Aorta | Left ventricle | ||||
| Duodenum | Myocardium | ||||
| Esophagus | Pulmonary artery | ||||
| Metric | ISO/REL/SA mean rank | SA–REL outcomes |
| Acquisition drift | 2.5/1.0/2.5 | REL 1; NS 1 |
| Acquisition CKA | 1.5/2.5/2.0 | REL 1; SA 1 |
| Acquisition transfer | 2.5/2.5/1.0 | SA 2 |
| Cross-modal mAP | 3.0/2.0/1.0 | SA 2 |
| Cross-modal Top-1 | 3.0/2.0/1.0 | SA 2 |
| Linear probe BA | 3.0/2.0/1.0 | SA 2 |
| Feature | ISO (mAP/Top-1) | REL (mAP/Top-1) | SA (mAP/Top-1) |
| CLS | / | / | / |
| GAP | / | / | / |
| Resource | Repository artifact | Role |
| Data and cohorts | docs/data.md ; src/reform3d/prepare.py | Dataset inputs, manifest preparation, and cohort validation. |
| Evaluation | src/reform3d/cli.py ; configs/ | Configured input validation and module execution. |
| Encoders | reproduction/checkpoints.json ; docs/encoders.md | Checkpoint identities, available weight hashes, interfaces, and dependencies. |
| Environment | requirements.txt ; pyproject.toml | Tested dependency versions and package installation. |
| Usage | docs/usage.md | Configuration, cache reuse, outputs, and failure handling. |
Medical Imaging Foundation Models
Medical imaging foundation models learn reusable representations for tasks across clinical images, modalities, and institutions.
See all 253 papers