Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation
Authors: Jiaqi Tang, Yu Wang, Yang Liu, Jing Tao, Fandong Zhang, Zhou Zhen, Xiahai Zhuang, Qingchao Chen, +2 more
Organizations: National Institution of Health Data Science, Peking University, Beijing, China. · Institute of Medical Technology, Peking University, Beijing, China. · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China. · Department of Cardiology, People’s Hospital of Xinjiang Uyghur Autonomous Region, Urumqi, China. · Deepwise Co., Ltd., China. · School of Data Science, Fudan University, Shanghai, China. · Xinjiang Key Laboratory of Cardiovascular Homeostasis and Regeneration Research, Urumqi, China. · The University of Hong Kong, Hong Kong, China.
Self-supervised pretraining has expanded the pool of 3D medical foundation models, but published evaluations use different datasets, endpoints and fine-tuning protocols. Models also differ in architecture, pretraining objective and data, while downstream targets range from voxel-level masks to patient-level diagnoses and outcomes. As a result, published findings provide little directly comparable evidence for choosing a model for a new task. Direct comparison of adaptation performance requires fine-tuning candidates on labelled target data, at substantial computational cost. We benchmark 23 candidates across the same 14 segmentation, classification and regression tasks spanning multiple anatomies and modalities. No candidate ranked among the top three across all segmentation tasks or across all classification and regression tasks; a convolutional encoder could outperform a transformer trained with the same objective on one task but underperform it on another. We benchmark transferability estimators against the fine-tuned rankings and find that their agreement varies across dense segmentation, classification and regression. InsightRank uses task-appropriate frozen-feature statistics and architecture-aware pairwise comparisons to rank candidates before fine-tuning. Across the 23-model pool, mean Kendall tau-b was 0.431 for segmentation and 0.352 for classification and regression. The segmentation Top 3 contained a task winner on four of nine tasks; the image-level Top 3 contained none on the five tasks, despite better full-ranking agreement than the matched conventional estimators. A three-model shortlist reduced recorded extraction-plus-training job time by 85.8% across seven segmentation tasks. Together, the benchmark provides a common reference for model selection and InsightRank offers a lower-cost way to prioritize fine-tuning, with shortlist performance that differs by target type.
Figures & tables
Figure 1: Study overview: benchmarking and transferability estimation for 3D medical imaging. a , Two complementary workflows use the same pool of 21 pretrained encoders and two scratch controls. Frozen features provide voxel-level evidence for segmentation and patient-level evidence for classification and regression, from which InsightRank produces pairwise rankings and a three-model shortlist. Separately, downstream fine-tuning of all 23 candidates across 14 tasks yields 322 three-fold mean outcomes and the reference rankings used to evaluate model selection. The reference benchmark covers the full candidate pool, not only shortlisted models. b , Indicator assignments depend on the target and encoder pair: segmentation uses class-balanced nearest-centroid accuracy ( G ), activation amplitude ( N ) and centroid margin ( B ); classification and regression use effective dimensionality ( C ) and neighbour-outcome agreement ( R ). Arrows indicate the preferred direction. Pairwise wins, ties and losses are aggregated with equal weight. c , Evaluation compares predicted and fine-tuned rankings, measures performance loss at a fixed shortlist size, and compares recorded extraction-plus-training time with exhaustive fine-tuning where timing data are available. Images, outcome curves, matrix colours, example shortlists and charts are schematic; they do not show measured predictions, survival-curve estimates or numerical results. Task-specific evaluation and timing coverage are detailed in Methods.
Figure 2: Pretraining provenance and downstream task atlas. a , The registered model pool contains 17 OpenMind-derived checkpoints (14 trained with the full collection, two using T1-weighted or image-quality-filtered subsets, and one DINO-derived), four external pretrained-weight models and two randomly initialized controls. ViT-VoCo inherits weights from the VoCo GitHub repository rather than OpenMind; the exact upstream checkpoint variant used here was not independently verified. Counts denote model candidates, not pretraining images. b , The 14 downstream datasets are grouped by anatomical target: nine brain, one head-and-neck, three heart or coronary, and one kidney task. TopCoW is assigned to the brain group because it segments the intracranial Circle of Willis. Card colour denotes segmentation (teal), classification (ochre) or regression (violet); each card identifies its target and imaging input. The coronary-plaque and coronary bridge-vessel CTA datasets are private, hospital-curated in-house datasets, as indicated on their cards. Plaque annotations remain under quality review (†). For the provisional bridge-vessel task (‡), 44 of 546 series lacked matched masks and were encoded as all-background labels in training and evaluation rather than excluded. c , All 23 candidates have three-fold downstream references on all 14 tasks (322 model–task means), and frozen-feature scoring covers the same candidates on nine segmentation and five image-level tasks.
Figure 3: Cross-task benchmark of 3D medical model candidates. a , Ranks of three-fold mean downstream outcomes for 23 candidates across nine segmentation and five classification or regression tasks. Numbers show within-task ranks; colour shows (r−1)/(M−1) , with rank r and candidate count M=23 . Outlines identify winners. Scratch models are non-pretrained controls. b,c , Maximum task coverage of any single fixed model at rank threshold k . The maximizing model may change with k , so each curve is an optimistic envelope rather than one model’s performance. Dagger and double dagger mark the coronary annotation qualifications detailed in Methods.
Figure 4: Comparison-dependent evidence for spatial targets. a , Three feature statistics assigned to model-pair types, followed by the shared net-win aggregation. b , Weighted Kendall correlations for the complete pool and within-architecture rankings across nine tasks. c , Pairwise concordance of each statistic, averaged across tasks; outlines mark fixed assignments. Predicted ties count as half agreement and 0.5 denotes chance-direction agreement. d , Top 1 and Top 3 regret in Dice percentage points. All candidates use the same task-specific probe and three-fold reference. Both coronary annotation qualifications are detailed in Methods.
Estimator
Kendall τ -b
Spearman ρ
Weighted Kendall τ
Top 3 hit / 5
Top 1 regret
InsightRank
0.3521
0.4654
0.3099
0/5
0.3118
LogME, deepest layer
0.0545
0.0700
0.0167
0/5
0.4834
LogME, four-layer mean
0.1051
0.1202
0.0700
0/5
0.3999
H-score, deepest layer
-0.0245
-0.0128
-0.0582
1/5
0.4956
GBC, deepest layer
-0.1889
-0.2994
-0.2044
0/5
0.6320
NLEEP, deepest layer
-0.0372
-0.0429
-0.0143
0/5
0.4424
Table 1: Image-level transferability results for the full 23-candidate benchmark across five classification and regression tasks. Correlations and regret are task-equal means. Top 3 hit counts tasks in which the predicted shortlist contains a best fine-tuned model; Top 1 regret is normalized by the observed performance range within each task. All methods use the same candidate pool. These tasks contributed to rule development, so the results are not independent task validation; values are descriptive and no hypothesis tests were performed.
Method
Weighted τ
τ -b
ρ
Top-3 hit
Top-1 regret
Top-3 regret
InsightRank
0.5492
0.4328
0.5623
3/4
0.443
0.160
Same-indicator global vote
0.4342
0.4798
0.6541
1/4
1.808
1.227
Fixed cross-task recommendation
0.3711
0.4901
0.6396
0/4
3.574
1.126
CC-FV, diagonal
0.3654
0.4466
0.6040
0/4
2.165
1.253
LogME
-0.0629
-0.1067
-0.1487
1/4
4.997
1.526
GBC, diagonal
-0.0522
-0.1877
-0.2720
2/4
3.745
1.121
Table 2: Controlled model selection in the predefined four-task spatial-target comparison. All methods rank the same 23 candidates. Correlations are task means; regret is in Dice percentage points. Two tasks retain annotation qualifications. Task identities, evaluation roles and comparator implementations are specified in Methods.
Figure 5: Selection loss at a matched training budget. a , Mean regret versus shortlist size in the predefined four-task comparison. The random curve is an exact expectation over uniformly sampled subsets. The fixed recommendation uses ranks from other tasks only, as specified in Methods. b , Task-level Top 3 regret. All methods use the same downstream reference; no additional models were trained. Both coronary tasks retain their annotation qualifications.
ID
Model
Encoder
Objective
Registered source
745
OpenMind-T1-MAE
ResEncL
MAE
OpenMind T1
752
OpenMind-HighIQS-MAE
ResEncL
MAE
OpenMind HighIQS
754
CNN-MAE
ResEncL
MAE
OpenMind Full
755
CNN-MG
ResEncL
Models Genesis
OpenMind Full
756
CNN-S3D
ResEncL
S3D
OpenMind Full
757
CNN-SimCLR
ResEncL
SimCLR
OpenMind Full
Table 3: The 23 model candidates. Source and objective labels follow the frozen experiment registry; published-weight entries identify model families rather than independently verified checkpoint releases. Scratch denotes random initialization.
The growing number of medical vision foundation models highlights the need for effective model selection. However, mainstream selection methods rely on exhaustive fine-tuning, which is computationally expensive. Most of the existing Transferability Estimation (TE) metrics are primarily designed for image-level classification. They fail to preserve spatial relationships and fine-grained boundary details, which are crucial for the segmentation task. Additionally, while image-level tasks typically process a single feature vector per input, dense prediction tasks in 3D medical imaging require voxel-wise evaluation against dense annotations. To bridge these gaps, we propose a non-parametric, topology-driven framework that estimates transferability directly from the alignment between the sparse 1-skeleton graph of dense features and semantic labels via Minimum Spanning Trees (MST). We decouple the alignment into two complementary geometric scales: Local Boundary-Aware Topological Consistency (LBTC) to assess boundary separability, where we prove that the MST leakage rate serves as a finite-sample lower bound on the Bayes error; and Global Representation Topology Divergence (GRTD) to evaluate the overall anatomical layout. Crucially, we formally justify a counterintuitive mechanism: Although without fine-tuning, the randomly initialized segmentation decoder acts as a topology-preserving spatial projector, reducing the variance of pairwise distance estimates and stabilizing global alignment evaluation. Fused via a task-adaptive gating mechanism, these dual metrics adapt to diverse clinical complexities. Evaluated on a large-scale benchmark of 114,000 3D medical volumes across diverse anatomical tasks, our topological framework achieves state-of-the-art transferability estimation with an average weighted Kendall (outperforming by 0.36) while accelerating evaluation by 56 times.
Jiaqi Tang, Shaoyang Zhang, Fandong Zhang +3
National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China +2
The advent of large-scale self-supervised learning (SSL) has produced a vast zoo of medical foundation models. However, selecting optimal medical foundation models for specific segmentation tasks remains a computational bottleneck. Existing Transferability Estimation (TE) metrics, primarily designed for classification, rely on global statistical assumptions and fail to capture the topological complexity essential for dense prediction. We propose a novel Topology-Driven Transferability Estimation framework that evaluates manifold tractability rather than statistical overlap. Our approach introduces three components: (1) Global Representation Topology Divergence (GRTD), utilizing Minimum Spanning Trees to quantify feature-label structural isomorphism; (2) Local Boundary-Aware Topological Consistency (LBTC), which assesses manifold separability specifically at critical anatomical boundaries; and (3) Task-Adaptive Fusion, which dynamically integrates global and local metrics based on the semantic cardinality of the target task. Validated on the large-scale OpenMind benchmark across diverse anatomical targets and SSL foundation models, our approach significantly outperforms state-of-the-art baselines by around 31% relative improvement in the weighted Kendall metric, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning. The code will be made publicly available upon acceptance.
Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang +3
Peking University, Beijing, China · National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China +3
Foundation models have transformed vision and language by learning general-purpose representations from large-scale unlabeled data, yet 3D medical imaging lacks analogous approaches. Existing self-supervised methods rely on low-level reconstruction or contrastive objectives that fail to capture the anatomical semantics critical for medical image analysis, limiting transfer to downstream tasks. We present MASS (MAsk-guided Self-Supervised learning), which treats in-context segmentation as the pretext task for learning general-purpose medical imaging representations. MASS's key insight is that automatically generated class-agnostic masks provide sufficient structural supervision for learning semantically rich representations. By training on thousands of diverse mask proposals spanning anatomical structures and pathological findings, MASS learns what semantically defines medical structures: the holistic combination of appearance, shape, spatial context, and anatomical relationships. We demonstrate effectiveness across data regimes: from small-scale pretraining on individual datasets (20-200 scans) to large-scale multi-modal pretraining on 5K CT, MRI, and PET volumes, all without annotations. MASS demonstrates: (i) few-shot segmentation on novel structures, (ii) matching full supervision with only 20-40% labeled data while outperforming self-supervised baselines by over 20 in Dice score in low-data regimes, and (iii) frozen-encoder classification on unseen pathologies that matches full supervised training with thousands of samples. Mask-guided self-supervised pretraining captures broadly generalizable knowledge, opening a path toward 3D medical imaging foundation models without expert annotations. Code is available: https://github.com/Stanford-AIMI/MASS.