Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation

Jiaqi Tang; Yu Wang; Yang Liu; Jing Tao; Fandong Zhang; Zhou Zhen; Xiahai Zhuang; Qingchao Chen

cs.CVOct 13, 2026

Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation

Authors: Jiaqi Tang, Yu Wang, Yang Liu, Jing Tao, Fandong Zhang, Zhou Zhen, Xiahai Zhuang, Qingchao Chen, +2 more

Organizations: National Institution of Health Data Science, Peking University, Beijing, China. · Institute of Medical Technology, Peking University, Beijing, China. · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China. · Department of Cardiology, People’s Hospital of Xinjiang Uyghur Autonomous Region, Urumqi, China. · Deepwise Co., Ltd., China. · School of Data Science, Fudan University, Shanghai, China. · Xinjiang Key Laboratory of Cardiovascular Homeostasis and Regeneration Research, Urumqi, China. · The University of Hong Kong, Hong Kong, China.

Abstract

Self-supervised pretraining has expanded the pool of 3D medical foundation models, but published evaluations use different datasets, endpoints and fine-tuning protocols. Models also differ in architecture, pretraining objective and data, while downstream targets range from voxel-level masks to patient-level diagnoses and outcomes. As a result, published findings provide little directly comparable evidence for choosing a model for a new task. Direct comparison of adaptation performance requires fine-tuning candidates on labelled target data, at substantial computational cost. We benchmark 23 candidates across the same 14 segmentation, classification and regression tasks spanning multiple anatomies and modalities. No candidate ranked among the top three across all segmentation tasks or across all classification and regression tasks; a convolutional encoder could outperform a transformer trained with the same objective on one task but underperform it on another. We benchmark transferability estimators against the fine-tuned rankings and find that their agreement varies across dense segmentation, classification and regression. InsightRank uses task-appropriate frozen-feature statistics and architecture-aware pairwise comparisons to rank candidates before fine-tuning. Across the 23-model pool, mean Kendall tau-b was 0.431 for segmentation and 0.352 for classification and regression. The segmentation Top 3 contained a task winner on four of nine tasks; the image-level Top 3 contained none on the five tasks, despite better full-ranking agreement than the matched conventional estimators. A three-model shortlist reduced recorded extraction-plus-training job time by 85.8% across seven segmentation tasks. Together, the benchmark provides a common reference for model selection and InsightRank offers a lower-cost way to prioritize fine-tuning, with shortlist performance that differs by target type.

Figures & tables

Figure 1: Study overview: benchmarking and transferability estimation for 3D medical imaging. a , Two complementary workflows use the same pool of 21 pretrained encoders and two scratch controls. Frozen features provide voxel-level evidence for segmentation and patient-level evidence for classification and regression, from which InsightRank produces pairwise rankings and a three-model shortlist. Separately, downstream fine-tuning of all 23 candidates across 14 tasks yields 322 three-fold mean outcomes and the reference rankings used to evaluate model selection. The reference benchmark covers the full candidate pool, not only shortlisted models. b , Indicator assignments depend on the target and encoder pair: segmentation uses class-balanced nearest-centroid accuracy ( $G$ ), activation amplitude ( $N$ ) and centroid margin ( $B$ ); classification and regression use effective dimensionality ( $C$ ) and neighbour-outcome agreement ( $R$ ). Arrows indicate the preferred direction. Pairwise wins, ties and losses are aggregated with equal weight. c , Evaluation compares predicted and fine-tuned rankings, measures performance loss at a fixed shortlist size, and compares recorded extraction-plus-training time with exhaustive fine-tuning where timing data are available. Images, outcome curves, matrix colours, example shortlists and charts are schematic; they do not show measured predictions, survival-curve estimates or numerical results. Task-specific evaluation and timing coverage are detailed in Methods.

Figure 2: Pretraining provenance and downstream task atlas. a , The registered model pool contains 17 OpenMind-derived checkpoints (14 trained with the full collection, two using T1-weighted or image-quality-filtered subsets, and one DINO-derived), four external pretrained-weight models and two randomly initialized controls. ViT-VoCo inherits weights from the VoCo GitHub repository rather than OpenMind; the exact upstream checkpoint variant used here was not independently verified. Counts denote model candidates, not pretraining images. b , The 14 downstream datasets are grouped by anatomical target: nine brain, one head-and-neck, three heart or coronary, and one kidney task. TopCoW is assigned to the brain group because it segments the intracranial Circle of Willis. Card colour denotes segmentation (teal), classification (ochre) or regression (violet); each card identifies its target and imaging input. The coronary-plaque and coronary bridge-vessel CTA datasets are private, hospital-curated in-house datasets, as indicated on their cards. Plaque annotations remain under quality review (†). For the provisional bridge-vessel task (‡), 44 of 546 series lacked matched masks and were encoded as all-background labels in training and evaluation rather than excluded. c , All 23 candidates have three-fold downstream references on all 14 tasks (322 model–task means), and frozen-feature scoring covers the same candidates on nine segmentation and five image-level tasks.

Figure 3: Cross-task benchmark of 3D medical model candidates. a , Ranks of three-fold mean downstream outcomes for 23 candidates across nine segmentation and five classification or regression tasks. Numbers show within-task ranks; colour shows $(r-1)/(M-1)$ , with rank $r$ and candidate count $M=23$ . Outlines identify winners. Scratch models are non-pretrained controls. b,c , Maximum task coverage of any single fixed model at rank threshold $k$ . The maximizing model may change with $k$ , so each curve is an optimistic envelope rather than one model’s performance. Dagger and double dagger mark the coronary annotation qualifications detailed in Methods.

Figure 4: Comparison-dependent evidence for spatial targets. a , Three feature statistics assigned to model-pair types, followed by the shared net-win aggregation. b , Weighted Kendall correlations for the complete pool and within-architecture rankings across nine tasks. c , Pairwise concordance of each statistic, averaged across tasks; outlines mark fixed assignments. Predicted ties count as half agreement and 0.5 denotes chance-direction agreement. d , Top 1 and Top 3 regret in Dice percentage points. All candidates use the same task-specific probe and three-fold reference. Both coronary annotation qualifications are detailed in Methods.

Estimator	Kendall $\tau$ -b	Spearman $\rho$	Weighted Kendall $\tau$	Top 3 hit / 5	Top 1 regret
InsightRank	0.3521	0.4654	0.3099	0/5	0.3118
LogME, deepest layer	0.0545	0.0700	0.0167	0/5	0.4834
LogME, four-layer mean	0.1051	0.1202	0.0700	0/5	0.3999
H-score, deepest layer	-0.0245	-0.0128	-0.0582	1/5	0.4956
GBC, deepest layer	-0.1889	-0.2994	-0.2044	0/5	0.6320
NLEEP, deepest layer	-0.0372	-0.0429	-0.0143	0/5	0.4424

Table 1: Image-level transferability results for the full 23-candidate benchmark across five classification and regression tasks. Correlations and regret are task-equal means. Top 3 hit counts tasks in which the predicted shortlist contains a best fine-tuned model; Top 1 regret is normalized by the observed performance range within each task. All methods use the same candidate pool. These tasks contributed to rule development, so the results are not independent task validation; values are descriptive and no hypothesis tests were performed.

Method	Weighted $\tau$	$\tau$ -b	$\rho$	Top-3 hit	Top-1 regret	Top-3 regret
InsightRank	0.5492	0.4328	0.5623	3/4	0.443	0.160
Same-indicator global vote	0.4342	0.4798	0.6541	1/4	1.808	1.227
Fixed cross-task recommendation	0.3711	0.4901	0.6396	0/4	3.574	1.126
CC-FV, diagonal	0.3654	0.4466	0.6040	0/4	2.165	1.253
LogME	-0.0629	-0.1067	-0.1487	1/4	4.997	1.526
GBC, diagonal	-0.0522	-0.1877	-0.2720	2/4	3.745	1.121

Table 2: Controlled model selection in the predefined four-task spatial-target comparison. All methods rank the same 23 candidates. Correlations are task means; regret is in Dice percentage points. Two tasks retain annotation qualifications. Task identities, evaluation roles and comparator implementations are specified in Methods.

Figure 5: Selection loss at a matched training budget. a , Mean regret versus shortlist size in the predefined four-task comparison. The random curve is an exact expectation over uniformly sampled subsets. The fixed recommendation uses ranks from other tasks only, as specified in Methods. b , Task-level Top 3 regret. All methods use the same downstream reference; no additional models were trained. Both coronary tasks retain their annotation qualifications.

ID	Model	Encoder	Objective	Registered source
745	OpenMind-T1-MAE	ResEncL	MAE	OpenMind T1
752	OpenMind-HighIQS-MAE	ResEncL	MAE	OpenMind HighIQS
754	CNN-MAE	ResEncL	MAE	OpenMind Full
755	CNN-MG	ResEncL	Models Genesis	OpenMind Full
756	CNN-S3D	ResEncL	S3D	OpenMind Full
757	CNN-SimCLR	ResEncL	SimCLR	OpenMind Full

Table 3: The 23 model candidates. Source and objective labels follow the frozen experiment registry; published-weight entries identify model families rather than independently verified checkpoint releases. Scratch denotes random initialization.

Explore similar work

Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models
Jul 5, 2026Jiaqi Tang, Shaoyang Zhang, Fandong Zhang +3Medical Imaging Foundation Models 3D Medical Imaging
National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China +2
Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation
Feb 27, 2026Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang +3Task Transferability Estimation Medical Imaging Foundation Models
Peking University, Beijing, China · National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China +3
Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision
Mar 14, 2026Yunhe Gao, Yabin Zhang, Chong Wang +5Medical Image Representation Learning Self-Supervised Pre-Training
Stanford University

Jul 5, 2026J/K move · Enter open · S save

Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models

Jiaqi Tang, Shaoyang Zhang, Fandong Zhang, Shu Zhang+2

National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China+2

Medical Imaging Foundation Models 3D Medical Imaging Task Transferability Estimation Transfer Learning

Open paper AskPDF arXiv

The growing number of medical vision foundation models highlights the need for effective model selection. However, mainstream selection methods rely on exhaustive fine-tuning, which is computationally expensive. Most of the existing Transferability Estimation (TE) metrics are primarily designed for image-level classification. They fail to preserve spatial relationships and fine-grained boundary details, which are crucial for the segmentation task. Additionally, while image-level tasks typically process a single feature vector per input, dense prediction tasks in 3D medical imaging require voxel-wise evaluation against dense annotations. To bridge these gaps, we propose a non-parametric, topology-driven framework that estimates transferability directly from the alignment between the sparse 1-skeleton graph of dense features and semantic labels via Minimum Spanning Trees (MST). We decouple the alignment into two complementary geometric scales: Local Boundary-Aware Topological Consistency (LBTC) to assess boundary separability, where we prove that the MST leakage rate serves as a finite-sample lower bound on the Bayes error; and Global Representation Topology Divergence (GRTD) to evaluate the overall anatomical layout. Crucially, we formally justify a counterintuitive mechanism: Although without fine-tuning, the randomly initialized segmentation decoder acts as a topology-preserving spatial projector, reducing the variance of pairwise distance estimates and stabilizing global alignment evaluation. Fused via a task-adaptive gating mechanism, these dual metrics adapt to diverse clinical complexities. Evaluated on a large-scale benchmark of 114,000 3D medical volumes across diverse anatomical tasks, our topological framework achieves state-of-the-art transferability estimation with an average weighted Kendall (outperforming by 0.36) while accelerating evaluation by 56 times.

Abstract

Explore similar work

Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models

Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision