Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

Jiaqi Tang; Shaoyang Zhang; Xiaoqi Wang; Jiaying Zhou; Yang Liu; Qingchao Chen

cs.CVFeb 27, 2026

Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

Authors: Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang, Jiaying Zhou, Yang Liu, Qingchao Chen

Organizations: Peking University, Beijing, China · National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University · Hohai University, Nanjing, China · Beijing Normal University-Hong Kong Baptist University United International College, Zhuhai, China

Abstract

The advent of large-scale self-supervised learning (SSL) has produced a vast zoo of medical foundation models. However, selecting optimal medical foundation models for specific segmentation tasks remains a computational bottleneck. Existing Transferability Estimation (TE) metrics, primarily designed for classification, rely on global statistical assumptions and fail to capture the topological complexity essential for dense prediction. We propose a novel Topology-Driven Transferability Estimation framework that evaluates manifold tractability rather than statistical overlap. Our approach introduces three components: (1) Global Representation Topology Divergence (GRTD), utilizing Minimum Spanning Trees to quantify feature-label structural isomorphism; (2) Local Boundary-Aware Topological Consistency (LBTC), which assesses manifold separability specifically at critical anatomical boundaries; and (3) Task-Adaptive Fusion, which dynamically integrates global and local metrics based on the semantic cardinality of the target task. Validated on the large-scale OpenMind benchmark across diverse anatomical targets and SSL foundation models, our approach significantly outperforms state-of-the-art baselines by around 31% relative improvement in the weighted Kendall metric, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning. The code will be made publicly available upon acceptance.

Figures & tables

Panel of Figure 1 — Figure 1: Left: Fine-tuning performance of diverse foundation models varies across downstream datasets, indicating that the optimal pre-trained encoder is task-dependent. Right: Visualizing transferability: Statistics vs. Topology. We compare the ground truth fine-tuning performance (Left) against rankings from the statistical metric (Middle) and our topology-driven metric (Right).

Figure 2: The overall framework. First, we extract multi-scale features via stratified sampling and construct topological graphs via MST. We then compute GRTD to quantify overall manifold alignment and LBTC to evaluate separability at critical anatomical boundaries. These metrics are integrated via a task-complexity gating factor, producing a final score $S_{\phi}$ that predicts fine-tuning performance without training.

Method	ID (Same Region)				OOD (Different Region)		Avg
Method	MSF	ISL	HNT	TPC	ACD	KIT	All
LogME [ 24 ]	-0.524	-0.223	-0.315	-0.545	0.575	-0.649	-0.280
LEEP [ 13 ]	-0.220	-0.427	-0.492	-0.051	-0.621	0.225	-0.264
GBC [ 14 ]	-0.707	-0.601	-0.537	-0.256	-0.325	-0.621	-0.508
CCFV [ 23 ]	0.277	0.503	0.869	0.664	0.817	0.180	0.552
Ours	0.705	0.814	0.905	0.664	0.671	0.578	0.723

Table 1: Weighted Kendall’s

\tau

for transferability estimation on the OpenMind benchmark.

Figure 3: Correlation between the fine-tuning performance and transferability metrics using MSF as an example. The vertical axis represents the average Dice, while the horizontal axis represents the standardized transferability metric. We want to observe a positive relationship between higher performance and higher transferability estimations.

Method	ISL	MSF	ACD	TPC	Avg.
Global (GRTD)	0.122	0.333	0.624	0.664	0.436
Local (LBTC)	0.814	0.705	0.319	0.062	0.475
Adaptive (Ours)	0.814	0.705	0.671	0.664	0.714

Table 3: Ablation study on topological priors of different datasets. We stratify tasks into Fragmented (ISL, MSF) and Structured (ACD, TPC) targets. Our Adaptive fusion achieves robust performance across all regimes.

Explore similar work

Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models
Jul 5, 2026Jiaqi Tang, Shaoyang Zhang, Fandong Zhang +3Medical Imaging Foundation Models 3D Medical Imaging
National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China +2
Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation
Oct 13, 2026Jiaqi Tang, Yu Wang, Yang Liu +7Medical Imaging Foundation Models Medical Image Benchmarks
National Institution of Health Data Science, Peking University, Beijing, China. · Institute of Medical Technology, Peking University, Beijing, China. · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China. +5
Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models
Oct 13, 2026Michalis Pistos, Bartłomiej W. PapieżObject-Centric Representation Learning Medical Image Representation Learning
Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, OX3 7LF, United Kingdom

Jul 5, 2026J/K move · Enter open · S save

Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models

Jiaqi Tang, Shaoyang Zhang, Fandong Zhang, Shu Zhang+2

National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China+2

Medical Imaging Foundation Models 3D Medical Imaging Task Transferability Estimation Transfer Learning

Open paper AskPDF arXiv

The growing number of medical vision foundation models highlights the need for effective model selection. However, mainstream selection methods rely on exhaustive fine-tuning, which is computationally expensive. Most of the existing Transferability Estimation (TE) metrics are primarily designed for image-level classification. They fail to preserve spatial relationships and fine-grained boundary details, which are crucial for the segmentation task. Additionally, while image-level tasks typically process a single feature vector per input, dense prediction tasks in 3D medical imaging require voxel-wise evaluation against dense annotations. To bridge these gaps, we propose a non-parametric, topology-driven framework that estimates transferability directly from the alignment between the sparse 1-skeleton graph of dense features and semantic labels via Minimum Spanning Trees (MST). We decouple the alignment into two complementary geometric scales: Local Boundary-Aware Topological Consistency (LBTC) to assess boundary separability, where we prove that the MST leakage rate serves as a finite-sample lower bound on the Bayes error; and Global Representation Topology Divergence (GRTD) to evaluate the overall anatomical layout. Crucially, we formally justify a counterintuitive mechanism: Although without fine-tuning, the randomly initialized segmentation decoder acts as a topology-preserving spatial projector, reducing the variance of pairwise distance estimates and stabilizing global alignment evaluation. Fused via a task-adaptive gating mechanism, these dual metrics adapt to diverse clinical complexities. Evaluated on a large-scale benchmark of 114,000 3D medical volumes across diverse anatomical tasks, our topological framework achieves state-of-the-art transferability estimation with an average weighted Kendall (outperforming by 0.36) while accelerating evaluation by 56 times.

Abstract

Explore similar work

Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models

Benchmarking Self-Supervised 3D Medical Vision Foundation Models and Transferability Estimation

Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models