Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models

Michalis Pistos; Bartłomiej W. Papież

cs.CVOct 13, 2026

Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models

Authors: Michalis Pistos, Bartłomiej W. Papież

Organizations: Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, OX3 7LF, United Kingdom

Abstract

Foundation models are increasingly used as feature extractors for downstream tasks with limited data, typically through representations obtained from their encoders. However, foundation models trained for structured tasks such as segmentation also form object-specific representations during task execution, whose transferability remains largely unexplored. We investigate Segment Anything Model 2 (SAM2) object pointers, compact object-level representations produced through the segmentation pathway, as transferable representations for downstream tumour prediction. Through experiments on two PET/CT tumour prediction tasks, we find that object pointers learned using segmentation supervision alone already encode tumour-discriminative information, despite never being optimised for the downstream labels. We further introduce task-guided object pointers, where an auxiliary objective jointly shapes the object pointer space for the downstream prediction together with segmentation. Task-guided pointers improve cancer-type classification, outperforming conventional encoder-based transfer and radiomics, while segmentation-only pointers perform best on the more challenging recurrence prediction task. Representation analyses reveal associations between object pointers and interpretable radiomics features, class-dependent organisation of the learned representation space, and explicit spatial grounding around the segmented tumour. Together, these findings suggest that useful transferable representations can emerge through the execution of a foundation model's primary task, providing an alternative to conventional encoder-based transfer.

Figures & tables

Figure 1: Overview of SAM2 representations for downstream tumour prediction. (a) SAM2 adaptation to 3D PET/CT, using modality-specific encoders and a single bounding-box prompt for tumour segmentation. (b) Encoder-based transfer, where image-encoder features are used by a CNN for downstream classification. (c) Object pointer transfer, where slice-level object pointers produced through segmentation are aggregated into a tumour-level representation and classified using a lightweight linear head.

Method	FDG-PET-CT-Lesions			HECKTOR 2022
Method	AUC $\uparrow$	B.Acc. $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	B.Acc. $\uparrow$	F1 $\uparrow$
Radiomics	0.833 $\pm$ 0.028	0.656 $\pm$ 0.046	0.645 $\pm$ 0.050	0.518 $\pm$ 0.060	0.517 $\pm$ 0.027	0.514 $\pm$ 0.029
Encoder + CNN	0.836 $\pm$ 0.020	0.650 $\pm$ 0.060	0.635 $\pm$ 0.075	0.504 $\pm$ 0.106	0.500 $\pm$ 0.000	0.457 $\pm$ 0.008
ObjPtr (Seg.)	0.781 $\pm$ 0.057	0.611 $\pm$ 0.075	0.604 $\pm$ 0.072	0.631 $\pm$ 0.031	0.574 $\pm$ 0.049	0.542 $\pm$ 0.034
ObjPtr (Task)	0.851 $\pm$ 0.017	0.684 $\pm$ 0.024	0.682 $\pm$ 0.023	0.584 $\pm$ 0.055	0.500 $\pm$ 0.000	0.457 $\pm$ 0.008
ObjPtr (Seg.) + Rad.	0.849 $\pm$ 0.019	0.674 $\pm$ 0.033	0.666 $\pm$ 0.039	0.549 $\pm$ 0.077	0.572 $\pm$ 0.066	0.511 $\pm$ 0.051

Table 1: Downstream prediction performance across representations and datasets. Results are reported as mean

\pm

standard deviation over five-fold cross-validation. The highest mean for each metric is highlighted in bold .

Figure 2: Canonical correlation analysis (CCA) between object pointer representations and radiomics feature groups. Results are shown for segmentation-only and task-guided object pointers across FDG-PET-CT-Lesions and HECKTOR 2022. Bars show the mean canonical correlation over five-fold cross-validation, with error bars indicating standard deviation.

Figure 3: Spatial comparison of encoder-based and object pointer representations. Representative FDG-PET-CT-Lesions cases spanning different spatial activation patterns of the Encoder + CNN baseline, with Grad-CAM maps shown in the top row and corresponding spatial attention maps for segmentation-only object pointers in the bottom row. White contours indicate the ground-truth tumour annotation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	FDG-PET-CT-Lesions		HECKTOR 2022
Method	Dice $\uparrow$	Seg. failures (n) $\downarrow$	Dice $\uparrow$	Seg. failures (n) $\downarrow$
ObjPtr (Seg.)	0.737 $\pm$ 0.013	9/489	0.693 $\pm$ 0.022	3/427
ObjPtr (Task)	0.715 $\pm$ 0.010	3/489	0.685 $\pm$ 0.017	0/427

Appendix

Table 2: Segmentation performance of object pointer models. Dice is reported as mean

\pm

standard deviation across five-fold cross-validation, alongside the total number of segmentation failures.

Figure 4: Qualitative segmentation results from the segmentation-only object pointer model. Representative successful segmentations (left) and examples of segmentation failures (right) are shown for FDG-PET-CT-Lesions and HECKTOR 2022. White contours indicate the ground-truth tumour annotation and blue regions indicate the predicted segmentation. Arrows highlight the ground-truth tumour location in failure cases. The corresponding downstream class is shown in the top-right corner of each image.

PET Normalisation	Classification			Segmentation
PET Normalisation	AUC $\uparrow$	B.Acc. $\uparrow$	F1 $\uparrow$	Dice $\uparrow$
Global normalisation	0.781 $\pm$ 0.057	0.611 $\pm$ 0.075	0.604 $\pm$ 0.072	0.737 $\pm$ 0.013
3D volume-wise normalisation	0.762 $\pm$ 0.044	0.567 $\pm$ 0.059	0.560 $\pm$ 0.056	0.728 $\pm$ 0.022
2D slice-wise normalisation	0.723 $\pm$ 0.059	0.525 $\pm$ 0.067	0.519 $\pm$ 0.060	0.703 $\pm$ 0.014

Appendix

Table 3: Effect of PET intensity normalisation on segmentation-only object pointer representation quality on FDG-PET-CT-Lesions. Results are reported as mean

\pm

standard deviation across five-fold cross-validation. The highest mean for each metric is highlighted in bold .

Figure 5: Organisation of the SAM2 object pointer representation space on FDG-PET-CT-Lesions. UMAP projections of tumour-level object pointers for (a) the segmentation-only model with global PET norm, (b) task-guided object pointer training, (c) segmentation-only 3D volume-wise, and (d) segmentation-only 2D slice-wise PET normalisation. Points represent individual tumours and are coloured by cancer type. The segmentation-only representation already exhibits class-dependent organisation, while changes in PET normalisation alter this structure and task-guided training further separates the downstream classes.

Figure 6: Pearson correlation coefficients (PCC) between individual object pointer dimensions and radiomics features. PCC is shown for segmentation-only and task-guided object pointers across FDG-PET-CT-Lesions and HECKTOR 2022.

Figure 7: Effect of auxiliary classification-loss weight on FDG-PET-CT-Lesions. Downstream classification AUC and segmentation Dice across different values of

\lambda_{\mathrm{cls}}

. Points show the mean across five-fold cross-validation, with error bars indicating standard deviation.

Object-Centric Representation Learning

Object-Centric Representation Learning studies visual models that organize scenes into distinct objects and their properties, positions, and relations.

See all 89 papers

Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation
Feb 27, 2026Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang +3Task Transferability Estimation Medical Imaging Foundation Models
Peking University, Beijing, China · National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China +3
Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis
Jun 15, 2026Jingyu Hu, Giuseppe Tripodi, Reed Naidoo +2Computational Pathology Multimodal Robustness
The Alan Turing Institute, London, United Kingdom · University of Manchester, Manchester, United Kingdom · The Institute of Cancer Research, London, United Kingdom +1
Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT
Sep 12, 2026Ramtin Mojtahedi, Mohammad Hamghalam, Jacob J. Peoples +2Medical Image Segmentation Parameter-Efficient Fine-Tuning
School of Computing, Queen’s University, Kingston, ON, Canada · Toronto General Research Institute, University Health Network, Toronto, ON, Canada · Department of Electrical Engineering, Qa.C., Islamic Azad University, Qazvin, Iran +3

Feb 27, 2026J/K move · Enter open · S save

Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang, Jiaying Zhou+2

Peking University, Beijing, China · National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China+3

Task Transferability Estimation Medical Imaging Foundation Models Medical Image Segmentation

Open paper AskPDF arXiv

The advent of large-scale self-supervised learning (SSL) has produced a vast zoo of medical foundation models. However, selecting optimal medical foundation models for specific segmentation tasks remains a computational bottleneck. Existing Transferability Estimation (TE) metrics, primarily designed for classification, rely on global statistical assumptions and fail to capture the topological complexity essential for dense prediction. We propose a novel Topology-Driven Transferability Estimation framework that evaluates manifold tractability rather than statistical overlap. Our approach introduces three components: (1) Global Representation Topology Divergence (GRTD), utilizing Minimum Spanning Trees to quantify feature-label structural isomorphism; (2) Local Boundary-Aware Topological Consistency (LBTC), which assesses manifold separability specifically at critical anatomical boundaries; and (3) Task-Adaptive Fusion, which dynamically integrates global and local metrics based on the semantic cardinality of the target task. Validated on the large-scale OpenMind benchmark across diverse anatomical targets and SSL foundation models, our approach significantly outperforms state-of-the-art baselines by around 31% relative improvement in the weighted Kendall metric, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning. The code will be made publicly available upon acceptance.

Abstract

Object-Centric Representation Learning

Explore similar work

Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT