Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models

Michalis Pistos; Bartłomiej W. Papież

cs.CVOct 13, 2026

Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models

Authors: Michalis Pistos, Bartłomiej W. Papież

Organizations: Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, OX3 7LF, United Kingdom

Abstract

Foundation models are increasingly used as feature extractors for downstream tasks with limited data, typically through representations obtained from their encoders. However, foundation models trained for structured tasks such as segmentation also form object-specific representations during task execution, whose transferability remains largely unexplored. We investigate Segment Anything Model 2 (SAM2) object pointers, compact object-level representations produced through the segmentation pathway, as transferable representations for downstream tumour prediction. Through experiments on two PET/CT tumour prediction tasks, we find that object pointers learned using segmentation supervision alone already encode tumour-discriminative information, despite never being optimised for the downstream labels. We further introduce task-guided object pointers, where an auxiliary objective jointly shapes the object pointer space for the downstream prediction together with segmentation. Task-guided pointers improve cancer-type classification, outperforming conventional encoder-based transfer and radiomics, while segmentation-only pointers perform best on the more challenging recurrence prediction task. Representation analyses reveal associations between object pointers and interpretable radiomics features, class-dependent organisation of the learned representation space, and explicit spatial grounding around the segmented tumour. Together, these findings suggest that useful transferable representations can emerge through the execution of a foundation model's primary task, providing an alternative to conventional encoder-based transfer.

Figures & tables

Figure 1: Overview of SAM2 representations for downstream tumour prediction. (a) SAM2 adaptation to 3D PET/CT, using modality-specific encoders and a single bounding-box prompt for tumour segmentation. (b) Encoder-based transfer, where image-encoder features are used by a CNN for downstream classification. (c) Object pointer transfer, where slice-level object pointers produced through segmentation are aggregated into a tumour-level representation and classified using a lightweight linear head.

Method	FDG-PET-CT-Lesions			HECKTOR 2022
Method	AUC $\uparrow$	B.Acc. $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	B.Acc. $\uparrow$	F1 $\uparrow$
Radiomics	0.833 $\pm$ 0.028	0.656 $\pm$ 0.046	0.645 $\pm$ 0.050	0.518 $\pm$ 0.060	0.517 $\pm$ 0.027	0.514 $\pm$ 0.029
Encoder + CNN	0.836 $\pm$ 0.020	0.650 $\pm$ 0.060	0.635 $\pm$ 0.075	0.504 $\pm$ 0.106	0.500 $\pm$ 0.000	0.457 $\pm$ 0.008
ObjPtr (Seg.)	0.781 $\pm$ 0.057	0.611 $\pm$ 0.075	0.604 $\pm$ 0.072	0.631 $\pm$ 0.031	0.574 $\pm$ 0.049	0.542 $\pm$ 0.034
ObjPtr (Task)	0.851 $\pm$ 0.017	0.684 $\pm$ 0.024	0.682 $\pm$ 0.023	0.584 $\pm$ 0.055	0.500 $\pm$ 0.000	0.457 $\pm$ 0.008
ObjPtr (Seg.) + Rad.	0.849 $\pm$ 0.019	0.674 $\pm$ 0.033	0.666 $\pm$ 0.039	0.549 $\pm$ 0.077	0.572 $\pm$ 0.066	0.511 $\pm$ 0.051

Table 1: Downstream prediction performance across representations and datasets. Results are reported as mean

\pm

standard deviation over five-fold cross-validation. The highest mean for each metric is highlighted in bold .

Figure 2: Canonical correlation analysis (CCA) between object pointer representations and radiomics feature groups. Results are shown for segmentation-only and task-guided object pointers across FDG-PET-CT-Lesions and HECKTOR 2022. Bars show the mean canonical correlation over five-fold cross-validation, with error bars indicating standard deviation.

Figure 3: Spatial comparison of encoder-based and object pointer representations. Representative FDG-PET-CT-Lesions cases spanning different spatial activation patterns of the Encoder + CNN baseline, with Grad-CAM maps shown in the top row and corresponding spatial attention maps for segmentation-only object pointers in the bottom row. White contours indicate the ground-truth tumour annotation.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	FDG-PET-CT-Lesions		HECKTOR 2022
Method	Dice $\uparrow$	Seg. failures (n) $\downarrow$	Dice $\uparrow$	Seg. failures (n) $\downarrow$
ObjPtr (Seg.)	0.737 $\pm$ 0.013	9/489	0.693 $\pm$ 0.022	3/427
ObjPtr (Task)	0.715 $\pm$ 0.010	3/489	0.685 $\pm$ 0.017	0/427

Appendix

Table 2: Segmentation performance of object pointer models. Dice is reported as mean

\pm

standard deviation across five-fold cross-validation, alongside the total number of segmentation failures.

Figure 4: Qualitative segmentation results from the segmentation-only object pointer model. Representative successful segmentations (left) and examples of segmentation failures (right) are shown for FDG-PET-CT-Lesions and HECKTOR 2022. White contours indicate the ground-truth tumour annotation and blue regions indicate the predicted segmentation. Arrows highlight the ground-truth tumour location in failure cases. The corresponding downstream class is shown in the top-right corner of each image.

PET Normalisation	Classification			Segmentation
PET Normalisation	AUC $\uparrow$	B.Acc. $\uparrow$	F1 $\uparrow$	Dice $\uparrow$
Global normalisation	0.781 $\pm$ 0.057	0.611 $\pm$ 0.075	0.604 $\pm$ 0.072	0.737 $\pm$ 0.013
3D volume-wise normalisation	0.762 $\pm$ 0.044	0.567 $\pm$ 0.059	0.560 $\pm$ 0.056	0.728 $\pm$ 0.022
2D slice-wise normalisation	0.723 $\pm$ 0.059	0.525 $\pm$ 0.067	0.519 $\pm$ 0.060	0.703 $\pm$ 0.014

Appendix

Table 3: Effect of PET intensity normalisation on segmentation-only object pointer representation quality on FDG-PET-CT-Lesions. Results are reported as mean

\pm

standard deviation across five-fold cross-validation. The highest mean for each metric is highlighted in bold .

Figure 5: Organisation of the SAM2 object pointer representation space on FDG-PET-CT-Lesions. UMAP projections of tumour-level object pointers for (a) the segmentation-only model with global PET norm, (b) task-guided object pointer training, (c) segmentation-only 3D volume-wise, and (d) segmentation-only 2D slice-wise PET normalisation. Points represent individual tumours and are coloured by cancer type. The segmentation-only representation already exhibits class-dependent organisation, while changes in PET normalisation alter this structure and task-guided training further separates the downstream classes.

Figure 6: Pearson correlation coefficients (PCC) between individual object pointer dimensions and radiomics features. PCC is shown for segmentation-only and task-guided object pointers across FDG-PET-CT-Lesions and HECKTOR 2022.

Figure 7: Effect of auxiliary classification-loss weight on FDG-PET-CT-Lesions. Downstream classification AUC and segmentation Dice across different values of

\lambda_{\mathrm{cls}}

. Points show the mean across five-fold cross-validation, with error bars indicating standard deviation.

Explore similar work

Feb 27, 2026

Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

The advent of large-scale self-supervised learning (SSL) has produced a vast zoo of medical foundation models. However, selecting optimal medical foundation models for specific segmentation tasks remains a computational bottleneck. Existing Transferability Estimation (TE) metrics, primarily designed for classification, rely on global statistical assumptions and fail to capture the topological complexity essential for dense prediction. We propose a novel Topology-Driven Transferability Estimation framework that evaluates manifold tractability rather than statistical overlap. Our approach introduces three components: (1) Global Representation Topology Divergence (GRTD), utilizing Minimum Spanning Trees to quantify feature-label structural isomorphism; (2) Local Boundary-Aware Topological Consistency (LBTC), which assesses manifold separability specifically at critical anatomical boundaries; and (3) Task-Adaptive Fusion, which dynamically integrates global and local metrics based on the semantic cardinality of the target task. Validated on the large-scale OpenMind benchmark across diverse anatomical targets and SSL foundation models, our approach significantly outperforms state-of-the-art baselines by around 31% relative improvement in the weighted Kendall metric, providing a robust, training-free proxy for efficient model selection without the cost of fine-tuning. The code will be made publicly available upon acceptance.

Jiaqi Tang, Shaoyang Zhang, Xiaoqi Wang +3

Peking University, Beijing, China · National Institute of Health Data Science, Peking University, Beijing, China · Institute of Medical Technology, Peking University, Beijing, China +3

Task Transferability Estimation Medical Imaging Foundation Models Medical Image Segmentation

Ask PDF

Jun 15, 2026

Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to datasets under distribution shift remains underexplored. This work systematically evaluates FM-based representations on a suite of computational pathology tasks across two real-world commercial cohorts, IH-BC and IH-NSCLC, drawn from the licensed in-house (IH) oncology dataset. The analysis focuses on two modalities, whole-slide images and transcriptomic profiles, drawn from the IH multimodal data. We first benchmark unimodal probing performance across five FMs on eight downstream classification tasks, and find that image and omics representations carry complementary predictive signals. Then we investigate whether multimodal fusion can yield additional gains over unimodal baselines by comparing three image-omics fusion strategies built on paired representations. The trustworthiness of selected unimodal and multimodal pipelines is further assessed through conformal prediction. Our results show that FM representations achieve competitive performance on out-of-distribution data and that multimodal fusion helps mainly when no single modality dominates the signal. Conformal prediction reveals that in the majority of cases where a point prediction fails, the true diagnosis remains recoverable within the prediction set, reinforcing the value of uncertainty-aware inference for clinical support.

Jingyu Hu, Giuseppe Tripodi, Reed Naidoo +2

The Alan Turing Institute, London, United Kingdom · University of Manchester, Manchester, United Kingdom · The Institute of Cancer Research, London, United Kingdom +1

Computational Pathology Multimodal Robustness Pathology Foundation Models Multimodal Classification Oncology Multimodal Foundation Models+1

Ask PDF

Sep 12, 2026

Parameter-Efficient Fine-Tuning of Foundation Models for Liver Tumor Segmentation in CT

We evaluated parameter-efficient fine-tuning (PEFT) of the Segment Anything Model (SAM) for liver tumor segmentation in abdominal CT of colorectal liver metastases. We compared Low-Rank Adaptation (LoRA), 4-bit Quantized LoRA (QLoRA), a convolutional adapter (Conv-Adapter), and our Directional Spectral Top-K adapter (DiSCo), training only adapters while freezing the SAM backbone. DiSCo derives spectral bases from singular value decomposition of row-normalized weights and learns rank-gated spectral coefficients, per-output magnitude offsets, and a spectral gain, with optional Top-K rank selection at inference and 0.14 M trainable parameters. We benchmarked five prompting regimes: no prompt, single-point, multi-point, and bounding boxes at intersection over union 0.50 and 0.75. Conv-Adapter and LoRA achieved the highest accuracy (overall Dice 0.793 and 0.792; single-point Dice 0.795 and 0.792; 95th-percentile Hausdorff distance (HD95) 32 mm). QLoRA was close (overall Dice 0.766; single-point Dice 0.768; HD95 36.41 mm), with 0.91 M trainable parameters, 120 ms latency, and 4.9 GB peak memory. DiSCo achieved the highest Dice per million trainable parameters (4.66), with overall Dice 0.653, single-point Dice 0.698, and HD95 49.53 mm. These results show an accuracy-efficiency trade-off and support PEFT for liver tumor segmentation with reduced adaptation costs when compute and labeled data are limited. Code: https://github.com/Ramtin-Mojtahedi/PEFT-SAM-Liver-CT

Ramtin Mojtahedi, Mohammad Hamghalam, Jacob J. Peoples +2

School of Computing, Queen’s University, Kingston, ON, Canada · Toronto General Research Institute, University Health Network, Toronto, ON, Canada · Department of Electrical Engineering, Qa.C., Islamic Azad University, Qazvin, Iran +3

Medical Image Segmentation Parameter-Efficient Fine-Tuning Vision Foundation Model Adaptation Medical Imaging Foundation Models Promptable Image Segmentation

Ask PDF