MUST-PET: MUltimodal Self-supervised learning across Tracers for whole-body PET/CT-based lesion segmentation

Bashirul Azam Biswas; Amartya Bhattacharya; Biratal Raj Wagle; Matthew E. Maeder; James B. Yu; Indrani Bhattacharya

cs.CVAug 20, 2026

MUST-PET: MUltimodal Self-supervised learning across Tracers for whole-body PET/CT-based lesion segmentation

Authors: Bashirul Azam Biswas, Amartya Bhattacharya, Biratal Raj Wagle, Matthew E. Maeder, James B. Yu, Indrani Bhattacharya

Organizations: Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA

Abstract

Deep learning-based whole-body PET-CT lesion segmentation can support cancer staging, treatment planning, and response assessment, but generalization is limited by scarce annotations and domain shifts. Self-supervised learning (SSL) can address these challenges but remains underexplored in pan-cancer, multi-tracer PET-CT. In this work, we propose MUST-PET (MUltimodal Self-Supervised learning across Tracers), a multimodal, multi-tracer SSL framework for generalizable whole-body PET-CT lesion segmentation. MUST-PET is trained and validated on a diverse, multi-institutional collection of pan-cancer PET-CT scans acquired with FDG and prostate-specific membrane antigen (PSMA)-targeted radiotracers. MUST-PET uses context-aware masked reconstruction, where one modality is partially masked and reconstructed using complementary information from both PET and CT. The pretrained model is subsequently fine-tuned with labeled samples and evaluated for reconstruction quality, lesion segmentation, label efficiency, and generalizability across independent held-out datasets. MUST-PET reduces reconstruction error, improves lesion segmentation over training from scratch, and performs well with limited labeled data and on unseen external datasets, demonstrating the potential of multi-tracer SSL for label-efficient, generalizable whole-body PET-CT. segmentation.

Figures & tables

Properties	AutoPET-III	DeepPSMA	DHMC	SPADE	VI-MED
Disease dist.	Me/Lu/Ly/Pr/Neg 188/168/145/537/573	Pr	Pr	–	–
FDG/PSMA	✓/✓	✓/✓	✗/✓	✓/✗	✓/✗
# lesions (per case)	17.47 $\pm$ 46.12	69.99 $\pm$ 60.04	10.13 $\pm$ 16.27	–	–
# slices (per vol)	135–963	195–1261	303–380	103-307	207-551
Pretraining	Yes	No	Yes	Yes	Yes
Segmentation	Yes	Yes	No	No	No

Table 1: Summary of datasets used. The FDG/PSMA row marks tracer availability (✓ present, ✗ absent). Lu=Lung, Ly=Lymphoma, Me=Melanoma, Pr=Prostate, Neg=Negative.

Model	FDG	PSMA
Model	AutoPET ( $n=199$ )	AutoPET ( $n=122$ )	DHMC ( $n=100$ )
FDG-only MAE	0.2431	0.2982	0.2970
MUST-PET (Ours)	0.2709	0.2908	0.2860

Table 2: Reconstruction MAE of the FDG-only [ 7 ] and our proposed MUST-PET model. Lower is better.

Figure 1: Label-Efficient Fine-Tuning Performance. Pretrained models provide substantial gains in low-label settings. Stronger generalisation is achieved in Deep PSMA dataset through the pretraining.

Dataset	Model	Training Strategy	Dice	FP Vol	FN Vol	Lesion Sensitivity	PPV
			( $\uparrow$ )	( $\downarrow$ )	( $\downarrow$ )	( $\uparrow$ )	( $\uparrow$ )
AutoPET III	SwinUNETR	Scratch	0.526	27.147	7.586	0.773	0.546
	nnUNet	Scratch	0.566	17.283	9.472	0.773	0.733
	MUST-PET	Decoder FT	0.565	19.874	8.169	0.794	0.670
		Enc.+Dec. FT	0.569	20.954	8.457	0.775	0.685
		Full FT	0.576	14.112	9.976	0.778	0.673

Table 3: Lesion segmentation performance on the test sets with 100% labeled AutoPET-III training data. FPVol and FNVol are reported in mL.

\uparrow

: higher is better;

\downarrow

: lower is better.

Figure 2: Qualitative comparison of lesion segmentation by the scratch and MUST-PET models, both trained on 100% of the data. Green contours indicate GT, while red and blue contours show scratch and fine-tuned predictions, respectively. Yellow arrows indicate false positives from the scratch model, and pink arrows highlight improved lesion delineation with MUST-PET.

Multimodal SSL

See all 14 papers

Foundation Model-Enabled Efficient Data Sampling (FEEDS): A label-efficient training strategy for pan-cancer, multi-tracer PET/CT datasets
Aug 11, 2026Biratal Raj Wagle, Bashirul Azam Biswas, Grant Chau +5Data Selection Medical Image Segmentation
Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Department of Urology Yale University New Haven, CT, USA +1
An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation
May 20, 2026Xiaofeng Liu, Qianru Zhang, Thibault Marin +4Cross-Modality Medical Image Segmentation Medical Imaging Foundation Models
Department of Radiology and Biomedical Imaging, Yale Biomedical Imaging Institute, Yale University, New Haven, CT, USA
Pretrained, Curriculum-Tuned, and Ensembled: A Tracer-Aware Interactive Segmentation Pipeline for AutoPET V
Aug 31, 2026Xinglong Liang, Chunyao Lu, Tianyu Zhang +4Interactive Medical Image Segmentation PET/CT Imaging
Department of Radiology, The Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX, Amsterdam, the Netherlands · Department of Radiology and Nuclear Medicine, Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA, Nijmegen, The Netherlands · Faculty of Applied Sciences, Macao Polytechnic University, 999078, Macao Special Administrative Region of China +2

Aug 11, 2026J/K move · Enter open · S save

Foundation Model-Enabled Efficient Data Sampling (FEEDS): A label-efficient training strategy for pan-cancer, multi-tracer PET/CT datasets

Biratal Raj Wagle, Bashirul Azam Biswas, Grant Chau, Matthew E. Maeder+4

Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Department of Urology Yale University New Haven, CT, USA+1

Data Selection Medical Image Segmentation PET/CT Tumor Segmentation Data-Efficient Learning

Open paper AskPDF arXiv

Automated lesion segmentation in whole-body PET/CT imaging can assist clinicians with cancer detection, staging, and treatment planning across radiotracers and cancer types. However, training lesion segmentation models that capture variations in lesion size, distribution, and appearance requires large annotated datasets, whose creation is both time- and expertise-intensive. As a result, models trained on limited labeled PET/CT data often lack the accuracy and generalizability needed for clinical use. We present FEEDS (Foundation model-Enabled Efficient Data Sampling), a label- and compute-efficient learning strategy that uses vision foundation model embeddings to select the most informative and diverse unlabeled cases for expert annotation. Unlike unsupervised, semi-supervised, and active learning approaches, FEEDS is a one-step training paradigm requiring only a limited, representative training set, making it label- and compute-efficient. We train and validate FEEDS using the AutoPET-III dataset. We test its accuracy and generalizability on three held-out sets: AutoPET-III, DeepPSMA, and an internal Dartmouth-Hitchcock Medical Center dataset. We evaluate clinical utility at the voxel, lesion, and anatomic region level to assess performance in high-risk areas and treatment planning utility. FEEDS outperforms random-sampling-based labeling, pseudolabel-based semi-supervised learning, and training with limited labeled data alone. It generalizes across all three test sets, FDG and PSMA tracers, and multiple diseases, matching fully-labeled (100%) training performance with 70% less annotation burden. FEEDS addresses the challenge of label scarcity in an automatic lesion segmentation framework by providing a practical approach for constructing representative and diverse annotation queues from large, unannotated clinical repositories.

Abstract

Multimodal SSL

Explore similar work

Foundation Model-Enabled Efficient Data Sampling (FEEDS): A label-efficient training strategy for pan-cancer, multi-tracer PET/CT datasets

An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation

Pretrained, Curriculum-Tuned, and Ensembled: A Tracer-Aware Interactive Segmentation Pipeline for AutoPET V