LoRe-MIM: Lost-or-Replaced Masked Image Modeling for Self-Supervised Representation Learning in Whole-Body PET/CT

Amartya Bhattacharya; Bashirul Azam Biswas; Biratal Raj Wagle; Michael S. Leapman; Matthew E. Maeder; James B. Yu; Indrani Bhattacharya

cs.CVOct 29, 2026

LoRe-MIM: Lost-or-Replaced Masked Image Modeling for Self-Supervised Representation Learning in Whole-Body PET/CT

Authors: Amartya Bhattacharya, Bashirul Azam Biswas, Biratal Raj Wagle, Michael S. Leapman, Matthew E. Maeder, James B. Yu, Indrani Bhattacharya

Organizations: Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Hanover, NH 03755, USA · Department of Urology, Yale University, New Haven, CT, USA · Radiology, Dartmouth Hitchcock Medical Center, Lebanon, NH 03766 , USA · Radiation Oncology, Dartmouth Hitchcock Medical Center, Lebanon, NH 03766 , USA

Abstract

Deep learning models for whole-body positron emission tomography/computed tomography (PET/CT) hold significant clinical potential for disease detection, treatment planning, and response assessment. However, the time and cost associated with expert annotations remain major bottlenecks in developing robust and generalizable models. Although self-supervised learning (SSL) has emerged as a promising approach to mitigating annotation scarcity, its application to whole-body PET/CT remains relatively underexplored due to the limited availability of large-scale datasets and the challenges associated with learning complementary cross-modal representations. To address these limitations, we curate a heterogeneous, multi-tracer, multi-institutional cohort comprising 6,531 whole-body PET/CT scans and propose LoRe-MIM (Lost-Or-Replaced Masked Image Modelling), a novel self-supervised framework designed to learn multimodal representations through masked image reconstruction. Specifically, LoRe-MIM introduces a masking strategy in which image regions are either replaced with zeros (Lost) or substituted with corresponding complementary imaging modality (Replaced), encouraging the model to capture both modality-specific characteristics and cross-modal relationships. We evaluate the learned representations across multiple downstream tasks, including multimodal PET/CT image reconstruction, patch-based classification into cancer vs. non-cancer, whole-body lesion segmentation, label-efficient training, and robustness assessment. Experimental results demonstrate that LoRe-MIM consistently outperforms the corresponding state-of-the-art baselines across these tasks, highlighting its potential as a scalable and generalizable self-supervised learning framework for whole-body PET/CT analysis.

Figures & tables

Properties	AutoPET-III	DeepPSMA	DHMC	SPADE	VI-MED
Disease dist.	Me/Lu/Ly/Pr/Neg 188/168/145/537/573	Pr	Pr	–	–
FDG/PSMA	✓/✓	✓/✓	✗/✓	✓/✗	✓/✗
# lesions (per case)	17.47 $\pm$ 46.12	69.99 $\pm$ 60.04	10.13 $\pm$ 16.27	–	–
# slices (per vol)	135–963	195–1261	303–380	103-307	207-551
Pretraining	Yes	No	Yes	Yes	Yes
Segmentation	Yes	Yes	No	No	No

Table 1: Summary of datasets used. The FDG/PSMA row marks tracer availability (✓ present, ✗ absent). Lu=Lung, Ly=Lymphoma, Me=Melanoma, Pr=Prostate, Neg=Negative.

Figure 1: Overview of the proposed masking approach.

Masking Approach	Reconstruction			Classification
Masking Approach	MAE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	AUC $\uparrow$
FDG-Only MAE [ 24 ]	0.2797	0.7288	32.56	0.6195
LO-MIM	0.4026	0.6473	30.01	0.6885
LB-MIM	0.3831	0.6835	30.14	0.6894
Re-MIM	0.2803	0.7586	32.73	0.7010
LoRE-MIM	0.2675	0.7762	33.07	0.7284

Table 2: Ablation study of masking strategies under fixed masking. Reconstruction performance on AutoPET (

n=247

) and DHMC (

n=149

) validation samples, averaged equally across CT and PET and across all 396 cases, is evaluated using MAE (

\downarrow

), SSIM (

\uparrow

), and PSNR in dB (

\uparrow

). Classification performance on clean test data is evaluated using AUC (

\uparrow

). Bold indicates the best result for each metric.

Model	Augmentation	Variable Masking	Reconstruction			Classification
Model	Augmentation	Variable Masking	MAE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	AUC $\uparrow$
Re-MIM	$\times$	$\times$	0.3913	0.6784	29.86	0.6855
	$\checkmark$	$\times$	0.4131	0.6513	29.70	0.5783
	$\times$	$\checkmark$	0.4258	0.6568	28.82	0.5695
	$\checkmark$	$\checkmark$	0.4989	0.5738	28.35	0.5525
LoRE-MIM	$\times$	$\times$	0.2718	0.7701	32.85	0.7091

Table 3: Ablation study of augmentation and variable masking for CM and LoRE-MIM on AutoPET-III (

n=321

) and DHMC (

n=100

) test sets. Reconstruction metrics are evaluated over masked regions after whole-volume reconstruction, averaged equally across CT and PET and across all 421 cases. Masked blocks are replaced with Gaussian samples of standard deviation 0.1. PSNR is in dB. Classification AUC values are retained from the original experiments. Bold indicates the best result for each metric.

Masking Approach	Reconstruction			Classification
Masking Approach	MAE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	AUC $\uparrow$
FDG-Only	0.2812	0.7256	32.45	0.6782
LO-MIM	0.4080	0.6432	29.74	0.6695
LB-MIM	0.3913	0.6784	29.86	0.6626
Re-MIM	0.2857	0.7510	32.51	0.6855
LoRE-MIM	0.2718	0.7701	32.85	0.7091

Table 4: Reconstruction and classification performance. Reconstruction is evaluated on the AutoPET-III (

n=321

) and DHMC (

n=100

) test sets using masked-region MAE (

\downarrow

), SSIM (

\uparrow

), and PSNR in dB (

\uparrow

), computed after whole-volume reconstruction and averaged equally across CT and PET modalities and across all 421 test cases. Classification performance is evaluated using AUC (

\uparrow

), with values retained from the original classification experiments.

Feature Extraction Stage / Model	Feature Type	Reduction	AUC $\uparrow$
Enc	Flatten	KPCA (128)	0.6898
	Flatten	PCA (128)	0.6565
	GAP	KPCA (128)	0.6808
		PCA (128)	0.6974
		KPCA (384)	0.7079
		PCA (384)	0.7180

Table 5: Evaluation of patch-based classification using features extracted from different model stages and baseline architectures under clean input conditions. We compare feature aggregation and dimensionality reduction strategies using logistic regression (LR) for classification. Performance is evaluated using AUC (

\uparrow

). Bold and underline indicate the best overall and best encoder performance, respectively.

Category	Model	Gaussian Noise	Misaligned	CT Missing	PET Missing
Baseline	ResNet50-3D	0.7268	0.7332	0.6217	0.5348
Baseline	DINOv3	0.7555	0.7286	0.6500	0.5344
Ours	LoRe-MIM (Dec 4 )	0.7584	0.7648	0.6975	0.5995

Table 6: Robustness evaluation of patch-based classification under four PET/CT input perturbations: Gaussian noise, PET/CT misalignment, missing CT, and missing PET. Performance is evaluated using AUC (

\uparrow

). Bold indicates the best result for each perturbation.

Dataset	Baseline	Training Strategy	Variant	Dice $\uparrow$	FPVol $\downarrow$	FNVol $\downarrow$	Lesion Sens. $\uparrow$	PPV $\uparrow$
AutoPET-III	nnU-Net	Scratch	–	0.566	17.283	9.472	0.773	0.733
	SwinUNETR	Scratch	–	0.526	27.147	7.586	0.773	0.546
		LoRe-MIM	Full FT	0.577	15.594	9.832	0.775	0.664
			Dec. FT	0.569	19.207	7.405	0.764	0.713
			Enc. + Dec. FT	0.581	15.662	8.362	0.776	0.675
Deep-PSMA	nnU-Net	Scratch	–	0.561	73.030	7.933	0.757	0.768

Table 7: Lesion segmentation performance on the AutoPET-III (

n=321

) and Deep-PSMA (

n=200

) test sets using different baseline architectures and training strategies. Three finetuning variants of LoRe-MIM (Ours) are evaluated: Full FT, Dec. FT, and Enc. + Dec. FT. FPVol and FNVol are reported in cc.

\uparrow

: higher is better;

\downarrow

: lower is better.

Panel of Figure 2 — Figure 2: Tracer-specific voxel-level evaluation on the AutoPET-III test set, comparing nnU-Net and SwinUNETR trained from scratch with SwinUNETR pretrained using LoRe-MIM under different finetuning configurations.

Panel of Figure 3 — Figure 3: Tracer-specific voxel-level evaluation on the Deep-PSMA test set, comparing nnU-Net and SwinUNETR trained from scratch with SwinUNETR pretrained using LoRe-MIM under different finetuning configurations.

Figure 4: Anatomical region-level comparison of lesion segmentation performance on AutoPET-III, comparing nnU-Net and SwinUNETR trained from scratch with three finetuning variants of our LoRe-MIM pretrained SwinUNETR.

Figure 5: Anatomical region-level comparison of lesion segmentation performance on Deep PSMA, comparing nnU-Net and SwinUNETR trained from scratch with three finetuning variants of our LoRe-MIM pretrained SwinUNETR.

Figure 6: Label-efficiency curves for whole-body lesion segmentation on the AutoPET-III and Deep-PSMA datasets. Segmentation performance is evaluated using Dice scores across varying proportions of labeled training data (1%–100%). All three finetuning strategies outperform SwinUNETR and nnU-Net trained from scratch, with more pronounced improvements under limited labeled data (1%–10%) and diminishing performance gains as the proportion of labeled data increases.

Figure 7: Qualitative comparison of lesion segmentation performance across four representative cases (Rows 1–4). Columns 1 and 2 show the CT and PET images with ground-truth lesion contours (green), respectively. Columns 3 and 4 present predictions from nnU-Net and SwinUNETR trained from scratch, respectively, while Columns 5–7 show predictions from three SwinUNETR finetuning strategies initialized with our pretrained LoRe-MIM model. Row 1 illustrates successful liver lesion recovery in an AutoPET-III case, where both scratch-trained models fail to detect the lesion, whereas all three LoRe-MIM variants successfully identify it. Row 2 demonstrates false-positive elimination in an AutoPET-III case, where both scratch-trained models incorrectly predict a lesion, while all three LoRe-MIM variants correctly produce no lesion segmentation. Row 3 presents a high-risk bone lesion from Deep-PSMA, where scratch-trained SwinUNETR misses the lesion, while all three LoRe-MIM variants successfully recover it. Finally, Row 4 illustrates a failure case involving a liver lesion from Deep-PSMA, where decoder-only finetuning fails to detect the lesion and encoder–decoder finetuning yields an incomplete segmentation, whereas full finetuning and scratch-trained nnU-Net successfully recover the lesion.

Figure 8: Qualitative comparison of whole-body PET/CT lesion segmentation between models trained from scratch (nnUNet and SwinUNETR) and our proposed LoRe-MIM pretrained SwinUNETR. Columns 1 and 2 show the coronal CT and PET images, respectively, with ground-truth lesion contours highlighted in green. Columns 3 and 4 present the segmentation predictions from nnUNet and SwinUNETR trained from scratch, respectively. Both models correctly identify the true-positive lesion (green arrow) but introduce an additional false-positive prediction (red arrow). In contrast, our proposed LoRe-MIM pretrained SwinUNETR (Column 5) successfully eliminates the false-positive prediction while preserving the true-positive lesion, demonstrating improved lesion discrimination.

Panel of Figure 9 — Figure 9: Qualitative comparison of masked-image reconstruction for CT (top) and PET (bottom). From left to right: clean reference, masked input, CM reconstruction, and LoRE-MIM reconstruction. Red region outlines highlight corresponding regions where LoRE-MIM provides visually closer agreement with the clean reference. CM and LoRE-MIM reconstructions are outlined in red and dark green, respectively.

Medical Image Representation Learning

Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.

See all 263 papers

MUST-PET: MUltimodal Self-supervised learning across Tracers for whole-body PET/CT-based lesion segmentation
Aug 20, 2026Bashirul Azam Biswas, Amartya Bhattacharya, Biratal Raj Wagle +3Multimodal SSL PET/CT Tumor Segmentation
Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA
Developing Foundation Models for Universal Segmentation from 3D Whole-Body Positron Emission Tomography
Mar 12, 2026Yichi Zhang, Le Xue, Wenbo Zhang +16Medical Imaging Foundation Models 3D Medical Image Segmentation
Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China. · Department of Nuclear Medicine and PET Center, Huashan Hospital, Fudan University, Shanghai, China. +6
An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation
May 20, 2026Xiaofeng Liu, Qianru Zhang, Thibault Marin +4Cross-Modality Medical Image Segmentation Medical Imaging Foundation Models
Department of Radiology and Biomedical Imaging, Yale Biomedical Imaging Institute, Yale University, New Haven, CT, USA

Aug 20, 2026J/K move · Enter open · S save

MUST-PET: MUltimodal Self-supervised learning across Tracers for whole-body PET/CT-based lesion segmentation

Bashirul Azam Biswas, Amartya Bhattacharya, Biratal Raj Wagle, Matthew E. Maeder+2

Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA

Multimodal SSL PET/CT Tumor Segmentation PET/CT Imaging Multimodal Learning Cross-Modality Medical Image Segmentation

Open paper AskPDF arXiv

Deep learning-based whole-body PET-CT lesion segmentation can support cancer staging, treatment planning, and response assessment, but generalization is limited by scarce annotations and domain shifts. Self-supervised learning (SSL) can address these challenges but remains underexplored in pan-cancer, multi-tracer PET-CT. In this work, we propose MUST-PET (MUltimodal Self-Supervised learning across Tracers), a multimodal, multi-tracer SSL framework for generalizable whole-body PET-CT lesion segmentation. MUST-PET is trained and validated on a diverse, multi-institutional collection of pan-cancer PET-CT scans acquired with FDG and prostate-specific membrane antigen (PSMA)-targeted radiotracers. MUST-PET uses context-aware masked reconstruction, where one modality is partially masked and reconstructed using complementary information from both PET and CT. The pretrained model is subsequently fine-tuned with labeled samples and evaluated for reconstruction quality, lesion segmentation, label efficiency, and generalizability across independent held-out datasets. MUST-PET reduces reconstruction error, improves lesion segmentation over training from scratch, and performs well with limited labeled data and on unseen external datasets, demonstrating the potential of multi-tracer SSL for label-efficient, generalizable whole-body PET-CT. segmentation.

Abstract

Medical Image Representation Learning

Explore similar work

MUST-PET: MUltimodal Self-supervised learning across Tracers for whole-body PET/CT-based lesion segmentation

Developing Foundation Models for Universal Segmentation from 3D Whole-Body Positron Emission Tomography

An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation