LoRe-MIM: Lost-or-Replaced Masked Image Modeling for Self-Supervised Representation Learning in Whole-Body PET/CT
Authors: Amartya Bhattacharya, Bashirul Azam Biswas, Biratal Raj Wagle, Michael S. Leapman, Matthew E. Maeder, James B. Yu, Indrani Bhattacharya
Organizations: Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Hanover, NH 03755, USA · Department of Urology, Yale University, New Haven, CT, USA · Radiology, Dartmouth Hitchcock Medical Center, Lebanon, NH 03766 , USA · Radiation Oncology, Dartmouth Hitchcock Medical Center, Lebanon, NH 03766 , USA
Deep learning models for whole-body positron emission tomography/computed tomography (PET/CT) hold significant clinical potential for disease detection, treatment planning, and response assessment. However, the time and cost associated with expert annotations remain major bottlenecks in developing robust and generalizable models. Although self-supervised learning (SSL) has emerged as a promising approach to mitigating annotation scarcity, its application to whole-body PET/CT remains relatively underexplored due to the limited availability of large-scale datasets and the challenges associated with learning complementary cross-modal representations. To address these limitations, we curate a heterogeneous, multi-tracer, multi-institutional cohort comprising 6,531 whole-body PET/CT scans and propose LoRe-MIM (Lost-Or-Replaced Masked Image Modelling), a novel self-supervised framework designed to learn multimodal representations through masked image reconstruction. Specifically, LoRe-MIM introduces a masking strategy in which image regions are either replaced with zeros (Lost) or substituted with corresponding complementary imaging modality (Replaced), encouraging the model to capture both modality-specific characteristics and cross-modal relationships. We evaluate the learned representations across multiple downstream tasks, including multimodal PET/CT image reconstruction, patch-based classification into cancer vs. non-cancer, whole-body lesion segmentation, label-efficient training, and robustness assessment. Experimental results demonstrate that LoRe-MIM consistently outperforms the corresponding state-of-the-art baselines across these tasks, highlighting its potential as a scalable and generalizable self-supervised learning framework for whole-body PET/CT analysis.
Figures & tables
Properties
AutoPET-III
DeepPSMA
DHMC
SPADE
VI-MED
Disease dist.
Me/Lu/Ly/Pr/Neg 188/168/145/537/573
Pr
Pr
–
–
FDG/PSMA
✓/✓
✓/✓
✗/✓
✓/✗
✓/✗
# lesions (per case)
17.47 ± 46.12
69.99 ± 60.04
10.13 ± 16.27
–
–
# slices (per vol)
135–963
195–1261
303–380
103-307
207-551
Pretraining
Yes
No
Yes
Yes
Yes
Segmentation
Yes
Yes
No
No
No
Table 1: Summary of datasets used. The FDG/PSMA row marks tracer availability (✓ present, ✗ absent). Lu=Lung, Ly=Lymphoma, Me=Melanoma, Pr=Prostate, Neg=Negative.
Figure 1: Overview of the proposed masking approach.
Masking Approach
Reconstruction
Classification
MAE ↓
SSIM ↑
PSNR ↑
AUC ↑
FDG-Only MAE [ 24 ]
0.2797
0.7288
32.56
0.6195
LO-MIM
0.4026
0.6473
30.01
0.6885
LB-MIM
0.3831
0.6835
30.14
0.6894
Re-MIM
0.2803
0.7586
32.73
0.7010
LoRE-MIM
0.2675
0.7762
33.07
0.7284
Table 2: Ablation study of masking strategies under fixed masking. Reconstruction performance on AutoPET ( n=247 ) and DHMC ( n=149 ) validation samples, averaged equally across CT and PET and across all 396 cases, is evaluated using MAE ( ↓ ), SSIM ( ↑ ), and PSNR in dB ( ↑ ). Classification performance on clean test data is evaluated using AUC ( ↑ ). Bold indicates the best result for each metric.
Model
Augmentation
Variable Masking
Reconstruction
Classification
MAE ↓
SSIM ↑
PSNR ↑
AUC ↑
Re-MIM
×
×
0.3913
0.6784
29.86
0.6855
✓
×
0.4131
0.6513
29.70
0.5783
×
✓
0.4258
0.6568
28.82
0.5695
✓
✓
0.4989
0.5738
28.35
0.5525
LoRE-MIM
×
×
0.2718
0.7701
32.85
0.7091
Table 3: Ablation study of augmentation and variable masking for CM and LoRE-MIM on AutoPET-III ( n=321 ) and DHMC ( n=100 ) test sets. Reconstruction metrics are evaluated over masked regions after whole-volume reconstruction, averaged equally across CT and PET and across all 421 cases. Masked blocks are replaced with Gaussian samples of standard deviation 0.1. PSNR is in dB. Classification AUC values are retained from the original experiments. Bold indicates the best result for each metric.
Masking Approach
Reconstruction
Classification
MAE ↓
SSIM ↑
PSNR ↑
AUC ↑
FDG-Only
0.2812
0.7256
32.45
0.6782
LO-MIM
0.4080
0.6432
29.74
0.6695
LB-MIM
0.3913
0.6784
29.86
0.6626
Re-MIM
0.2857
0.7510
32.51
0.6855
LoRE-MIM
0.2718
0.7701
32.85
0.7091
Table 4: Reconstruction and classification performance. Reconstruction is evaluated on the AutoPET-III ( n=321 ) and DHMC ( n=100 ) test sets using masked-region MAE ( ↓ ), SSIM ( ↑ ), and PSNR in dB ( ↑ ), computed after whole-volume reconstruction and averaged equally across CT and PET modalities and across all 421 test cases. Classification performance is evaluated using AUC ( ↑ ), with values retained from the original classification experiments.
Feature Extraction Stage / Model
Feature Type
Reduction
AUC ↑
Enc
Flatten
KPCA (128)
0.6898
PCA (128)
0.6565
GAP
KPCA (128)
0.6808
PCA (128)
0.6974
KPCA (384)
0.7079
PCA (384)
0.7180
Table 5: Evaluation of patch-based classification using features extracted from different model stages and baseline architectures under clean input conditions. We compare feature aggregation and dimensionality reduction strategies using logistic regression (LR) for classification. Performance is evaluated using AUC ( ↑ ). Bold and underline indicate the best overall and best encoder performance, respectively.
Category
Model
Gaussian Noise
Misaligned
CT Missing
PET Missing
Baseline
ResNet50-3D
0.7268
0.7332
0.6217
0.5348
DINOv3
0.7555
0.7286
0.6500
0.5344
Ours
LoRe-MIM (Dec 4 )
0.7584
0.7648
0.6975
0.5995
Table 6: Robustness evaluation of patch-based classification under four PET/CT input perturbations: Gaussian noise, PET/CT misalignment, missing CT, and missing PET. Performance is evaluated using AUC ( ↑ ). Bold indicates the best result for each perturbation.
Dataset
Baseline
Training Strategy
Variant
Dice ↑
FPVol ↓
FNVol ↓
Lesion Sens. ↑
PPV ↑
AutoPET-III
nnU-Net
Scratch
–
0.566
17.283
9.472
0.773
0.733
SwinUNETR
Scratch
–
0.526
27.147
7.586
0.773
0.546
LoRe-MIM
Full FT
0.577
15.594
9.832
0.775
0.664
Dec. FT
0.569
19.207
7.405
0.764
0.713
Enc. + Dec. FT
0.581
15.662
8.362
0.776
0.675
Deep-PSMA
nnU-Net
Scratch
–
0.561
73.030
7.933
0.757
0.768
Table 7: Lesion segmentation performance on the AutoPET-III ( n=321 ) and Deep-PSMA ( n=200 ) test sets using different baseline architectures and training strategies. Three finetuning variants of LoRe-MIM (Ours) are evaluated: Full FT, Dec. FT, and Enc. + Dec. FT. FPVol and FNVol are reported in cc. ↑ : higher is better; ↓ : lower is better.
Figure 2: Tracer-specific voxel-level evaluation on the AutoPET-III test set, comparing nnU-Net and SwinUNETR trained from scratch with SwinUNETR pretrained using LoRe-MIM under different finetuning configurations.
Figure 3: Tracer-specific voxel-level evaluation on the Deep-PSMA test set, comparing nnU-Net and SwinUNETR trained from scratch with SwinUNETR pretrained using LoRe-MIM under different finetuning configurations.
Figure 4: Anatomical region-level comparison of lesion segmentation performance on AutoPET-III, comparing nnU-Net and SwinUNETR trained from scratch with three finetuning variants of our LoRe-MIM pretrained SwinUNETR.
Figure 5: Anatomical region-level comparison of lesion segmentation performance on Deep PSMA, comparing nnU-Net and SwinUNETR trained from scratch with three finetuning variants of our LoRe-MIM pretrained SwinUNETR.
Figure 6: Label-efficiency curves for whole-body lesion segmentation on the AutoPET-III and Deep-PSMA datasets. Segmentation performance is evaluated using Dice scores across varying proportions of labeled training data (1%–100%). All three finetuning strategies outperform SwinUNETR and nnU-Net trained from scratch, with more pronounced improvements under limited labeled data (1%–10%) and diminishing performance gains as the proportion of labeled data increases.
Figure 7: Qualitative comparison of lesion segmentation performance across four representative cases (Rows 1–4). Columns 1 and 2 show the CT and PET images with ground-truth lesion contours (green), respectively. Columns 3 and 4 present predictions from nnU-Net and SwinUNETR trained from scratch, respectively, while Columns 5–7 show predictions from three SwinUNETR finetuning strategies initialized with our pretrained LoRe-MIM model. Row 1 illustrates successful liver lesion recovery in an AutoPET-III case, where both scratch-trained models fail to detect the lesion, whereas all three LoRe-MIM variants successfully identify it. Row 2 demonstrates false-positive elimination in an AutoPET-III case, where both scratch-trained models incorrectly predict a lesion, while all three LoRe-MIM variants correctly produce no lesion segmentation. Row 3 presents a high-risk bone lesion from Deep-PSMA, where scratch-trained SwinUNETR misses the lesion, while all three LoRe-MIM variants successfully recover it. Finally, Row 4 illustrates a failure case involving a liver lesion from Deep-PSMA, where decoder-only finetuning fails to detect the lesion and encoder–decoder finetuning yields an incomplete segmentation, whereas full finetuning and scratch-trained nnU-Net successfully recover the lesion.
Figure 8: Qualitative comparison of whole-body PET/CT lesion segmentation between models trained from scratch (nnUNet and SwinUNETR) and our proposed LoRe-MIM pretrained SwinUNETR. Columns 1 and 2 show the coronal CT and PET images, respectively, with ground-truth lesion contours highlighted in green. Columns 3 and 4 present the segmentation predictions from nnUNet and SwinUNETR trained from scratch, respectively. Both models correctly identify the true-positive lesion (green arrow) but introduce an additional false-positive prediction (red arrow). In contrast, our proposed LoRe-MIM pretrained SwinUNETR (Column 5) successfully eliminates the false-positive prediction while preserving the true-positive lesion, demonstrating improved lesion discrimination.
Figure 9: Qualitative comparison of masked-image reconstruction for CT (top) and PET (bottom). From left to right: clean reference, masked input, CM reconstruction, and LoRE-MIM reconstruction. Red region outlines highlight corresponding regions where LoRE-MIM provides visually closer agreement with the clean reference. CM and LoRE-MIM reconstructions are outlined in red and dark green, respectively.
Deep learning-based whole-body PET-CT lesion segmentation can support cancer staging, treatment planning, and response assessment, but generalization is limited by scarce annotations and domain shifts. Self-supervised learning (SSL) can address these challenges but remains underexplored in pan-cancer, multi-tracer PET-CT. In this work, we propose MUST-PET (MUltimodal Self-Supervised learning across Tracers), a multimodal, multi-tracer SSL framework for generalizable whole-body PET-CT lesion segmentation. MUST-PET is trained and validated on a diverse, multi-institutional collection of pan-cancer PET-CT scans acquired with FDG and prostate-specific membrane antigen (PSMA)-targeted radiotracers. MUST-PET uses context-aware masked reconstruction, where one modality is partially masked and reconstructed using complementary information from both PET and CT. The pretrained model is subsequently fine-tuned with labeled samples and evaluated for reconstruction quality, lesion segmentation, label efficiency, and generalizability across independent held-out datasets. MUST-PET reduces reconstruction error, improves lesion segmentation over training from scratch, and performs well with limited labeled data and on unseen external datasets, demonstrating the potential of multi-tracer SSL for label-efficient, generalizable whole-body PET-CT. segmentation.
Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA
Positron emission tomography (PET) is a key nuclear medicine imaging modality that visualizes radiotracer distributions to quantify in vivo physiological and metabolic processes, playing an irreplaceable role in disease management. Despite its clinical importance, the development of deep learning models for quantitative PET image analysis remains severely limited, driven by both the inherent segmentation challenge from PET's paucity of anatomical contrast and the high costs of data acquisition and annotation. To bridge this gap, we develop generalist foundational models for universal segmentation from 3D whole-body PET imaging. We first build the largest and most comprehensive PET dataset to date, comprising 11041 3D whole-body PET scans with 59831 segmentation masks for model development. Based on this dataset, we present SegAnyPET, an innovative foundational model with general-purpose applicability to diverse segmentation tasks. Built on a 3D architecture with a prompt engineering strategy for mask generation, SegAnyPET enables universal and scalable organ and lesion segmentation, supports efficient human correction with minimal effort, and enables a clinical human-in-the-loop workflow. Extensive evaluations on multi-center, multi-tracer, multi-disease datasets demonstrate that SegAnyPET achieves strong zero-shot performance across a wide range of segmentation tasks, highlighting its potential to advance the clinical applications of molecular imaging.
Yichi Zhang, Le Xue, Wenbo Zhang +16
Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of Artificial Intelligence for Science, Shanghai, China. · Department of Nuclear Medicine and PET Center, Huashan Hospital, Fudan University, Shanghai, China. +6
The synergistic interpretation of anatomical information from computed tomography (CT) and metabolic information from positron emission tomography (PET) is important to oncologic imaging. However, existing deep learning methods for PET/CT remain largely task-specific, are often trained on single-center cohorts, or adopt dual-branch fusion schemes that delay cross-modal interaction and underutilize early spatial correspondence between PET and CT. To address these limitations, we present an open-source, multi-center, whole-body FDG PET/CT foundation model utilizing 4,997 harmonized scans from four public datasets. Our framework employs hierarchical UNet-shaped backbones with early channel-wise concatenation, enabling anatomical and metabolic features to interact from the first embedding layer onward. We further introduce a masked autoencoding objective based on zero-mean imputation, combined with a weighted global reconstruction loss. This design avoids non-physical intensity discontinuities at masked-region boundaries that arise from learnable mask tokens. On downstream AutoPET lesion segmentation, the proposed models demonstrate strong label efficiency: with only 10% of the labeled training data, they achieve performance comparable to models trained from scratch on the full dataset. Under extreme 5-shot linear probing, joint PET/CT pretraining also achieves higher Dice scores than separated-modality pretraining. This multi-center foundation model demonstrates label efficiency and cross-modality representation learning for PET/CT tumor segmentation. It provides a robust, open-source basis for advancing automated oncologic imaging, significantly reducing the need for large-scale manual annotations in clinical practice.
Xiaofeng Liu, Qianru Zhang, Thibault Marin +4
Department of Radiology and Biomedical Imaging, Yale Biomedical Imaging Institute, Yale University, New Haven, CT, USA