Spatial Atlas-Informed Loss Ensembles for Whole-Body PSMA PET/CT Metastasis Segmentation
Authors: Grant N. Chau, Bashirul A. Biswas, Biratal Wagle, Matthew E. Maeder, Michael S. Leapman, James B. Yu, Indrani Bhattacharya
Organizations: Department of Computer Science, Dartmouth College, Hanover, NH, USA · Department of Biomedical Data Science, Geisel School of Medicine at Dartmouth, Hanover, NH, USA · Radiology, Dartmouth Hitchcock Medical Center, Lebanon, NH, USA · Department of Urology, Yale School of Medicine, New Haven, CT, USA · Radiation Oncology, Dartmouth Hitchcock Medical Center, Lebanon, NH, USA
Existing whole-body PSMA PET/CT prostate metastasis segmentation methods generally rely on images alone, without explicitly incorporating where metastases tend to occur. We investigate whether a population metastasis atlas can improve lesion detection when integrated into model training. We constructed voxel-wise mean and standard deviation maps from 375 expert-annotated training scans and used them to weight the cross-entropy loss of an nnU-Net. The additional penalties for missed lesions and false positives depend on spatial atlas statistics and diminish as predictions become confidently correct. Seven configurations were compared on validation data; three were combined by majority voting into SAILS (Spatial Atlas-Informed Loss enSemble). We compared SAILS with a baseline nnU-Net and inference-time atlas fusion on AutoPETIII (110 disease-positive cases) and Deep-PSMA (100 external cases). No atlas or registration is needed at inference. Results: On AutoPETIII, SAILS improved whole-body Dice (0.531 vs. 0.520; p=0.006), reduced missed-lesion volume (9.5 vs. 12.1 cc/case; p<0.01), and increased pooled lesion sensitivity (0.857 vs. 0.843; p=0.004). On Deep-PSMA, SAILS detected 126 additional reference lesions (sensitivity 0.769 vs. 0.751; p<0.001), without a significant whole-body Dice change (-0.009; p=0.36). High-risk bone Dice improved in both cohorts (0.576 vs. 0.552 and 0.667 vs. 0.653; both p<0.01). False-positive volume increased by 2.3 and 18.2 cc/case, respectively (p<0.001). Atlas-weighted training improves lesion detection, especially in high-risk bone, without test-time atlas registration, but increases false positives.
Figures & tables
Property
AutoPET-III
Deep-PSMA
# Scans
597
100
Sex (M/F)
597/0
100/0
Disease (Pr/Neg)
537/60
100/0
# Lesions per case
35.04±51.70
68.41±58.35
μPET per lesion
7.48±7.12
5.64±3.33
# Slices per volume
135–963
195–1261
Table 1: Summary of the two PSMA PET/CT cohorts. Pr = prostate cancer, Neg = negative, dz = slice spacing, μPET = mean PET intensity (per lesion). Cohort statistics are reproduced from the dataset summary in [ 53 ] . The train/val/test counts sum to 593 rather than 597 since repeat scans of patients already in the training partition were excluded (Section 2.2 ).
Figure 1: Distribution of scanner, tracer, metastatic pattern, and number of scans per patient (longitudinal) in the two evaluation cohorts. Each pie is divided into two halves, AutoPET III (top) and Deep-PSMA (bottom), each subdivided by that cohort’s category proportions. Scanner and tracer are not reported per case for Deep-PSMA ( ∗ ) and are shown as a uniform distribution over the reported categories.
Figure 2: Overview of the proposed SAIL and SAILS framework. (A) Atlas construction: 375 annotated PSMA PET/CT training scans are registered to a common reference, and their warped lesion masks are aggregated voxel-wise to construct the mean ( μ ) and standard deviation ( σ ) metastasis atlases. These atlases are then mapped to individual training cases. (B) SAIL training: the atlas maps, reference lesion mask, and network predictions are used to compute an atlas-weighted loss that modulates cross-entropy according to spatial metastasis statistics and prediction confidence. The segmentation network receives only CT and PET as inputs. (C) SAILS inference: networks trained with different atlas-weighted loss configurations produce binary segmentations that are combined through voxel-wise majority voting. No atlas or registration is required at inference.
Class
Prediction
pv
Kernel
CE Weight ( 1+Wv )
FG
Correct
→1
gα(pv)→0
→1
Incorrect
→0
gα(pv)→1
→1+Afg(v)
BG
Correct
→0
gα(1−pv)→0
→1
Incorrect
→1
gα(1−pv)→1
→1+Abg(v)
Table 2: Effect of the exponential weighting kernel on atlas-derived contributions for correctly and incorrectly predicted foreground (FG, reference lesion) and background (BG) voxels.
Figure 3: Behavior of the confidence kernel and the atlas-weighted multiplier. (a) The confidence kernel gα(p) for several α ; dotted line is the α→0 limit 1−p and the dashed line is α=10 , close to the step-function limit. All experiments use α=1 . (b) The resulting multiplier 1+g1(p)A for several atlas weights A (with α=1 ); A=0 recovers standard cross-entropy. For background voxels, p is replaced by 1−p .
Group
Configuration
Atlas
Afg
Abg
Atlas at test
(A) Baseline
nnU-Net
None
—
—
No
(B) Atlas as input
Extra input channel
Mean
—
—
Yes
(C) Inference fusion
Calibrated log-odds [ 42 ]
Mean
—
—
Yes
(D) Atlas-weighted
D1 Fg only
Mean
μ~
—
No
loss
D2 Fg only
Std
σ~
—
No
D3 Bg only
Mean
—
1−μ~
No
Table 3: Configurations compared, grouped by where the atlas enters the pipeline. Atlas: which atlas statistic is used (none, mean μ~ , standard deviation σ~ , or both). Afg , Abg : atlas maps weighting the foreground (missed lesion) and background (false-positive) cross-entropy terms (Eq. ( 6 )); a dash means that term is unweighted. All loss configurations use α=1 . Atlas at test: whether per-patient registration of the atlas is needed at test time. SAILS combines the predictions of D3, D5 and D6 by majority voting (Section 2.7 ).
Configuration
Dice
Δ Dice (95% CI)
Δ FNVol
Δ FPVol
Baseline
0.519
—
—
—
Atlas as input (B)
0.508
−0.011 [ −0.025 , +0.002 ]
0.0
+0.3
Inference-time fusion (C)
0.526
+0.007 [ +0.002 , +0.013 ]
−1.0
+4.2
(D) Atlas-weighted loss configurations
D1: Fg μ~
0.506
−0.013 [ −0.025 , −0.000 ]
−1.2
−1.1
D2: Fg σ~
0.520
+0.001 [ −0.010 , +0.012 ]
−1.1
+5.2
Table 4: Validation-set results (AutoPET III, 84 disease-positive cases). Fg/Bg: atlas map weighting the foreground/background cross-entropy term, as in Table 3 ; all loss configurations use α=1 . Voting variants combine D3, D5 and D6: voxel vote, a voxel is labeled lesion if at least k networks predict it; lesion vote, a connected component of the union is kept if at least k networks detect it. Δ Dice: paired mean difference to the baseline with bootstrap 95% CI. Δ FNVol / Δ FPVol: paired mean change in false-negative / false-positive volume per case (cc); negative values are better. Inference-time fusion parameters were tuned on this set. † : selected as SAILS.
Cohort
Method
Dice
Δ Dice (95% CI), p
FNVol ( Δ , p )
FPVol ( Δ , p )
AutoPET III
Baseline
0.520±0.292
—
12.14
5.47
( n=110 )
Fusion
0.526±0.292
+0.007 [ −0.002 , +0.014 ]
10.99 ( −1.15 )
7.14 ( +1.67 )
SAILS
0.531±0.290
+0.011 [ +0.000 , +0.022 ], 0.006
9.46 ( −2.68 , < 0.01)
7.75 ( +2.28 , < 0.001)
Deep-PSMA
Baseline
0.590±0.156
—
12.65
25.15
( n=100 )
Fusion
0.619±0.160
+0.028 [ +0.025 , +0.032 ]
11.45 ( −1.21 )
27.11 ( +1.96 )
SAILS
0.582±0.153
−0.009 [ −0.021 , +0.002 ], 0.359
9.95 ( −2.70 , < 0.01)
43.36 ( +18.21 , < 0.001)
Table 5: Whole-body voxel-level performance on the test cohorts (disease-positive cases). Dice: mean ± SD. Δ Dice: paired mean difference to the baseline with bootstrap 95% CI. FNVol / FPVol: mean missed-lesion and false-positive volume per case (cc), with the paired difference to the baseline in parentheses. p : two-sided Wilcoxon signed-rank test vs. baseline (Holm-adjusted over Dice and FNVol). Inference-time fusion requires registering the atlas to every test patient; SAILS does not.
Cohort (reference lesions)
Method
Sens / PPV
p (Sens)
+TP / +FP
AutoPET III
Baseline
0.843 / 0.793
—
—
(3,231)
Fusion
0.859 / 0.762
—
54 / 110
SAILS
0.857 / 0.759
0.004
46 / 150
Deep-PSMA
Baseline
0.751 / 0.887
—
—
(6,841)
Fusion
0.770 / 0.876
—
139 / 89
SAILS
0.769 / 0.865
< 0.001
126 / 198
Table 6: Lesion-level detection on the test cohorts. Sens / PPV: lesion sensitivity and positive predictive value, pooled over all lesions in the cohort. +TP / +FP: total number of additional detected reference lesions and additional false-positive components in the cohort, relative to the baseline. p : two-sided Wilcoxon signed-rank test on per-patient sensitivity vs. baseline.
Cohort
Region
n
Baseline
Fusion
SAILS
AutoPET III
Bone
86
0.553
0.551 ( −0.001 [ −0.020 , +0.017 ])
0.560 ( +0.007 [ −0.003 , +0.017 ]) ∗
HR-bone
73
0.552
0.574 ( +0.021 [ +0.008 , +0.035 ]) ∗
0.576 ( +0.024 [ +0.006 , +0.047 ]) ∗
Liver
2
—
—
—
Deep-PSMA
Bone
98
0.588
0.613 ( +0.024 [ +0.020 , +0.028 ]) ∗
0.596 ( +0.008 [ −0.004 , +0.017 ]) ∗
HR-bone
91
0.653
0.684 ( +0.030 [ +0.026 , +0.035 ]) ∗
0.667 ( +0.014 [ +0.010 , +0.018 ]) ∗
Liver
18
0.187
0.231 ( +0.044 [ +0.022 , +0.069 ]) ∗
0.222 ( +0.035 [ −0.021 , +0.131 ])
Table 7: Region-level metastasis Dice on the test cohorts: the Dice of the predicted and reference lesion masks restricted to voxels inside each region (not the Dice of the organ itself). Means are over cases evaluable for all three methods ( n ). Each cell gives the mean Dice and the paired difference to the baseline [95% bootstrap CI]. ∗ Wilcoxon signed-rank p<0.01 vs. baseline. HR-bone: high-risk bone. Liver in AutoPET III has too few cases (2) and is not tested.
Figure 4: Change in error type relative to the baseline for D3, D5, D6 and their ensemble (SAILS): false-negative volume, false-positive volume (cm 3 per case) and per-case lesion sensitivity. Teal marks improvement and brown marks worsening; each panel has its own symmetric color scale. ∗ Holm-adjusted p<0.01 .
Figure 5: Lesions recovered by SAILS. Each row shows one example lesion, at the center of the panel, that the baseline missed or segmented only minimally and that SAILS segmented more completely, for each anatomic region (tag in the CT panel) and test cohort. Columns: CT, PET (SUV 0–8, inverted gray), and the ground truth (green), baseline (orange), and SAILS (blue) outlined on the CT. Each row is an axial slice through the lesion. The number in the ground-truth panel is the volume of the example lesion. Other lesions visible in the panel are also outlined. Examples were selected based on improvement in lesion coverage and lesion volume, accounting for false-positive volume added within 4 cm, and illustrate the effect of the method rather than typical performance. Regions or cohorts without an eligible lesion have no row.
Figure 6: Lesions refined by SAILS. Each row shows one example lesion that the baseline detected but under-segmented and that SAILS segmented more completely. Display conventions, lesion volumes, and example selection are as in Fig. 5 .
Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.
Accurate lesion segmentation from whole-body Positron Emission Tomography (PET)/Computed Tomography (CT) scans is essential for cancer staging and treatment planning. PET provides functional metabolic information with different radiotracers, while CT offers anatomical localization. Lesion delineation from PET/CT imaging is clinically challenging due to subtle imaging features, confounders, and inter-reader variability. Existing deep learning approaches suffer from training-related stochasticity, inconsistent predictions, missed lesions in high tumor-burden cases, and lack uncertainty quantification, limiting their clinical reliability. Using nnU-Net as a baseline, we propose an uncertainty-aware framework for whole-body PET/CT lesion segmentation that integrates (1) Bayesian ensembling to reduce training stochasticity, (2) voxel-wise uncertainty quantification with epistemic and aleatoric decomposition, and (3) epistemic uncertainty-augmented training to improve lesion detection. Two public datasets, AutoPET-III (1,611 scans) and Deep-PSMA (200 scans), comprising FDG and PSMA studies across multiple cancer types, are used for training and evaluation. Bayesian ensembling improves robustness and performance over deterministic nnU-Net models on the unseen AutoPET-III test set. Uncertainty maps highlight regions of model disagreement and correlate with misclassifications, particularly false positives. Uncertainty-augmented training improves lesion recovery at the cost of increased FPVol, reflecting a precision-recall trade-off. A case-adaptive routing strategy further improves Dice by selecting between the base and augmented models. To our knowledge, this is the first study to systematically investigate uncertainty quantification in multi-tracer, pan-cancer PET/CT segmentation and to combine Bayesian ensembling with uncertainty-aware modeling for this task.
Bashirul Azam Biswas, Biratal Raj Wagle, Zhihan Yang +4
Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA
We present an anatomy-aware, promptable model for whole-body lesion segmentation in FDG and PSMA PET/CT, developed for the AUTOPET V challenge. The proposed method is built as family of nnU-Net-based models and trained in two stages: i) a pre-training stage that produces a strong initial segmentation, and ii) an online interactive stage that learns to exploit scribble prompts, refining the prediction over successive interactions. Anatomical context is incorporated through organ supervision using a single shared head that predicts lesions and organs from the same features, which reduces false positives arising from physiological uptake. Also as the tracer (i.e., FDG/PSMA) is not provided at inference, we add a tracer classifier based on image processing and a random forest over coronal MIP features, routing each study to a combined FDG+PSMA model or to a PSMA-specific model. Across four-fold cross-validation the organ-supervised model achieves the best and most stable performance, the interactive stage improves the Dice score monotonically with each prompt, and PSMA-specific training yields the strongest tracer-wise results.
Pablo Lozano-Jimenez, Sergio Romero-Tapiador, Ruben Tolosana
University of Amsterdam, Faculty of Science, Amsterdam, The Netherlands · BiometricsAI, Universidad Autónoma de Madrid, 28049 Madrid, Spain
Deep learning-based whole-body PET-CT lesion segmentation can support cancer staging, treatment planning, and response assessment, but generalization is limited by scarce annotations and domain shifts. Self-supervised learning (SSL) can address these challenges but remains underexplored in pan-cancer, multi-tracer PET-CT. In this work, we propose MUST-PET (MUltimodal Self-Supervised learning across Tracers), a multimodal, multi-tracer SSL framework for generalizable whole-body PET-CT lesion segmentation. MUST-PET is trained and validated on a diverse, multi-institutional collection of pan-cancer PET-CT scans acquired with FDG and prostate-specific membrane antigen (PSMA)-targeted radiotracers. MUST-PET uses context-aware masked reconstruction, where one modality is partially masked and reconstructed using complementary information from both PET and CT. The pretrained model is subsequently fine-tuned with labeled samples and evaluated for reconstruction quality, lesion segmentation, label efficiency, and generalizability across independent held-out datasets. MUST-PET reduces reconstruction error, improves lesion segmentation over training from scratch, and performs well with limited labeled data and on unseen external datasets, demonstrating the potential of multi-tracer SSL for label-efficient, generalizable whole-body PET-CT. segmentation.
Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA