When Two Tracers Disagree: An Investigation of Multimodal Fusion for Clinical PET/CT Segmentation

Jack A. Johnson; Bartłomiej W. Papież

cs.CVAug 19, 2026

When Two Tracers Disagree: An Investigation of Multimodal Fusion for Clinical PET/CT Segmentation

Authors: Jack A. Johnson, Bartłomiej W. Papież

Organizations: Nuffield Department of Medicine, University of Oxford, UK · Department of Oncology, University of Oxford, UK · Big Data Institute, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom

Abstract

PSMA and FDG PET/CT visualise complementary biological information in prostate cancer. Combining both tracers could capture heterogeneous tumour phenotypes that may be missed by either alone, yet there is no consensus on effective deep learning architectures for fusing these modalities. We evaluated multimodal image-fusion strategies for automatic whole-body PET/CT lesion segmentation to estimate total tumour burden. Using the public DEEP-PSMA Challenge dataset, we trained tracer-specific 3D nnU-Net baselines and compared (i) early fusion with a single encoder and one decoder (OEOD) or two decoders (OETD), and (ii) intermediate fusion via a dual-encoder cross-attention U-Net (DECA-UNet). Tracer-specific baselines performed strongly (PSMA Dice = 0.93; FDG = 0.81). Fusion yielded mixed results: OEOD produced a combined Dice of 0.90 (on an easier, non-tracer-specific task), whilst the tracer-specific fusion models reached PSMA/FDG = 0.69/0.64 (OETD) and 0.76/0.57 (DECA-UNet). Whilst fusion often provided reasonable PSMA segmentation, FDG performance degraded and no strategy consistently exceeded the single-tracer baselines. Under the evaluated setting, tracer-specific models remain the stronger baseline; clinically useful gains from multimodal fusion will likely require architectures that better preserve tracer specific representations. Our code is available at: https://github.com/JackJ3636/DEEP_PSMA_code

Figures & tables

Model	Tracer	Dice	FP	FN	SD	SUV-R	TTB-R
Baseline FDG	FDG	0.81 $\pm$ 0.27	6.5 $\pm$ 12	18 $\pm$ 38	0.81 $\pm$ 0.25	1.0 $\pm$ 0.09	1.6 $\pm$ 5.3
Baseline PSMA	PSMA	0.93 $\pm$ 0.14	23 $\pm$ 52	33 $\pm$ 97	0.91 $\pm$ 0.14	1.0 $\pm$ 0.12	1.0 $\pm$ 0.39
OEOD ∗ (Early)	Comb.	0.90 $\pm$ 0.05	0.83 $\pm$ 1.9	3.8 $\pm$ 5.4	0.74 $\pm$ 0.09	0.99 $\pm$ 0.08	0.87 $\pm$ 0.09
OETD (Early)	PSMA	0.69 $\pm$ 0.32	59 $\pm$ 70	120 $\pm$ 130	0.59 $\pm$ 0.27	1.2 $\pm$ 2.6	1.3 $\pm$ 1.7
	FDG	0.64 $\pm$ 0.32	26 $\pm$ 33	46 $\pm$ 56	0.55 $\pm$ 0.27	0.53 $\pm$ 0.63	1.2 $\pm$ 0.88
DECA-UNet	PSMA	0.76 $\pm$ 0.30	56 $\pm$ 67	170 $\pm$ 410	0.71 $\pm$ 0.24	0.98 $\pm$ 0.16	0.92 $\pm$ 0.49

Table 1: Dice, FP and FN volumes (mL), Surface Dice (SD), SUV Mean Ratio (SUV-R), and TTB Volume Ratio (TTB-R) for all models. Mean

\pm

\sigma

reported to two significant figures. ∗ OEOD predicts a single combined-tracer mask and is evaluated with one combined score; its row is therefore not directly comparable to the per-tracer rows and is excluded from the tracer-specific comparisons.

Figure 1: Coronal slices of segmentation performance for Baseline models, OETD (early fusion), DECA-UNet (intermediate fusion), and OEOD (early fusion) on Training Case 96 for both FDG and PSMA tracers (combined mapping for OEOD). Red : ground truth only; Blue : prediction only (false positive); Purple : overlap between prediction and ground truth (true positive).

Model	Tracer	Mean Dice	95% CI
Baseline	PSMA	0.93	[0.90, 0.96]
Baseline	FDG	0.81	[0.76, 0.86]
OEOD (ref.)	Comb.	0.90	[0.89, 0.91]
OETD	PSMA	0.69	[0.63, 0.75]
OETD	FDG	0.64	[0.58, 0.70]
DECA-UNet	PSMA	0.76	[0.70, 0.82]

Table 2: Mean Dice with

95\%

confidence intervals for the mean (

\bar{x}\pm t_{0.975,99}\,s/\sqrt{N}

N=100

), derived from the reported per-case mean and standard deviation. OEOD predicts a single combined-tracer mask and is shown for reference only.

Panel of Figure 2 — Figure 2: Zoomed coronal view comparison of the ground truth, false positives and segmentations for the abdomen region for baseline nnU-Nets (left) and DECA-UNet (intermediate fusion) (right) on Training Case 96. Whilst intermediate fusion did occasionally prevent some false positives in PSMA segmentation (left), this was accompanied by corresponding FDG performance (right) to degrade. Red : ground truth only; Blue : prediction only (false positive); Purple : overlap between prediction and ground truth (true positive).

Explore similar work

The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization
May 7, 2026Jakob Dexl, Katharina Jeblick, Andreas Mittermeier +27Domain Generalization PET/CT Imaging
Department of Radiology, LMU University Hospital, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Comprehensive Pneumology Center (CPC-M), Member of the German Center for Lung Research (DZL), Munich, Germany +13
An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation
May 20, 2026Xiaofeng Liu, Qianru Zhang, Thibault Marin +4Cross-Modality Medical Image Segmentation Medical Imaging Foundation Models
Department of Radiology and Biomedical Imaging, Yale Biomedical Imaging Institute, Yale University, New Haven, CT, USA
Improving PET/CT-Based Whole-Body Lesion Segmentation Using Prediction Uncertainty-Augmented Models
Jun 8, 2026Bashirul Azam Biswas, Biratal Raj Wagle, Zhihan Yang +4Ensemble Learning PET/CT Imaging
Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA

May 7, 2026J/K move · Enter open · S save

The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization

Jakob Dexl, Katharina Jeblick, Andreas Mittermeier, Balthasar Schachtner+26

Department of Radiology, LMU University Hospital, LMU Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Comprehensive Pneumology Center (CPC-M), Member of the German Center for Lung Research (DZL), Munich, Germany+13

Domain Generalization PET/CT Imaging Medical Image Segmentation Medical Image Benchmarks PET/CT Tumor Segmentation

Open paper AskPDF arXiv

We report the design and results of the third autoPET challenge (MICCAI 2024), which benchmarked automated lesion segmentation in whole-body PET/CT under a compositional generalization setting. Training data comprised 1,014 [18F]-FDG PET/CT studies from the University Hospital Tübingen and 597 [18F]/[68Ga]-PSMA PET/CT studies from the LMU University Hospital Munich, constituting the largest publicly available annotated PSMA PET/CT dataset to date. The held-out test set of 200 studies covered four tracer-center combinations, two of which represented unseen compositional pairings. A complementary data-centric award category isolated the contribution of data handling strategies by restricting participants to a fixed baseline model. Seventeen teams submitted 27 algorithms, predominantly nnU-Net-based 3D networks with PET/CT channel concatenation. The top-ranked algorithm achieved a mean DSC of 0.66, FNV of 3.18 mL, and FPV of 2.78 mL across all four test conditions, improving DSC by 8% and reducing the false-negative volume by 5 mL relative to the provided baseline. Ranking was stable across bootstrap resampling and alternative ranking schemes for the top tier. Beyond the benchmark, we provide an in-depth analysis of segmentation performance at the patient and lesion level. Three main conclusions can be drawn: (1) in-domain multitracer PET/CT segmentation is sufficient and probably approaching reader agreement; (2) compositional generalization to unseen tracer-center combinations remains an open problem mainly driven by systematic volume overestimation; (3) heterogeneity and case difficulty drive performance variation substantially more than the choice of algorithm among top-ranked teams.

Abstract

Explore similar work

The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT \unicodex2013\unicode{x2013}\unicodex2013 Multitracer Multicenter Generalization

An Open Multi-Center Whole-Body FDG PET/CT Foundation Model for Tumor Segmentation

Improving PET/CT-Based Whole-Body Lesion Segmentation Using Prediction Uncertainty-Augmented Models

The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT $\unicode{x2013}$ Multitracer Multicenter Generalization