Bridging MRI and PET physiology: Untangling complementarity through orthogonal representations
Authors: Sonja Adomeit, Kartikay Tehlan, Lukas Förner, Katharina Weisser, Helen Scholtiseek, David Kaufmann, Julie Steinestel, Constantin Lapa, +2 more
Organizations: Dept. of diagnostic and interventional Radiology and Neuroradiology, University Hospital Augsburg, Germany · Digital Medicine, University Hospital Augsburg, Germany · Chair for Computer Aided Medical Procedures and Augmented Reality, Technical University of Munich, Germany · Bavarian Center for Cancer Research (BZKF) Augsburg, Germany · Dept. of Nuclear Medicine, University Hospital Augsburg, Germany · Dept. of Urology, University Hospital Augsburg, Germany · Center for Advanced Analytics and Predictive Sciences, University of Augsburg, Germany
Multimodal imaging analysis often relies on joint latent representations, yet these approaches rarely define what information is shared versus modality-specific. Clarifying this distinction is clinically relevant, as it delineates the irreducible contribution of each modality and informs rational acquisition strategies. We propose a subspace decomposition framework that reframes multimodal fusion as a problem of orthogonal subspace separation rather than translation. We decompose Prostate-Specific Membrane Antigen (PSMA) PET uptake into an MRI-explainable physiological envelope and an orthogonal residual reflecting signal components not expressible within the MRI feature manifold. Using multiparametric MRI, we train an intensity-based, non-spatial implicit neural representation (INR) to map MRI feature vectors to PET uptake. We introduce a projection-based regularization using singular value decomposition to penalize residual components lying within the span of the MRI feature manifold. This enforces mathematical orthogonality between tissue-level physiological properties (structure, diffusion, perfusion) and intracellular PSMA expression. Tested on 13 prostate cancer patients, the model demonstrates that residual components spanned by MRI features are absorbed into the learned envelope, while the orthogonal residual is largest in tumour regions. This indicates that PSMA PET contains signal components not recoverable from MRI-derived physiological descriptors. The resulting decomposition provides a structured characterization of modality complementarity grounded in representation geometry rather than image translation.
Figures & tables
Figure 1: Graphical abstract. Seven MRI sequences are used to synthesize a PSMA PET image using an INR; synthesis here is a means to isolate the orthogonal contribution of PSMA PET with respect to MRI. The network is trained with a loss that penalizes the parallel residual r∥ more heavily than the orthogonal residual r⊥ , where the total residual e=r∥+r⊥ is decomposed via SVD projection onto the subspace Π spanned by the MRI feature data.
Figure 2: Visualization of a patient case featuring: (a) the squared parallel residual; (b) the squared orthogonal residual; (c) the reconstructed SUV slice and (d) the ground truth SUV slice. Prostate masks are delineated white, while tumour ones in cyan (panels (b), (c) and (d)), and in pink (panel (b)).
Figure 3: Comparison of orthogonal and parallel error metrics, stratified by tissue type: intra-tumoral regions, non-cancerous prostate tissue, and remaining pelvic anatomy. n state remaining data points after outlier removal for visual clarity.
Tissue
Total MSE ± SD
MSE ∥± SD
MSE ⊥± SD
Tumour
0.9723±6.7172
6.1e−4±1.4e−3
0.9717±6.7171
Prostate
0.0875±0.4364
1.3e−4±4.0e−4
0.0874±0.4364
Surrounding
0.0942±0.8127
1.6e−5±1.1e−4
0.0941±0.8127
Table 1: Quantitative results for the full model for different tissues: differences are statistically significant ( p≪0.05 ).
Ablation Set
Total MSE ± SD
MSE ∥± SD
MSE ⊥± SD
Full Model
0.0967±0.8275
1.9e−5±1.2e−4
0.0967±0.8275
Leave-one-out
Minus T1
0.1323±0.9724
1.9e−5±1.4e−4
0.1323±0.9723
Minus T2
0.1410±0.9674
3.9e−5±3.6e−4
0.1410±0.9674
Minus ADC
0.1369±1.0131
3.5e−5±4.3e−4
0.1369±1.0131
Minus TTP
0.1228±0.9017
2.1e−5±1.1e−4
0.1228±0.9017
Table 2: Ablation study: Error metrics represent mean ± standard deviation across the cohort. MSE∥ and MSE⊥ denote parallel and orthogonal error components. Evaluated differences are significant ( p≪0.05 ) between the full model and both leave-one-out (minus vp ) and group ablation (no tofts).
PSMA and FDG PET/CT visualise complementary biological information in prostate cancer. Combining both tracers could capture heterogeneous tumour phenotypes that may be missed by either alone, yet there is no consensus on effective deep learning architectures for fusing these modalities. We evaluated multimodal image-fusion strategies for automatic whole-body PET/CT lesion segmentation to estimate total tumour burden. Using the public DEEP-PSMA Challenge dataset, we trained tracer-specific 3D nnU-Net baselines and compared (i) early fusion with a single encoder and one decoder (OEOD) or two decoders (OETD), and (ii) intermediate fusion via a dual-encoder cross-attention U-Net (DECA-UNet). Tracer-specific baselines performed strongly (PSMA Dice = 0.93; FDG = 0.81). Fusion yielded mixed results: OEOD produced a combined Dice of 0.90 (on an easier, non-tracer-specific task), whilst the tracer-specific fusion models reached PSMA/FDG = 0.69/0.64 (OETD) and 0.76/0.57 (DECA-UNet). Whilst fusion often provided reasonable PSMA segmentation, FDG performance degraded and no strategy consistently exceeded the single-tracer baselines. Under the evaluated setting, tracer-specific models remain the stronger baseline; clinically useful gains from multimodal fusion will likely require architectures that better preserve tracer specific representations. Our code is available at: https://github.com/JackJ3636/DEEP_PSMA_code
Jack A. Johnson, Bartłomiej W. Papież
Nuffield Department of Medicine, University of Oxford, UK · Department of Oncology, University of Oxford, UK · Big Data Institute, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom
Raw image intensities are a poor surrogate for similarity across images in correspondence problems. Further, while auxiliary segmentation labels can provide semantic identification and improve matching, they can assign only coarse identities to fine-grained anatomy and often cover only a subset of structures, particularly in whole-body settings. We present an approach that instead establishes correspondence using complementary representations of anatomy and demonstrate its utility for longitudinal whole-body PET/CT registration in the Learn2Reg 2026 PSMAReg Challenge. Our 3D foundation model, anatomix, is used to derive fine-grained, per-voxel semantic representations that are consistent across volumes, and we further employ off-the-shelf segmentation models to obtain coarse anatomical identities. Together, these representations drive a GPU-accelerated iterative registration solver (FireANTs) in a hybrid multistage framework: the first stage estimates alignment from dense multi-channel features, after which a second refines the deformation using both anatomix- and segmentation-based losses. Lastly, because registration should not introduce non-biological tumor volume changes across time, we develop lesion-specific component-wise volume preservation schemes. Without any task-specific training, our system won first place in the PSMAReg Challenge leaderboard. Code and interactive visualizations are available at http://www.neeldey.com/reps-for-reg.
Neel Dey, Adrian V. Dalca
A. A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital · Department of Radiology, Harvard Medical School · Computer Science and AI Lab, Massachusetts Institute of Technology
Clinicians diagnose brain tumors by synthesizing patient symptoms, medical history, and quantitative imaging data from modalities such as MRI and CT scans into a unified clinical judgement. However, most deep learning models rely on MRI/CT images alone, failing to replicate the clinicians multimodal reasoning. We explore a two-branch multimodal network combining raw MRI scans with 91 extracted radiomic features (intensity, texture, shape, and boundary descriptors) to classify brain tumors into glioma, meningioma, pituitary, and no-tumor. A pre-trained CNN backbone encodes the image stream, whereas a dedicated MLP encodes the radiomic stream. Both streams are fused via concatenation, gated, or bidirectional cross-modal attention strategies. Across nine experimental runs on a balanced 7,200 image dataset, all multimodal configurations outperform unimodal baselines with gated fusion achieving the best accuracy of 96.13%.
Wajih ul Islam, Muhammad Yaqoob, Javed Ali Khan +1
School of Physics, Engineering and Computer Science, University of Hertfordshire, UK