Bridging MRI and PET physiology: Untangling complementarity through orthogonal representations
Authors: Sonja Adomeit, Kartikay Tehlan, Lukas Förner, Katharina Weisser, Helen Scholtiseek, David Kaufmann, Julie Steinestel, Constantin Lapa, +2 more
Organizations: Dept. of diagnostic and interventional Radiology and Neuroradiology, University Hospital Augsburg, Germany · Digital Medicine, University Hospital Augsburg, Germany · Chair for Computer Aided Medical Procedures and Augmented Reality, Technical University of Munich, Germany · Bavarian Center for Cancer Research (BZKF) Augsburg, Germany · Dept. of Nuclear Medicine, University Hospital Augsburg, Germany · Dept. of Urology, University Hospital Augsburg, Germany · Center for Advanced Analytics and Predictive Sciences, University of Augsburg, Germany
Multimodal imaging analysis often relies on joint latent representations, yet these approaches rarely define what information is shared versus modality-specific. Clarifying this distinction is clinically relevant, as it delineates the irreducible contribution of each modality and informs rational acquisition strategies. We propose a subspace decomposition framework that reframes multimodal fusion as a problem of orthogonal subspace separation rather than translation. We decompose Prostate-Specific Membrane Antigen (PSMA) PET uptake into an MRI-explainable physiological envelope and an orthogonal residual reflecting signal components not expressible within the MRI feature manifold. Using multiparametric MRI, we train an intensity-based, non-spatial implicit neural representation (INR) to map MRI feature vectors to PET uptake. We introduce a projection-based regularization using singular value decomposition to penalize residual components lying within the span of the MRI feature manifold. This enforces mathematical orthogonality between tissue-level physiological properties (structure, diffusion, perfusion) and intracellular PSMA expression. Tested on 13 prostate cancer patients, the model demonstrates that residual components spanned by MRI features are absorbed into the learned envelope, while the orthogonal residual is largest in tumour regions. This indicates that PSMA PET contains signal components not recoverable from MRI-derived physiological descriptors. The resulting decomposition provides a structured characterization of modality complementarity grounded in representation geometry rather than image translation.
Figures & tables
Figure 1: Graphical abstract. Seven MRI sequences are used to synthesize a PSMA PET image using an INR; synthesis here is a means to isolate the orthogonal contribution of PSMA PET with respect to MRI. The network is trained with a loss that penalizes the parallel residual r∥ more heavily than the orthogonal residual r⊥ , where the total residual e=r∥+r⊥ is decomposed via SVD projection onto the subspace Π spanned by the MRI feature data.
Figure 2: Visualization of a patient case featuring: (a) the squared parallel residual; (b) the squared orthogonal residual; (c) the reconstructed SUV slice and (d) the ground truth SUV slice. Prostate masks are delineated white, while tumour ones in cyan (panels (b), (c) and (d)), and in pink (panel (b)).
Figure 3: Comparison of orthogonal and parallel error metrics, stratified by tissue type: intra-tumoral regions, non-cancerous prostate tissue, and remaining pelvic anatomy. n state remaining data points after outlier removal for visual clarity.
Tissue
Total MSE ± SD
MSE ∥± SD
MSE ⊥± SD
Tumour
0.9723±6.7172
6.1e−4±1.4e−3
0.9717±6.7171
Prostate
0.0875±0.4364
1.3e−4±4.0e−4
0.0874±0.4364
Surrounding
0.0942±0.8127
1.6e−5±1.1e−4
0.0941±0.8127
Table 1: Quantitative results for the full model for different tissues: differences are statistically significant ( p≪0.05 ).
Ablation Set
Total MSE ± SD
MSE ∥± SD
MSE ⊥± SD
Full Model
0.0967±0.8275
1.9e−5±1.2e−4
0.0967±0.8275
Leave-one-out
Minus T1
0.1323±0.9724
1.9e−5±1.4e−4
0.1323±0.9723
Minus T2
0.1410±0.9674
3.9e−5±3.6e−4
0.1410±0.9674
Minus ADC
0.1369±1.0131
3.5e−5±4.3e−4
0.1369±1.0131
Minus TTP
0.1228±0.9017
2.1e−5±1.1e−4
0.1228±0.9017
Table 2: Ablation study: Error metrics represent mean ± standard deviation across the cohort. MSE∥ and MSE⊥ denote parallel and orthogonal error components. Evaluated differences are significant ( p≪0.05 ) between the full model and both leave-one-out (minus vp ) and group ablation (no tofts).
Nuffield Department of Medicine, University of Oxford, UK · Department of Oncology, University of Oxford, UK · Big Data Institute, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom
A. A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital · Department of Radiology, Harvard Medical School · Computer Science and AI Lab, Massachusetts Institute of Technology