Towards Personalized Multi-Modal MRI Synthesis across Heterogeneous Datasets
Authors: Yue Zhang, Zhizheng Zhuo, Siyao Xu, Shan Lv, Zhaoxi Liu, Jun Qiu, Qiuli Wang, Yaou Liu, +1 more
Organizations: School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, 215123, China · National Key Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China, Chengdu, 610097, China · Department of Radiology, Beijing Tiantan Hospital, Capital Medical University, Beijing, 100070, China · Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230036, China · Department of Radiology, Southwest Hospital, Army Medical University, Chongqing, 400038, China · China National Clinical Research Center for Neurological Diseases, Beijing, 100070, China · State Key Laboratory of Precision and Intelligent Chemistry, University of Science and Technology of China, Hefei, Anhui, 230026, China · Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, USTC, Suzhou Jiangsu, 215123, China
Synthesizing missing modalities in multi-modal magnetic resonance imaging (MRI) is vital for ensuring diagnostic completeness, particularly when full acquisitions are infeasible due to time constraints, motion artifacts, and patient tolerance. Recent unified synthesis models have enabled flexible synthesis tasks by accommodating various input-output configurations. However, their training and evaluation are typically restricted to a single dataset, limiting their generalizability across diverse clinical datasets and impeding practical deployment. To address this limitation, we propose PMM-Synth, a personalized MRI synthesis framework that not only supports various synthesis tasks but also generalizes effectively across heterogeneous datasets. PMM-Synth is jointly trained on multiple multi-modal MRI datasets that differ in modality coverage, disease types, and intensity distributions. It achieves cross-dataset generalization through three core innovations: a Personalized Feature Modulation module that dynamically adapts feature representations based on dataset identifier to mitigate the impact of distributional shifts; a Modality-Consistent Batch Scheduler that facilitates stable and efficient batch training under inconsistent modality conditions; and a selective supervision loss to ensure effective learning when ground truth modalities are partially missing. Evaluated on four clinical multi-modal MRI datasets, PMM-Synth consistently outperforms state-of-the-art methods in both one-to-one and many-to-one synthesis tasks, achieving superior PSNR and SSIM scores. Qualitative results further demonstrate improved preservation of anatomical structures and pathological details. Additionally, downstream tumor segmentation and radiological reporting studies suggest that PMM-Synth holds potential for supporting reliable diagnosis under real-world modality-missing scenarios.
Figures & tables
Figure 1: Overview of PMM-Synth. (a) Commonly used multi-modal brain MR imaging includes T1, T2, T1C, FLAIR, DWI, and ADC sequences. (b) Illustration of inter-dataset heterogeneity, which mainly includes modality inconsistency and distributional shifts. Modality inconsistency refers to the fact that different datasets cover different modality combinations, and cases within the same dataset may also contain varying subsets of modalities (e.g., the TTI dataset). Distributional shifts are exemplified using the FLAIR modality, where images from different datasets show clear differences in visual appearance and intensity distribution. (c) Architectural design of PMM-Synth. The framework includes three core components: the PFM to model dataset-specific distributions, the MCBS to enable efficient batch training under modality inconsistency, and the selective supervision loss to enable effective learning with partially missing ground truth. PMM-Synth improves the performance in downstream clinical tasks such as tumor segmentation and radiological reporting.
Table 1: Quantitative evaluation of one-to-one MRI synthesis across multiple datasets.
Table 2: Quantitative evaluation of many-to-one MRI synthesis across multiple datasets.
Figure 2: Comprehensive evaluation and ablation analysis of PMM-Synth across heterogeneous multi-modal MRI datasets.
Figure 3: Examples of cases with real and synthesized sequences. ‘3-to-3’ refers to synthesizing T1C, DWI, and ADC from T1, T2, and FLAIR; ‘2-to-4’ uses T1 and T2 to generate the remaining four modalities; ‘1-to-5’ uses only T1 to synthesize the other five modalities. The synthesized modalities are highlighted with yellow dashed boxes.
Table 3: Examples of radiological reports written by two radiologists. Each report consists of the Findings and Impression . The differences between each report and the ground truth are marked in red . □□□ indicates missing findings/impression compared to the ground truth.
Table 4: Radiological reporting analysis on the TTG dataset.
Figure S1 : The synthesis backbone adopted in this work is extended from our previous study. It comprises one common encoding stream, six modality-specific encoding streams, six decoding streams, and six discriminators, each specifically designed to process a corresponding imaging modality. A DFUM module is employed to integrate feature representations from a variable number of input modalities.
Multimodal MRI provides complementary information for neuroimaging analysis, where different imaging modalities capture distinct anatomical, tissue, and pathological features that support the development and evaluation of downstream AI applications. Although large-scale structural MRI resources are increasingly available, their modality coverage is often uneven across public and pooled neuroimaging datasets. This uneven modality coverage is further complicated by heterogeneity across sites, scanners, and acquisition protocols, as well as demographic and clinical variables that are often sparse, inconsistently recorded, or unavailable across studies. Synthetic MRI generation can help address this imbalance by synthesizing target-modality volumes for dataset augmentation and controlled synthetic cohort creation. However, many existing MRI synthesis approaches are trained on narrow modality sets or relatively homogeneous cohorts, limiting their applicability to large pooled neuroimaging resources where modality availability, acquisition protocols, and metadata coverage vary substantially across datasets. Diffusion models have become an attractive approach for MRI synthesis because of their strong sample fidelity and diversity, but sampling directly in 3D voxel space is computationally expensive and slow at inference. Latent diffusion improves practicality by synthesizing MRI in a learned, 3D latent space, although generation quality depends on the autoencoder's reconstruction fidelity and the resulting latent distribution. Our approach combines a Wavelet-Fusion variational autoencoder (WF-VAE) latent compressor with a conditional 3D U-Net diffusion model trained in the learned latent space using explicit modality and metadata conditioning. Our proposed Wavelet-Fusion Diffusion Model (WFDM) achieved the strongest distributional alignment among the evaluated synthetic MRI generators.
Muhammad Nabi Yasinzai, Remika Mito, Mangor Pedersen
Department of Psychology & Neuroscience, Auckland University of Technology, Auckland, New Zealand. · Department of Psychiatry, University of Melbourne, Melbourne, Victoria 3053, Australia. · Auckland University of Technology, Auckland, New Zealand.
Missing data problems, such as missing modalities in multi-modal brain MRI and missing slices in cardiac MRI, pose significant challenges in clinical practice. Existing methods rely on external guidance to supply detailed missing-state information for instructing generative models to synthesize missing MRIs. However, manual indicators are not always available or reliable in real-world scenarios due to the unpredictable nature of clinical environments. Moreover, these explicit masks are not informative enough to provide guidance for improving semantic consistency. In this work, we argue that generative models should infer and recognize missing states in a self-perceptive manner, enabling them to better capture subtle anatomical and pathological variations. Towards this goal, we propose CoPeDiT, a shared completeness-perception framework for 3D MRI synthesis, following a common conditioning strategy with task-specific instantiations for different missing-data scenarios. Specifically, we incorporate dedicated pretext tasks into our tokenizer, CoPeVAE, empowering it to learn completeness-aware discriminative prompt tokens, and design MDiT3D, a specialized diffusion transformer architecture for 3D MRI synthesis that effectively uses the completeness-aware prompt tokens as guidance to enhance semantic consistency in 3D space. Comprehensive evaluations on three large-scale MRI datasets demonstrate that CoPeDiT consistently improves upon state-of-the-art methods across diverse missing patterns, yielding high-fidelity and structurally consistent MRI synthesis. Our code is available at https://github.com/JK-Liu7/CoPeDiT.
Junkai Liu, Nay Aung, Theodoros N. Arvanitis +3
School of Engineering, University of Birmingham, UK · William Harvey Research Institute, Queen Mary University London, UK · Barts Heart Centre, St Bartholomew’s Hospital, Barts Health NHS Trust, UK +1
Accurate synthesis of computed tomography (CT) images from magnetic resonance imaging (MRI) is clinically valuable for cranial applications such as attenuation correction, radiotherapy planning, and image-guided interventions. However, heterogeneity across MRI field strengths and acquisition protocols limits the generalizability of existing methods. In this study, we formulate cranial CT synthesis as a modular, structurally coupled problem and propose a deep learning framework to improve robustness across heterogeneous MRI conditions. The model is designed to adapt to variations in field strength and imaging protocols while preserving anatomical consistency. Experiments on multi-site datasets demonstrate improved performance and generalization compared with conventional approaches. The proposed method enables reliable CT synthesis across heterogeneous MRI settings, supporting broader clinical translation.
Zhuoyao Xin, Yiren Zhang, Christopher Wu +6
F.M. Kirby Research Center for Brain Imaging, Kennedy Krieger Institute, Baltimore, Maryland, 21205, United States. · Neurosection, Division of MR Research, Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, Baltimore, Maryland, 21287, United States. · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Maryland, 21218, United States. +6