Brain MR Image Synthesis with 3D Multi-Contrast Self-Attention GAN

Zaid A. Abod; Furqan Aziz

eess.IVMar 31, 2026

Brain MR Image Synthesis with 3D Multi-Contrast Self-Attention GAN

Authors: Zaid A. Abod, Furqan Aziz

Organizations: School of Computing and Mathematical Sciences, University of Leicester, Leicester, United Kingdom.

Abstract

Complete and high-quality multi-modal Magnetic Resonance Imaging (MRI) is essential for accurate neuro-oncological assessment, as each contrast provides complementary anatomical and pathological information. However, acquiring all modalities (e.g., T1c, T1n, T2w, T2f) for every patient is often impractical due to prolonged scan times, cost, and patient discomfort, potentially limiting comprehensive tumour evaluation. We propose 3D-MC-SAGAN (3D Multi-Contrast Self-Attention Generative Adversarial Network), a unified 3D multi-contrast synthesis framework that generates high-fidelity missing modalities from a single T2w input while explicitly preserving tumour characteristics. The model employs a multi-scale 3D encoder--decoder generator with residual connections and a novel Memory-Bounded Hybrid Attention (MBHA) block to capture long-range dependencies efficiently, and is trained with a WGAN-GP critic and an auxiliary domain classification head to produce T2f, T1n, and T1c volumes within a unified network. To ensure anatomical and pathological fidelity, we incorporate a frozen 3D U-Net-based segmentation network that enforces a tumour-consistency constraint during training. A composite objective combining adversarial, reconstruction, perceptual, structural similarity, contrast-classification, and segmentation-guided losses further promotes both global realism and tumour-preserving structure. Extensive experiments on 3D brain MRI datasets demonstrate that 3D-MC-SAGAN achieves state-of-the-art quantitative performance and produces visually coherent, anatomically plausible contrasts with improved distributional realism. Importantly, the proposed method maintains tumour segmentation accuracy comparable to that achieved using fully acquired multi-modal inputs, highlighting its potential to reduce acquisition burden while preserving clinically meaningful information.

Figures & tables

Fig. 1: Overview of 3D-MC-SAGAN framework.

Fig. 2: Generator architecture. A 3D encoder-decoder with residual blocks, attention-gated skip connections, and mixed attention.

Fig. 3: Attention mechanisms used in the generator: (A) MBHA for high-resolution stages and (B) standard 3D self-attention for low-resolution stages.

Method	PSNR $\uparrow$			SSIM $\uparrow$			MSE $\downarrow$			2D/3D
Method	T2f	T1n	T1c	T2f	T1n	T1c	T2f	T1n	T1c	2D/3D
Pix2Pix	21.150 $\pm$ 1.240	23.775 $\pm$ 1.133	19.167 $\pm$ 0.913	0.862 $\pm$ 0.0214	0.907 $\pm$ 0.0240	0.846 $\pm$ 0.018	0.0080 $\pm$ 0.0026	0.0043 $\pm$ 0.0012	0.0123 $\pm$ 0.0025	2D
CycleGAN	20.903 $\pm$ 1.892	22.055 $\pm$ 1.268	19.435 $\pm$ 1.506	0.845 $\pm$ 0.035	0.873 $\pm$ 0.015	0.811 $\pm$ 0.025	0.0089 $\pm$ 0.0043	0.0065 $\pm$ 0.0024	0.0121 $\pm$ 0.0053	2D
3D-mADUNet	24.692 $\pm$ 3.893	20.800 $\pm$ 4.257	26.645 $\pm$ 3.938	0.913 $\pm$ 0.030	0.919 $\pm$ 0.040	0.935 $\pm$ 0.025	0.0049 $\pm$ 0.0043	0.0124 $\pm$ 0.0105	0.0033 $\pm$ 0.0039	3D
PTNet3D	21.267 $\pm$ 0.942	21.824 $\pm$ 0.691	21.746 $\pm$ 1.050	0.888 $\pm$ 0.022	0.923 $\pm$ 0.018	0.893 $\pm$ 0.021	0.0076 $\pm$ 0.0019	0.0066 $\pm$ 0.0011	0.0069 $\pm$ 0.0022	3D
Ea-GAN	24.671 $\pm$ 2.381	26.001 $\pm$ 1.812	23.175 $\pm$ 1.614	0.928 $\pm$ 0.024	0.954 $\pm$ 0.020	0.913 $\pm$ 0.021	0.0039 $\pm$ 0.0025	0.0027 $\pm$ 0.0014	0.0051 $\pm$ 0.0021	3D

TABLE I: Quantitative evaluation results (PSNR

\uparrow

, SSIM

\uparrow

, MSE

\downarrow

) for T2f, T1n, and T1c. The best results are in bold .

Method	MFD $\downarrow$ (T2f)	MFD $\downarrow$ (T1n)	MFD $\downarrow$ (T1c)
Pix2Pix	0.747908490	0.588629693	0.988509299
CycleGAN	0.376297022	0.346849312	0.571771837
3D-mADUNet	1.142354795	1.755380266	1.875490582
PTNet3D	2.796942846	0.873885344	1.608726699
Ea-GAN	0.479778619	0.530060148	0.594216597
MT-Net	1.225086295	0.464382954	1.542097224

TABLE II: MFD results (

\times 10^{3}

) for synthesised MRI modalities (T2f, T1, and T1c). Lower is better (

\downarrow

Fig. 4: Representative qualitative results for multi-contrast brain MRI synthesis. Rows correspond to target modalities; columns show the source T2w image, comparative results, the 3D-MC-SAGAN result, the real target image, and the binary whole-tumour reference.

Method	Dice Score (only Real T2w)	Dice Score (Real Data)	Dice Score (Generated Data)
Pix2Pix	0.6975	0.8085	0.6556
CycleGAN	0.6975	0.8085	0.5914
3D-mADUNet	0.7136	0.8618	0.8082
PTNet3D	0.7136	0.8618	0.5747
Ea-GAN	0.6998	0.8332	0.8041
MT-Net	0.6967	0.8068	0.7416

TABLE III: Dice score results. Higher values indicate better segmentation performance.

Method	PSNR			SSIM			MSE			Dice score
Method	T2f	T1n	T1c	T2f	T1n	T1c	T2f	T1n	T1c	Dice score
w/o MBHA	26.681 $\pm$ 1.980	27.538 $\pm$ 2.087	26.031 $\pm$ 2.356	0.940 $\pm$ 0.013	0.956 $\pm$ 0.014	0.932 $\pm$ 0.014	0.0023 $\pm$ 0.0011	0.0020 $\pm$ 0.0011	0.0029 $\pm$ 0.0019	0.8472
w/o Perc. Loss	27.069 $\pm$ 2.202	27.467 $\pm$ 2.363	26.367 $\pm$ 2.267	0.944 $\pm$ 0.014	0.960 $\pm$ 0.014	0.936 $\pm$ 0.013	0.0022 $\pm$ 0.0012	0.0021 $\pm$ 0.0013	0.0026 $\pm$ 0.0015	0.8518
w/o Seg. Loss	26.929 $\pm$ 2.344	27.449 $\pm$ 2.076	25.897 $\pm$ 2.257	0.941 $\pm$ 0.015	0.959 $\pm$ 0.013	0.932 $\pm$ 0.014	0.0023 $\pm$ 0.0013	0.0020 $\pm$ 0.0011	0.0029 $\pm$ 0.0016	0.8237
w/o All above	26.672 $\pm$ 2.436	26.951 $\pm$ 2.291	25.879 $\pm$ 2.596	0.931 $\pm$ 0.020	0.948 $\pm$ 0.017	0.924 $\pm$ 0.014	0.0025 $\pm$ 0.0020	0.0023 $\pm$ 0.0015	0.0031 $\pm$ 0.0020	0.8121
3D-MC-SAGAN	27.680 $\pm$ 2.104	28.428 $\pm$ 2.218	26.738 $\pm$ 2.573	0.947 $\pm$ 0.014	0.966 $\pm$ 0.013	0.943 $\pm$ 0.015	0.0019 $\pm$ 0.0011	0.0016 $\pm$ 0.0009	0.0025 $\pm$ 0.0017	0.8631

TABLE IV: Ablation study results. PSNR, SSIM, and MSE are reported for T2f, T1n, and T1c. Dice score reflects downstream tumour segmentation performance. Best results are in bold .

Explore similar work

Jun 7, 2026

Segmentation-Assisted Brain MRI Synthesis with Cross-Image Multi-Contrast Feature Memory Bank Retrieval Augmentation

Multi-contrast brain MRI provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-contrast images. While current approaches excel in image synthesis, they often struggle to synthesize critical tumor regions and exploit contextual information in multi-contrast brain MRI effectively. To address this issue, we propose a synthesis-centric, segmentation-assisted closed-loop framework with retrieval augmentation synthesis. Our method overall takes a generative adversarial architecture, which aims to synthesize missing contrasts from any combination of available ones with a single model. To explicitly capture tumor semantics and focus synthesis on tumor regions, we add an auxiliary segmentation branch that predicts tumor masks and feeds them back as semantic conditioning in synthesis branch, thereby learning tumor-aware representations in the model and improving synthesis fidelity. Furthermore, we propose a dual-bank retrieval augmentation strategy. It dynamically queries two external knowledge bases, namely a tumor masks memory bank for crucial tumor context and cross-image contrast feature memory bank for global style information, to augment synthesis. Verified on two public multi-contrast magnetic resonance brain datasets: BraTs2020 and UCSF-BMSR, the proposed method is effective in handling medical brain images synthesis tasks and shows superior performance compared to previous methods. Code is available at:https://github.com/iBizzard/SSCF.git

Wenwei Huang, Jia Wei, Jianlong Zhou

South China University of Technology, Guangzhou 510640, China · UTS Data Science Institute, University of Technology Sydney, Ultimo, NSW 2007, Australia

Image Generation Medical Image Synthesis Generative Adversarial Networks

Ask PDF

Mar 19, 2026

TuLaBM: Tumor-Biased Latent Bridge Matching for Contrast-Enhanced MRI Synthesis

Contrast-enhanced magnetic resonance imaging (CE-MRI) plays a crucial role in brain tumor assessment; however, its acquisition requires gadolinium-based contrast agents (GBCAs), which increase costs and raise safety concerns. Consequently, synthesizing CE-MRI from non-contrast MRI (NC-MRI) has emerged as a promising alternative. Early Generative Adversarial Network (GAN)-based approaches suffered from instability and mode collapse, while diffusion models, despite impressive synthesis quality, remain computationally expensive and often fail to faithfully reproduce critical tumor contrast patterns. To address these limitations, we propose Tumor-Biased Latent Bridge Matching (TuLaBM), which formulates NC-to-CE MRI translation as Brownian bridge transport between source and target distributions in a learned latent space, enabling efficient training and inference. To enhance tumor-region fidelity, we introduce a Tumor-Biased Attention Mechanism (TuBAM) that amplifies tumor-relevant latent features during bridge evolution, along with a boundary-aware loss that constrains tumor interfaces to improve margin sharpness. While bridge matching has been explored for medical image translation in pixel space, our latent formulation substantially reduces computational cost and inference time. Experiments on BraTS2023-GLI (BraSyn) and Cleveland Clinic (in-house) liver MRI dataset show that TuLaBM consistently outperforms state-of-the-art baselines on both whole-image and tumor-region metrics, generalizes effectively to unseen liver MRI data in zero-shot and fine-tuned settings, and achieves inference times under 0.097 seconds per image.

Atharva Rege, Adinath Madhavrao Dukre, Numan Balci +2

MBZUAI, Abu Dhabi, UAE · National Institute of Technology Karnataka, Surathkal, India · Cleveland Clinic Abu Dhabi, UAE +1

Medical Image Synthesis Magnetic Resonance Imaging Neuroimaging

Ask PDF

Jul 8, 2026

AA-ViT: Anatomically Aware Vision Transformer with Structural and Frequency Guidance for Contrast Enhanced Brain MRI Synthesis

Accurate tumour localization and diagnosis is a critical component of clinical care for brain cancers. Magnetic Resonance Imaging (MRI) is the most commonly used imaging modality due to its superior soft-tissue contrast. However, standard MRI often exhibits limited contrast and imaging artifacts, which necessitates the use of contrast agents to enhance lesion visibility. The administration of chemical contrast agents is not always feasible and may be contraindicated in patients with renal impairment or other health conditions. As a result, developing accurate and non-invasive contrast enhanced MRI (CEMRI) synthesis methods has clinical importance. In recent years, numerous approaches for CEMRI synthesis have been proposed, predominantly relying on generative artificial intelligence models. While these methods demonstrate promising performance, their dependence on implicit feature learning often limits their ability to preserve anatomical boundaries and tumour-specific fine structures. To address these challenges, we propose an anatomically aware frequency-and-structure-guided vision transformer (AA-ViT), for CEMRI synthesis using pre-contrast MRI modalities (T1, T2, and FLAIR). Experiments on the BraTS 2021 dataset demonstrate that the proposed method preserves anatomical and lesion boundaries, achieving higher PSNR and SSIM than state-of-the-art approaches. Clinical evaluation by three neuroradiologists and a neurosurgeon on 19 randomly selected cases across diverse gliomas yielded a mean score of 3.94/5, providing preliminary clinical validation rarely seen in prior studies. Synthetic post-contrast scans from our model could lower scanning costs, shorten imaging time, and avoid the potential risks of using gadolinium-based contrast agents.

Talha Meraj, Tom Flannery, Charlie Cummins +6

Department of Computing and Electronic Engineering, Atlantic Technological University, Sligo, Ireland · Mathematical Modelling and Intelligent Systems for Health and Environment (MISHE), Faculty of Engineering and Design, Atlantic Technological University, Sligo, Ireland · JANUS Research Centre, Atlantic Technological University, Letterkenny, Ireland +5

Vision Transformer Medical Image Synthesis Brain MRI

Ask PDF