S2T-Unet: A Structure-to-Style Framework for Inter-Modality MRI Translation

Yichao Liu

cs.CVSep 29, 2026

S2T-Unet: A Structure-to-Style Framework for Inter-Modality MRI Translation

Authors: Yichao Liu

Organizations: Heidelberg University, IWR, Germany

Abstract

Inter-modality MRI translation aims to synthesize missing MRI modalities from available acquisitions, reducing the need for additional scanning while preserving clinically relevant anatomical information. However, existing image translation methods often learn intensity mappings without explicitly separating modality-invariant structural information from modality-specific appearance, which may lead to structural information loss or unrealistic image details. In this work, we propose S2T-Unet, a structure-to-style framework that explicitly models these two aspects. Specifically, vector quantization is introduced at the lower-level bottleneck to encode modality-invariant structural information using a learned discrete codebook. At higher levels, a modality transformation module uses decoder features to condition and transform encoder representations toward the target modality, thereby recovering modality-specific intensity and contrast information. Experiments on the IXI multi-contrast MRI dataset across four translation tasks demonstrate that S2T-Unet is comparable or outperform with state-of-art method.

Figures & tables

Figure 1 : Overview of S2T-Unet. It comprises three components: a Unet encoder–decoder, style transformation modules, and vector quantization modules. Dashed arrows denote features extracted before batch normalization and ReLU function, which are fed into the style transformation module. IN denotes that the normalization in the Conv block is instance normalization.

Model	T1 $\rightarrow$ T2			T2 $\rightarrow$ T1			T1 $\rightarrow$ PD			PD $\rightarrow$ T1
Model	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE
UNet	24.95	0.885	0.0576	21.81	0.862	0.0846	22.63	0.815	0.0752	21.63	0.813	0.085
Pix2Pix	26.50	0.899	0.0477	23.38	0.875	0.0693	25.02	0.891	0.0572	23.30	0.874	0.07
CycleGAN	25.44	0.889	0.0522	22.69	0.869	0.075	23.45	0.858	0.066	22.51	0.847	0.076
ResViT	27.37	0.917	0.0432	23.72	0.887	0.0681	25.11	0.899	0.0569	23.28	0.886	0.0719
Ours	27.48	0.916	0.0428	24.10	0.893	0.0657	25.34	0.900	0.0554	23.78	0.892	0.0677

Table 1 : Quantitative performance comparison between CNN and Transformer-based architectures for one-to-one image-to-image translation using the IXI dataset.

Panel of Figure 2 — Figure 2 : Qualitative comparison of MRI cross-modality translation in four directions: (a) T1 $\rightarrow$ T2, (b) T2 $\rightarrow$ T1, (c) T1 $\rightarrow$ PD, and (d) PD $\rightarrow$ T1. In each panel, the input (source modality) and ground truth (GT, target modality) are shown on the left, and the synthesized results of Pix2Pix, ResViT, and our method are shown on the right. The bottom row of each panel shows the corresponding difference maps between the synthesized image and the GT, where red and blue indicate positive and negative errors, respectively.

	without VQ			without MT			with both
	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE
S2T-Unet	26.10	0.900	0.0510	25.20	0.888	0.0559	27.48	0.916	0.0428

Table 2 : Ablation study for vector quantization (VQ) and modality transformation (MT).

Explore similar work

Posterior Samplings are Missing Modalities Generators for Medical Image Translation
Jul 21, 2026Jonghun KimMissing Modalities Cross-Modality
Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea
Unified MRI Brain Image Translation via Hierarchical Tumor Structure Comparison
Jun 11, 2026Yupeng Cai, Jia Wei, Jianlong ZhouBrain Tumor Segmentation Conditional Gans
South China University of Technology, Guangzhou 510640, China · UTS Data Science Institute, University of Technology Sydney, Ultimo, NSW 2007, Australia
Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching
Aug 8, 2026Daniele Molino, Alessio Zoboli, Camillo Maria Caruso +2Cross-Modality Conditional Variational Autoencoder
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy · UniCamillus – Saint Camillus International University of Health Sciences, Rome, Italy · Department of Diagnostics and Intervention, Biomedical Engineering and Radiation Physics, Umeå University, Umeå, Sweden

Jul 21, 2026, cs.CVJ/K move · Enter open · S save

Posterior Samplings are Missing Modalities Generators for Medical Image Translation

Jonghun Kim

Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Korea

Missing Modalities Cross-Modality Magnetic Resonance Imaging Posterior Sampling

Open paper AskPDF arXiv

Magnetic resonance imaging comes in various modality contrasts that provide complementary anatomical and pathological information. Complete multimodal acquisitions are often unavailable due to time and protocol constraints. This leads to real-world datasets with missing modalities, where conventional medical image translation methods are typically limited to fixed source-target settings or require retraining for each observed source-target pair. We propose a unified framework that formulates missing-modality generation as a linear inverse problem under a joint distribution and solves it via posterior sampling with a flow matching model. By learning a joint prior over the complete modality set, our method can reconstruct arbitrary missing modalities at inference time by guiding the sampling trajectory to enforce measurement consistency with observed modalities. We further mitigate inter-modality error propagation in multi-target generation by adopting a many-to-one sampling strategy. Experiments on BraTS and IXI datasets show that our method achieves the best performance over baselines across most missing-modality scenarios. In downstream tumor segmentation, synthesized images from our method result in higher segmentation performance, indicating better preservation of clinically relevant structures.

Abstract

Explore similar work

Posterior Samplings are Missing Modalities Generators for Medical Image Translation

Unified MRI Brain Image Translation via Hierarchical Tumor Structure Comparison

Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching