VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers

Marvin Seyfarth; Salman Ul Hassan Dar; Yannik Frisch; Philipp Wild; Norbert Frey; Florian André; Sandy Engelhardt

cs.CVMar 26, 2026

VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers

Authors: Marvin Seyfarth, Salman Ul Hassan Dar, Yannik Frisch, Philipp Wild, Norbert Frey, Florian André, Sandy Engelhardt

Organizations: Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University, Heidelberg, Germany · Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital, Heidelberg, Germany · DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany · University Medical Center of the Johannes Gutenberg-University Mainz, Mainz, Germany

Abstract

Diffusion models have become a leading approach for high-fidelity medical image synthesis. However, most existing methods for 3D medical image generation rely on convolutional U-Net backbones within latent diffusion frameworks. While effective, these architectures impose strong locality biases and limited receptive fields, which may constrain scalability, global context integration, and flexible conditioning. In this work, we introduce VolDiT, the first purely transformer-based 3D Diffusion Transformer for volumetric medical image synthesis. Our approach extends diffusion transformers to native 3D data through volumetric patch embeddings and global self-attention operating directly over 3D tokens. To enable structured control, we propose a timestep-gated control adapter that maps segmentation masks into learnable control tokens that modulate transformer layers during denoising. This token-level conditioning mechanism allows precise spatial guidance while preserving the modeling advantages of transformer architectures. We evaluate our model on high-resolution 3D medical image synthesis tasks and compare it to state-of-the-art 3D latent diffusion models based on U-Nets. Results demonstrate improved global coherence, superior generative fidelity, and enhanced controllability. Our findings suggest that fully transformerbased diffusion models provide a flexible foundation for volumetric medical image synthesis. The code and models trained on public data are available at https://github.com/Cardio-AI/voldit.

Figures & tables

Figure 1: Overview of the VolDiT framework. Input volumes are encoded into latents, patchified into tokens, and processed by the 3D Diffusion Transformer for denoising, with spatial conditioning injected through the timestep-gated control adapter. The transformer outputs are then unpatchified and decoded to reconstruct volumetric images.

Figure 2: Synthetic examples on LUNA16 and TaviCT. HA-GAN produces anatomically implausible structures, while the U-Net LDM yields high-quality images but struggles with global anatomical consistency (red circles). VolDiT generates anatomically coherent volumes with sharper structural detail, reflecting the benefit of global self-attention.

Dataset	Model	$\text{FID}_{\text{test}}$ $(\downarrow)$	P $(\uparrow)$	R $(\uparrow)$	D $(\uparrow)$	C $(\uparrow)$	MS-SSIM $(\downarrow)$
LUNA16	Real train	0.005	0.96	0.96	0.96	1.0	0.31 $\pm$ 0.08
	U-Net LDM [ 23 ]	0.031	0.44	0.94	0.18	0.76	0.36 $\pm$ 0.09
	HA-GAN [ 24 ]	0.126	1.00	0.00	0.32	0.06	0.99 $\pm$ 0.00
	VolDiT (L, p=4)	0.004	0.91	0.90	0.82	0.92	0.38 $\pm$ 0.08
TaviCT	Real train	3.1	0.95	0.95	1.07	0.90	0.30 $\pm$ 0.08
	U-Net LDM [ 23 ]	36.8	0.95	0.63	0.89	0.68	0.38 $\pm$ 0.07

Table 1: Generative metrics as measured by FID, precision (P), recall (R), density (D), coverage (C), and MS-SSIM.

Dataset	Model	Parameter Count	p=2	p=4
LUNA16	VolDiT-XS	17.2M	0.009	0.015
	VolDiT-S	33.2M	0.009	0.023
	VolDiT-B	131.0M	0.014	0.017
	VolDiT-L	580.0M	–	0.004
TaviCT	VolDiT-XS	17.2M	22.0	134.6
	VolDiT-S	33.2M	23.6	131.2

Table 2: Effect of model size and patch size (FID

{}_{\text{test}}

), lower is better)

	Heart		Aorta		Gen. Metrics
Model	Dice $\uparrow$	HD 95 $\downarrow$	Dice $\uparrow$	HD 95 $\downarrow$	FID ${}_{\text{test}}$ $\downarrow$	MS-SSIM $\downarrow$
U-Net LDM	0.89 (0.03)	5.39 (2.0)	0.83 (0.10)	4.69 (2.0)	25.8	0.34 $\pm$ 0.09
VolDiT ( $\pi_{0.1}$ )	0.92 (0.04)	4.12 (6.7)	0.90 (0.10)	2.83 (4.1)	25.1	0.36 $\pm$ 0.07
VolDiT ( $\pi_{1.0}$ )	0.94 (0.02)	2.83 (0.8)	0.92 (0.02)	2.00 (0.5)	24.4	0.33 $\pm$ 0.08
VolDiT ( $\pi_{\theta}$ )	0.94 (0.02)	2.83 (0.9)	0.92 (0.02)	2.00 (0.3)	22.7	0.34 $\pm$ 0.08

Table 3: Conditional generation performance on TaviCT using held-out test masks. Median (IQR) reported for Dice and HD 95 . Lower FID and MS-SSIM indicate better fidelity and higher diversity, respectively.

Figure 3: Conditionally generated TaviCT samples based on held-out test masks of heart structures (blue) and the aorta (green). The U-Net-based model shows less alignment with input masks and does not preserve anatomical realism when enforcing the condition (highlighted by red squares).

Medical Image Synthesis

Medical image synthesis creates, translates, reconstructs, and edits clinical images to fill missing data and support analysis.

See all 310 papers

Jun 18, 2026

Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation

Generating high-resolution 3D CT volumes with fine details remains challenging due to substantial computational demands and optimization difficulties inherent to existing generative models. In this paper, we propose the Pixel-Level Residual Diffusion Transformer (PRDiT), a scalable generative framework that synthesizes high-quality 3D medical volumes directly at voxel-level. PRDiT introduces a two-stage training architecture comprising 1) a local denoiser in the form of an MLP-based blind estimator operating on overlapping 3D patches to separate low-frequency structures efficiently, and 2) a global residual diffusion transformer employing memory-efficient attention to model and refine high-frequency residuals across entire volumes. This coarse-to-fine modeling strategy simplifies optimization, enhances training stability, and effectively preserves subtle structures without the limitations of an autoencoder bottleneck. Extensive experiments conducted on the LIDC-IDRI and RAD-ChestCT datasets demonstrate that PRDiT consistently outperforms state-of-the-art models, such as HA-GAN, 3D LDM and WDM-3D, achieving significantly lower 3D FID, MMD and Wasserstein distance scores.

Zhenkai Zhang, Markus Hiller, Krista A. Ehinger +1

School of Computing and Information Systems, The University of Melbourne · School of Computing

Diffusion Transformer 3D Medical Imaging Medical Image Synthesis Computed Tomography

Ask PDF

Oct 19, 2026

3D Diffusion Models for Medical Imaging

Generative deep learning methods such as Variational Autoencoders, Generative Adversarial Networks or Diffusion Models have achieved remarkable results in image synthesis. In particular, Diffusion Models have demonstrated superior image fidelity and sample diversity compared to other approaches on standard benchmarks, while also offering more flexible training dynamics as evidenced both in natural image domains and in medical applications. In medical imaging including Computed Tomography, Magnetic Resonance Imaging or Positron Emission Tomography, such models have various important use-cases including data augmentation, domain transfer, image reconstruction, inpainting, image augmentation and artifact correction. Further, generated images can be conditioned on patient characteristics, medical reports, segmentation masks or other images. Specific challenges arise due to the 3D volumetric nature of medical images, which translate to substantial demands in terms of memory, compute, and training data. This article provides a comparative overview of 3D Diffusion Models which have been developed for medical images, with a focus on current performance, limitations, and an outlook for future work.

Frank-Peter Schilling, Daniel Barco, Philipp Denzel +1

Centre for AI (CAI), Zurich University of Applied Sciences (ZHAW), Technikumstrasse 9, Winterthur, 8400, Zurich, Switzerland

3D Diffusion Models Medical Image Synthesis 3D Medical Imaging

Ask PDF

Apr 24, 2026

VS-DDPM: Efficient Low-Cost Diffusion Model for Medical Modality Translation

Diffusion models produce high-quality synthetic data but suffer from slow inference. We propose 3D Variable-Step Denoising Diffusion Probabilistic Model (VS-DDPM) a framework engineered to maintain generative quality while accelerating inference by several factors. We tested our approach on four tasks (missing MRI, tumor removal, MRI-to-sCT, and CBCT-to-sCT) within the BraTS2025 and SynthRAD2025 challenges. Designed for high efficiency under hardware and time constrains imposed by both challenges. VS-DDPM achieved state-of-the-art (SOTA) performance in missing MRI synthesis, yielding Dice scores of 0.80, 0.83, and 0.88 for the enhancing tumor, tumor core, and whole tumor regions, respectively, alongside a structural similarity index (SSIM) of 0.95. For MRI tumor removal, the model attained a root mean squared error (RMSE) of 0.053, a peak signal-to-noise ratio (PSNR) of 26.77, and an SSIM of 0.918. While the framework demonstrated competitive performance in MRI-to-sCT and CBCT-to-sCT tasks, it did not reach SOTA benchmarks, potentially due to sensitivities in data pre and post-processing pipelines or specific loss function configurations. These results demonstrate that VS-DDPM provides a robust and tunable solution for high-fidelity 3D medical image synthesis. The code is available in https://github.com/andre-fs-ferreira/SynthRAD_by_Faking_it.

Nikoo Moradi, Gijs Luijten, Behrus Hinrichs-Puladi +4

Institute for Artificial Intelligence in Medicine (IKIM), Essen University Hospital (AöR), University of Duisburg-Essen, Essen, Germany · Institute of Computer Graphics and Vision, Graz University of Technology, Inffeldgasse 16, Graz, 8010, Austria · Center for Virtual and Extended Reality in Medicine (ZvRM), University Hospital Essen (AöR), Essen, Germany +7

Image-to-Image Translation 3D Medical Imaging Few-Step Diffusion Sampling Medical Image Synthesis 3D Diffusion Models Diffusion Model Inference Acceleration

Ask PDF

Abstract

Medical Image Synthesis

Explore similar work

Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation

3D Diffusion Models for Medical Imaging

VS-DDPM: Efficient Low-Cost Diffusion Model for Medical Modality Translation