VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers

Marvin Seyfarth; Salman Ul Hassan Dar; Yannik Frisch; Philipp Wild; Norbert Frey; Florian André; Sandy Engelhardt

cs.CVMar 26, 2026

VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers

Authors: Marvin Seyfarth, Salman Ul Hassan Dar, Yannik Frisch, Philipp Wild, Norbert Frey, Florian André, Sandy Engelhardt

Organizations: Institute for Artificial Intelligence in Cardiovascular Medicine, Medical Faculty of Heidelberg University, Heidelberg University, Heidelberg, Germany · Department of Cardiology, Angiology, Pneumology, Heidelberg University Hospital, Heidelberg, Germany · DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany · University Medical Center of the Johannes Gutenberg-University Mainz, Mainz, Germany

Abstract

Diffusion models have become a leading approach for high-fidelity medical image synthesis. However, most existing methods for 3D medical image generation rely on convolutional U-Net backbones within latent diffusion frameworks. While effective, these architectures impose strong locality biases and limited receptive fields, which may constrain scalability, global context integration, and flexible conditioning. In this work, we introduce VolDiT, the first purely transformer-based 3D Diffusion Transformer for volumetric medical image synthesis. Our approach extends diffusion transformers to native 3D data through volumetric patch embeddings and global self-attention operating directly over 3D tokens. To enable structured control, we propose a timestep-gated control adapter that maps segmentation masks into learnable control tokens that modulate transformer layers during denoising. This token-level conditioning mechanism allows precise spatial guidance while preserving the modeling advantages of transformer architectures. We evaluate our model on high-resolution 3D medical image synthesis tasks and compare it to state-of-the-art 3D latent diffusion models based on U-Nets. Results demonstrate improved global coherence, superior generative fidelity, and enhanced controllability. Our findings suggest that fully transformerbased diffusion models provide a flexible foundation for volumetric medical image synthesis. The code and models trained on public data are available at https://github.com/Cardio-AI/voldit.

Figures & tables

Figure 1: Overview of the VolDiT framework. Input volumes are encoded into latents, patchified into tokens, and processed by the 3D Diffusion Transformer for denoising, with spatial conditioning injected through the timestep-gated control adapter. The transformer outputs are then unpatchified and decoded to reconstruct volumetric images.

Figure 2: Synthetic examples on LUNA16 and TaviCT. HA-GAN produces anatomically implausible structures, while the U-Net LDM yields high-quality images but struggles with global anatomical consistency (red circles). VolDiT generates anatomically coherent volumes with sharper structural detail, reflecting the benefit of global self-attention.

Dataset	Model	$\text{FID}_{\text{test}}$ $(\downarrow)$	P $(\uparrow)$	R $(\uparrow)$	D $(\uparrow)$	C $(\uparrow)$	MS-SSIM $(\downarrow)$
LUNA16	Real train	0.005	0.96	0.96	0.96	1.0	0.31 $\pm$ 0.08
	U-Net LDM [ 23 ]	0.031	0.44	0.94	0.18	0.76	0.36 $\pm$ 0.09
	HA-GAN [ 24 ]	0.126	1.00	0.00	0.32	0.06	0.99 $\pm$ 0.00
	VolDiT (L, p=4)	0.004	0.91	0.90	0.82	0.92	0.38 $\pm$ 0.08
TaviCT	Real train	3.1	0.95	0.95	1.07	0.90	0.30 $\pm$ 0.08
	U-Net LDM [ 23 ]	36.8	0.95	0.63	0.89	0.68	0.38 $\pm$ 0.07

Table 1: Generative metrics as measured by FID, precision (P), recall (R), density (D), coverage (C), and MS-SSIM.

Dataset	Model	Parameter Count	p=2	p=4
LUNA16	VolDiT-XS	17.2M	0.009	0.015
	VolDiT-S	33.2M	0.009	0.023
	VolDiT-B	131.0M	0.014	0.017
	VolDiT-L	580.0M	–	0.004
TaviCT	VolDiT-XS	17.2M	22.0	134.6
	VolDiT-S	33.2M	23.6	131.2

Table 2: Effect of model size and patch size (FID

{}_{\text{test}}

), lower is better)

	Heart		Aorta		Gen. Metrics
Model	Dice $\uparrow$	HD 95 $\downarrow$	Dice $\uparrow$	HD 95 $\downarrow$	FID ${}_{\text{test}}$ $\downarrow$	MS-SSIM $\downarrow$
U-Net LDM	0.89 (0.03)	5.39 (2.0)	0.83 (0.10)	4.69 (2.0)	25.8	0.34 $\pm$ 0.09
VolDiT ( $\pi_{0.1}$ )	0.92 (0.04)	4.12 (6.7)	0.90 (0.10)	2.83 (4.1)	25.1	0.36 $\pm$ 0.07
VolDiT ( $\pi_{1.0}$ )	0.94 (0.02)	2.83 (0.8)	0.92 (0.02)	2.00 (0.5)	24.4	0.33 $\pm$ 0.08
VolDiT ( $\pi_{\theta}$ )	0.94 (0.02)	2.83 (0.9)	0.92 (0.02)	2.00 (0.3)	22.7	0.34 $\pm$ 0.08

Table 3: Conditional generation performance on TaviCT using held-out test masks. Median (IQR) reported for Dice and HD 95 . Lower FID and MS-SSIM indicate better fidelity and higher diversity, respectively.

Figure 3: Conditionally generated TaviCT samples based on held-out test masks of heart structures (blue) and the aorta (green). The U-Net-based model shows less alignment with input masks and does not preserve anatomical realism when enforcing the condition (highlighted by red squares).

Medical Image Synthesis

Medical image synthesis creates, translates, reconstructs, and edits clinical images to fill missing data and support analysis.

See all 310 papers

Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation
Jun 18, 2026Zhenkai Zhang, Markus Hiller, Krista A. Ehinger +1Diffusion Transformer 3D Medical Imaging
School of Computing and Information Systems, The University of Melbourne · School of Computing
3D Diffusion Models for Medical Imaging
Oct 19, 2026Frank-Peter Schilling, Daniel Barco, Philipp Denzel +13D Diffusion Models Medical Image Synthesis
Centre for AI (CAI), Zurich University of Applied Sciences (ZHAW), Technikumstrasse 9, Winterthur, 8400, Zurich, Switzerland
VS-DDPM: Efficient Low-Cost Diffusion Model for Medical Modality Translation
Apr 24, 2026Nikoo Moradi, Gijs Luijten, Behrus Hinrichs-Puladi +4Image-to-Image Translation 3D Medical Imaging
Institute for Artificial Intelligence in Medicine (IKIM), Essen University Hospital (AöR), University of Duisburg-Essen, Essen, Germany · Institute of Computer Graphics and Vision, Graz University of Technology, Inffeldgasse 16, Graz, 8010, Austria · Center for Virtual and Extended Reality in Medicine (ZvRM), University Hospital Essen (AöR), Essen, Germany +7

Jun 18, 2026J/K move · Enter open · S save

Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation

Zhenkai Zhang, Markus Hiller, Krista A. Ehinger, Tom Drummond

School of Computing and Information Systems, The University of Melbourne · School of Computing

Diffusion Transformer 3D Medical Imaging Medical Image Synthesis Computed Tomography

Open paper AskPDF arXiv

Generating high-resolution 3D CT volumes with fine details remains challenging due to substantial computational demands and optimization difficulties inherent to existing generative models. In this paper, we propose the Pixel-Level Residual Diffusion Transformer (PRDiT), a scalable generative framework that synthesizes high-quality 3D medical volumes directly at voxel-level. PRDiT introduces a two-stage training architecture comprising 1) a local denoiser in the form of an MLP-based blind estimator operating on overlapping 3D patches to separate low-frequency structures efficiently, and 2) a global residual diffusion transformer employing memory-efficient attention to model and refine high-frequency residuals across entire volumes. This coarse-to-fine modeling strategy simplifies optimization, enhances training stability, and effectively preserves subtle structures without the limitations of an autoencoder bottleneck. Extensive experiments conducted on the LIDC-IDRI and RAD-ChestCT datasets demonstrate that PRDiT consistently outperforms state-of-the-art models, such as HA-GAN, 3D LDM and WDM-3D, achieving significantly lower 3D FID, MMD and Wasserstein distance scores.

Abstract

Medical Image Synthesis

Explore similar work

Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation

3D Diffusion Models for Medical Imaging

VS-DDPM: Efficient Low-Cost Diffusion Model for Medical Modality Translation