Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generation

Delin An; Chaoli Wang

cs.CVMar 23, 2026

Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generation

Authors: Delin An, Chaoli Wang

Organizations: University of Notre Dame

Abstract

Diffusion probabilistic models have demonstrated significant potential in generating high-quality, realistic medical images, providing a promising solution to the persistent challenge of data scarcity in the medical field. Nevertheless, producing 3D medical volumes with anatomically consistent structures under multimodal conditions remains a complex and unresolved problem. We introduce Sketch2CT, a multimodal diffusion framework for structure-aware 3D medical volume generation, jointly guided by a user-provided 2D sketch and a textual description that captures 3D geometric semantics. The framework initially generates 3D segmentation masks of the target organ from random noise, conditioned on both modalities. To effectively align and fuse these inputs, we propose two key modules that refine sketch features with localized textual cues and integrate global sketch-text representations. Built upon a capsule-attention backbone, these modules leverage the complementary strengths of sketches and text to produce anatomically accurate organ shapes. The synthesized segmentation masks subsequently guide a latent diffusion model for 3D CT volume synthesis, enabling realistic reconstruction of organ appearances that are consistent with user-defined sketches and descriptions. Extensive experiments on public CT datasets demonstrate that Sketch2CT achieves superior performance in generating multimodal medical volumes. Its controllable, low-cost generation pipeline enables principled, efficient augmentation of medical datasets. Code is available at https://github.com/adlsn/Sketch2CT.

Figures & tables

Figure 2: Overview of our Sketch2CT framework. (a) A capsule-based sketch encoder and a sentence transformer extract structural and semantic features, which are fused via a FiLM module. (b) The fused representation conditions a segmentation latent diffusion model to generate 3D organ masks. (c) The predicted segmentation latent guides an image latent diffusion model to synthesize 3D medical volumes.

	CHAOS liver (CT)		AVT aorta (CT)		Decathlon liver (CT)		Decathlon heart (MRI)
method	FID $\downarrow$	LPIPS $\uparrow$	FID $\downarrow$	LPIPS $\uparrow$	FID $\downarrow$	LPIPS $\uparrow$	FID $\downarrow$	LPIPS $\uparrow$
Med-DDPM	114.4	0.220	119.6	0.213	115.3	0.207	128.7	0.192
MedGen3D	43.6	0.300	47.1	0.294	45.7	0.296	96.8	0.248
Seg-Diff	37.8	0.310	38.9	0.313	34.8	0.335	68.4	0.265
Sketch2CT	33.7	0.332	36.9	0.321	36.5	0.328	65.1	0.269

Table 1: Quantitative evaluation of synthetic images. We report FID (lower is better) and LPIPS (higher is better) across four datasets. Since baseline models cannot generate conditional segmentations, we provide synthesized masks as their inputs.

	CHAOS liver (CT)		AVT aorta (CT)		Decathlon liver (CT)		Decathlon heart (MRI)
model	Dice( $m^{pred}_{gen},m$ )	Dice( $m^{pred}_{gen},m^{pred}_{real}$ )	Dice( $m^{pred}_{gen},m$ )	Dice( $m^{pred}_{gen},m^{pred}_{real}$ )	Dice( $m^{pred}_{gen},m$ )	Dice( $m^{pred}_{gen},m^{pred}_{real}$ )	Dice( $m^{pred}_{gen},m$ )	Dice( $m^{pred}_{gen},m^{pred}_{real}$ )
Med-DDPM	0.501	0.492	0.487	0.474	0.512	0.495	0.421	0.408
MedGen3D	0.814	0.797	0.842	0.828	0.821	0.803	0.612	0.587
Seg-Diff	0.827	0.809	0.866	0.861	0.892	0.873	0.638	0.601
Sketch2CT	0.868	0.852	0.894	0.887	0.912	0.904	0.642	0.614

Table 2: Faithfulness of generated images to input masks. Dice is computed between segmentations predicted from generated images and the input masks (i.e., Dice(

m^{pred}_{gen},m

)), and between segmentations predicted from generated images and segmentations predicted from real images (i.e., Dice(

m^{pred}_{gen},m^{pred}_{real}

)). Sketch2CT consistently achieves the best fidelity across all datasets.

	CHAOS liver	AVT aorta	Decathlon liver	Decathlon heart
	(CT)	(CT)	(CT)	(MRI)
real training set	0.897	0.904	0.912	0.823
Med-DDPM	0.574	0.561	0.598	0.412
MedGen3D	0.814	0.801	0.823	0.681
Seg-Diff	0.826	0.812	0.887	0.702
Sketch2CT	0.893	0.889	0.904	0.711

Table 3: Downstream segmentation performance (Dice) on real datasets. A segmentation model trained on synthetic images generated by Sketch2CT achieves the closest performance to the model trained on real data, demonstrating superior anatomical realism.

Figure 3: Qualitative comparison of baseline methods. For each case, we extract a sketch and text description from the ground-truth mask to encode organ geometry. These features guide the generation of 3D masks and the synthesis of 3D volumes.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Examples of textual geometry descriptions used as the text-based conditioning input in Sketch2CT. For each organ, we show the input sketch, the generated mask, and the corresponding structured text description that captures the global shape, surface characteristics, symmetry, and topology.

Figure A2 : Effect of sketch granularity on segmentation mask generation. For each dataset, we vary the sketch detail in three levels, coarse, medium, and fine, and visualize the corresponding 3D masks produced by Sketch2CT. Increased sketch detail introduces more local structural variations, while the overall anatomical geometry remains consistent across granularity levels.

	CHAOS liver	AVT aorta	Decathlon liver	Decathlon heart
	(CT)	(CT)	(CT)	(MRI)
full model	0.893	0.889	0.904	0.711
w/o TSFE	0.864	0.859	0.872	0.683
w/o CGFM	0.825	0.818	0.831	0.671
w/o Seg-LDM	0.642	0.629	0.633	0.545

Appendix

Table A1 : Ablation study of Sketch2CT. Removing TSFE, CGFM, or the segmentation latent diffusion model (Seg-LDM) reduces downstream segmentation performance, confirming that all components are essential for generating anatomically coherent and text-aligned 3D masks.

Figure A3 : Comparison of segmentation masks generated using sketch-only and text-only conditions. Sketch-only guidance fails to recover full 3D geometry due to single-view ambiguity. In contrast, text-only guidance yields incorrect global shape and spatial placement, underscoring the need to combine sketches and text.

Figure A4 : Qualitative diversity demonstration using a single Decathlon liver case. Under identical sketch and text conditions, three independent runs (r1-r3) generate anatomically consistent yet appearance-varying CT volumes, illustrating the stochastic diversity of Sketch2CT.

Medical Image Synthesis

Medical image synthesis creates, translates, reconstructs, and edits clinical images to fill missing data and support analysis.

See all 310 papers

MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts
May 31, 2026Weicheng Dai, Chenyu Wang, Binxu Li +4Diffusion Transformer Conditional Image Generation
Boston University School of Engineering, Boston, MA, USA · Stanford University · University of Pittsburgh Medical Center +1
Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation
Jun 18, 2026Zhenkai Zhang, Markus Hiller, Krista A. Ehinger +1Diffusion Transformer 3D Medical Imaging
School of Computing and Information Systems, The University of Melbourne · School of Computing
Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation
Mar 9, 2026Daniele Molino, Camillo Maria Caruso, Paolo Soda +1Medical Image Synthesis Computed Tomography
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy · Department of Diagnostics and Intervention, Biomedical Engineering and Radiation Physics, Umeå University, Umeå, Sweden

May 31, 2026J/K move · Enter open · S save

MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts

Weicheng Dai, Chenyu Wang, Binxu Li, Shantanu Ghosh+3

Boston University School of Engineering, Boston, MA, USA · Stanford University · University of Pittsburgh Medical Center+1

Diffusion Transformer Conditional Image Generation 3D Medical Imaging Medical Image Synthesis Controllable Image Generation Multimodal Diffusion Models

Open paper AskPDF arXiv

Generative models for volumetric medical images have found many applications in medical imaging, ranging from data augmentation to serving as priors for inverse problems. For these applications, generating high-resolution 3D images with strong controllability is essential but remains highly challenging. Existing approaches typically control generation either through radiology reports used as text prompts or through full image segmentation. While text-based prompting is flexible, it provides limited spatial control over the location, shape, and boundary of abnormalities. In contrast, segmentation-based methods receive precise spatial guidance but are restrictive in requiring full-organ annotations. In this work, we propose a flexible multimodal framework for controllable volumetric image generation that supports input from radiology reports and segmentation prompts (both optional). Our approach allows users to provide segmentation of a specific anatomy or abnormality without requiring full-organ annotations. The semantic meaning of the segmentation mask is specified through an accompanying text description, resulting in a highly flexible and scalable conditioning mechanism. We develop a memory-efficient architecture based on a modified diffusion transformer that jointly processes image and segmentation tokens. The model further incorporates gated attention to effectively attend to long radiology reports. Experiments demonstrate that our method achieves state-of-the-art perceptual and semantic scores (e.g., 24% relative improvement in mean FID), generates high-resolution anatomically consistent CT volumes, and improves data efficiency when used for data augmentation. Radiologists' evaluation further confirms strong alignment between generated and real medical images.

Abstract

Medical Image Synthesis

Explore similar work

MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts

Pixel-Level Residual Diffusion Transformer: Scalable 3D CT Volume Generation

Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation