The scarcity of diverse, high-quality medical images remains a significant bottleneck in advancing AI-driven medical diagnostics and treatment planning. Traditional datasets often suffer from class imbalances, limited disease coverage, and insufficient variations in imaging conditions - challenges that hinder the development of robust, generalizable models. To address this, we introduce MedNet, a large-scale, multi-modal text-to-image dataset spanning 61 disease categories with 160 classes, covering both humans and animals. We built this with images from trusted public repositories, ensuring data diversity, quality, and ethical compliance. Additionally, we propose U2-ViT, a novel diffusion-based text-to-image generation architecture arranging Vision Transformers (ViTs) in a nested U-Net structure to enhance the generation of diverse, and high-fidelity medical images. Through extensive experiments, we demonstrate that our model achieves the best Fr'echet Inception Distance (FID) score compared to state-of-the-art models. The source code is available at https://github.com/LakshmikarPolamreddy/MedNet-U2-ViT.
Figures & tables
Fig. 1: U2 -ViT generated images of 256 × 256 resolution using MedNet.
Fig. 2: Overview of the MedNet dataset construction pipeline. Images are collected from multiple public repositories and specialized medical datasets, followed by data cleaning, format standardization, and organization into human and animal disease domains across multiple imaging modalities.
Fig. 3: Overview of the proposed U2 -ViT diffusion architecture for text-guided medical image generation.
Fig. 4: Qualitative comparison of medical images generated by different text-to-image models on the MedNet dataset. These results demonstrate the ability of U2 -ViT to better capture fine-grained medical details and produce high-fidelity images.
Model
Params(M)
Year
Venue
FID
Precision
Recall
DDPM [ 10 ]
-
2020
NeurIPS
52.33
0.71
0.48
LDMs [ 26 ]
400
2022
CVPR
45.60
0.74
0.48
Dreambooth [ 27 ]
-
2023
CVPR
48.15
0.68
0.49
LCMs [ 18 ]
-
2023
-
56.24
0.64
0.44
U-ViT [ 2 ]
58
2023
CVPR
33.70
0.76
0.51
DiM [ 30 ]
380
2024
-
36.42
0.75
0.49
TABLE I: Quantitative comparison of U2 -ViT with state-of-the-art text-to-image generation models on the MedNet dataset.
U-ViT Configuration
Params (M)
FID
2 × E, B, 2 × D
17
182.44
3 × E, B, 3 × D
24
160.50
4 × E, B, 4 × D
31
132.00
5 × E, B, 5 × D
38
100.20
6 × E, B, 6 × D
45
50.12
7 × E, B, 7 × D
52
38.26
TABLE II: FID scores on the MedNet dataset for different U-ViT configurations. E, B, and D denote encoder, bottleneck, and decoder layers, respectively.
U2 -ViT Configuration
FIDw/o−C
FIDC
Linear stack of 3 U-ViTs
38.22
–
1 × E, B, 1 × D
35.99
34.20
2 × E, B, 2 × D
32.15
23.04
3 × E, B, 3 × D
33.91
28.16
4 × E, B, 4 × D
35.45
29.20
TABLE III: FID scores on the MedNet dataset for different U2 -ViT configurations. FIDw/o−C and FIDC denote results without and with intermediate feature concatenation, respectively.
Medical image synthesis is crucial for alleviating data scarcity and privacy constraints. However, fine-tuning general text-to-image (T2I) models remains challenging, mainly due to the significant modality gap between complex visual details and abstract clinical text. In addition, semantic entanglement persists, where coarse-grained text embeddings blur the boundary between anatomical structures and imaging styles, thus weakening controllability during generation. To address this, we propose a Visually-Guided Text Disentanglement framework. We introduce a cross-modal latent alignment mechanism that leverages visual priors to explicitly disentangle unstructured text into independent semantic representations. Subsequently, a Hybrid Feature Fusion Module (HFFM) injects these features into a Diffusion Transformer (DiT) through separated channels, enabling fine-grained structural control. Experimental results in three datasets demonstrate that our method outperforms existing approaches in terms of generation quality and significantly improves performance on downstream classification tasks. The source code is available at https://github.com/hx111/VG-MedGen.
Xin Huang, Junjie Liang, Qingshan Hou +4
Computer Science and Engineering, Northeastern University, Shenyang, China · Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University, Shenyang, China · National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, China +2
Medical image generators trained on imbalanced data can fail at demographic intersections absent from training. We introduce CompDiff, which encodes age, sex and race separately and composes supervised demographic tokens alongside clinical text. Across chest radiographs and fundus images, CompDiff improves overall and subgroup fidelity relative to prompt conditioning (RoentGen-v2) and loss reweighting (FairDiffusion). It generalises in zero-shot generation to 16 chest X-ray intersections excluded from training, achieving the lowest mean FID-RadImageNet in every intersection. In a blinded reader study of these unseen intersections, two radiologists gave CompDiff the highest mean scores among generators for anatomical realism and agreement with the clinical impression, and selected its images most often as the most realistic. Pretraining with CompDiff images improved downstream classification, while CompDiff audit cohorts reduced estimation error on rare intersections. These findings support compositional demographic conditioning for extending medical image synthesis to underserved populations. Code: https://github.com/mahmoudibrahim98/CompDiff
Mahmoud K. Ibrahim, Bart Elen, Chang Sun +5
Institute of Data Science, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands · Department of Advanced Computing Sciences, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands · VITO, Belgium
Data-driven medical AI is traditionally formulated as a discriminative mapping from input X to output Y via a learned function f, which does not generalize well across heterogeneous data and modalities encountered in real-world clinical settings. In this work, we propose a fundamentally different, generative paradigm. We model the joint distribution P(X,Y) using diffusion models and reframe inference as a test-time output optimization problem. By guiding the generative process to match observed inputs, our framework enables flexible, gradient-based conditioning at inference time without architectural changes or retraining, effectively supporting arbitrary and previously unseen combinations of observations. Extensive experiments demonstrate strong performance across standard and cross-modality medical image segmentation, few-shot segmentation with only 2 or 4 training samples, degraded-input segmentation, shape completion from sparse and partial observations, and zero-shot application to demonstrate generality. To support these evaluations, we curated and released a large-scale text-shape dataset derived from MedShapeNet. Our results highlight the versatility of generative joint modeling as a foundation for reusable, task-agnostic medical AI systems.
Hantao Zhang, Weidong Guo, Yuhe Liu +5
CVLab, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland · Fudan University, Shanghai, China · Beihang University, Beijing, China +3