Can General-Purpose Generative AI Supply Training Data for Medical Vision Models? A Multimodal Retrospective Study
Authors: Jiachuan Peng, Kyle Lam, Jierui Qu, Lin Li, Haining Zheng, Bang Zheng, Aniruddha Agarwal, Zaki Almallah, Min Chen, Vicente Grau, Hagai Rossman, Eran Segal, Jianing Qiu
Organizations: Biological and Life Sciences Division, MBZUAI, UAE. · Department of Engineering Science, University of Oxford, UK. · Department of Surgery and Cancer, Imperial College London, UK. · OATML, Department of Computer Science, University of Oxford, UK. · Department of Ultrasound, Peking University International Hospital, China. · School of Public Health, Peking University, China. · Eye Institute, Cleveland Clinic Abu Dhabi, UAE. · Cleveland Clinic Lerner College of Medicine, Case Western Reserve University, USA. · Integrated Surgical Institute, Cleveland Clinic Abu Dhabi, UAE. · Pheno.AI, Israel. · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Israel.
Background: Well-annotated, diverse medical imaging datasets remain a major constraint for vision-based medical AI. We evaluated Nano Banana, a general-purpose image generation and editing model, as a source of synthetic medical images for model training, subgroup augmentation, and model auditing. Methods: In this retrospective study across four imaging modalities, chest radiography, fundus photography, and CT models were trained only on synthetic images and externally validated on real-world data against real-data-trained models. Carotid ultrasound models were trained on a private Human Phenotype Project cohort (aged 40-75 years) with and without synthetic augmentation. Clinicians rated image plausibility. Evaluation used AUROC, AUPR, calibration, and decision curve analysis. Findings: Synthetic-only models achieved AUROCs of 0.877 (95% CI 0.843-0.910) for pneumonia (n=14863), 0.963 (0.948-0.975) for retinal detachment (n=1149), and 0.711 (0.600-0.823) for renal calculus (n=4122), comparable to real-data-trained models, but their calibration and net benefit were not consistently more favorable. With mixed synthetic-real training, a carotid ultrasound model produced an ordinal vascular age index that stratified 633 external subjects into subgroups differing in BMI and glucose (both p<0.01), linking vascular aging to cardiometabolic health. To stress-test shortcut learning, editing a moderate tessellated pattern into fundus photographs reduced the true-class probability for central serous chorioretinopathy from 0.996 to 0.822. Clinician-guided editing raised low-quality CT plausibility from 3.10/5 to 4.30/5. Interpretation: Synthetic data can complement real data in model training, subgroup representation, and model testing via controlled editing, but its clinical value should be assessed with calibration, decision curve analysis, and clinician review, not discrimination alone.
Figures & tables
Figure 1: Overview of the proposed pipeline for developing and validating vision-based medical AI models with synthetic data generation and image editing using Nano Banana. (A) Synthetic Data Generation and Image Editing: prompts are used to generate synthetic medical images with variations in demographics, anatomical structures, modalities, and medical artifacts, and to edit real medical images by removing non-anatomical cues, modifying clinical attributes, or adding imaging artifacts for shortcut and robustness testing. Generated images undergo quality checks and cropping. (B) Model Training: models pre-trained on ImageNet are fine-tuned using two strategies: synthetic medical data only and a mix of synthetic and real-world data. (C) Model Evaluation: models are assessed quantitatively using discrimination metrics, calibration, clinical utility metrics, and qualitatively using heatmap visualization, latent space analysis, and clinical plausibility inspection. Experiments include both private and public datasets to ensure a fair evaluation of model generalization.
Figure 2: Comprehensive evaluation of models fine-tuned with Nano Banana synthetic data across multiple medical imaging tasks. (A-D) AUROC and AUPR curves for disease recognition on the Chest X-ray Dataset (A-B) and RSNA Pneumonia Detection Dataset (C-D), comparing cropped and raw Nano Banana synthetic images with real-image and RoentGen-v2 baselines. (E-F) Performance comparison on the CT-Kidney Dataset between models fine-tuned using generated CT images and an equal number of real CT images for renal calculus detection. (G-J) RD detection performance on the Eye Disease Image Dataset (G-H) and HPP Dataset (I-J), comparing cropped synthetic images, raw synthetic images, and VisionFM fine-tuned on JSIEC. (K-L) Results for HPP age prediction using HPP data alone or augmented with cropped/raw synthetic data across different age groups. (M–P) Age predictor external evaluation on the CUBS dataset. (M–N) Carotid ultrasound-derived ordinal age index grouping and glucose distributions using the HPP-only model. (O–P) Corresponding results using the synthetic data-augmented model. Bottom and top 10% subgroups were selected. Brackets denote paired statistical comparisons, and ns indicates not significant.
Figure S1: Uniform Manifold Approximation and Projection (UMAP) visualization of features extracted from the classifiers fine-tuned on different datasets. For each feature set, we performed K-means clustering and reported the resulting classification Accuracy (Acc) and adjusted Rand index (ARI). We conducted evaluation separately on the Chest X-Ray dataset (n=5856) (A-D) and the 2018 RSNA Pneumonia Detection Challenge dataset (n=14 863) (E-H).
Figure S2: Grad-CAM visualization of the feature map of the last convolutional layer before the classification head. A: original image; B: fine-tuned on cropped CXRs generated by Nano Banana; C: on raw Nano Banana-generated CXRs; and D: on CXRs generated by RoentGen-v2.
Figure S3: Examples of RSNA chest X-ray image editing. The top row shows two examples of removing image markers and labels from the original images, while the bottom row shows two examples of adding a mild calcified granuloma. In each pair, the left image is the original image and the right image is the edited version.
Figure S4: Calibration and decision-curve analyses for CXR and CT diagnostic tasks. In the calibration plots, solid curves represent locally smoothed calibration estimates, open triangles represent grouped estimates from ten approximately equal-sized probability groups, and the grey dashed diagonal denotes perfect calibration. Smoothed calibration plots (A and C) compare model estimated probabilities with observed event proportions. In the decision curves, colored curves represent smoothed model net benefit, while the grey horizontal line and black curve denote the treat-none and treat-all strategies, respectively. A model provides greater decision-analytic utility where its curve lies above both reference strategies. Decision curves (B and D) show the corresponding net benefit across decision thresholds. Panels A-B show pneumonia recognition on the Chest X-Ray dataset, comparing classifiers trained on cropped Nano Banana-generated CXRs, raw Nano Banana-generated CXRs, and size matched real CXRs from CheXpert. Panels C-D show renal calculus detection on the CT-Kidney dataset, comparing classifiers trained on generated CT images and real CT images. For each training condition, predicted probabilities were averaged across repeated seed runs before plotting.
Figure S5: Additional calibration and decision-curve analyses on external chest-radiograph and fundus datasets. Smoothed calibration plots (A, C, and E) and decision curves (B, D, and F) are shown for the RSNA Pneumonia Detection dataset (A-B), the Eye Disease Image Dataset (C-D), and the HPP retinal detachment cohort (E-F).
Figure S6: Grad-CAM visualization of the feature map. A: original image; B: fine-tuned on processed CT scans generated by Nano Banana; and C: on real CT scans.
Figure S7: Grad-CAM visualization of the feature map of the last convolutional layer before the classification head. A: original image; B: fine-tuned on cropped Nano Banana-generated fundus images; and C: on raw Nano Banana-generated fundus images.
Figure S8: Example images demonstrating Nano Banana editing function. From left to right: the original image sampled from JSIEC , edited shadow artifact image, edited mild tessellated fundus image, and edited moderate tessellated fundus image.
Figure S9: Results of three-class age prediction ( ≤44 , 45–59, and ≥60 ). (A) AUROC and (B) AUPR evaluated across the three age groups and the overall mean. Each bar represents a different training configuration, including the baseline model trained with HPP data alone and models augmented with either cropped or raw synthetic images generated for specific age groups (40-44, 60-75, or both). Additional replacement experiments were performed by randomly deleting real images from each of the ≤44 and ≥60 groups and adding an equal number of cropped or raw synthetic images, thereby keeping the total training set size unchanged. Error bars indicate 95% CI. (C) shows the BMI prediction results for HPP data added with cropped synthetic data (left), added with raw synthetic data (middle), and HPP data alone (right).
Figure S10: Clinical characteristics of participants with low and high carotid ultrasound-derived ordinal age index. (A) Model trained using HPP data alone. (B) Model trained using HPP data supplemented with synthetic images generated for the 40–44 age group. Results are presented as box plots with individual participant values, with the center line indicating the median and boxes indicating the interquartile range. Welch t-test was used for statistical significance test, and the corresponding P values are shown.
Figure S11: Grad-CAM visualizations of the feature maps for models trained with: (B) mixed data (HPP data and cropped synthetic images); (C) mixed data (HPP data and raw synthetic images); and (D) HPP data alone.
Figure S12: Example of a generated carotid Doppler ultrasound image showing B-mode anatomy, color-flow mapping, and the corresponding spectral Doppler waveform in moderate carotid stenosis.
Figure S13: Screenshot of the developed website for CXR Turing Test. 50 synthetic CXRs and 50 real CXRs randomly selected from Chest X-ray Dataset and RSNA Pneumonia Dataset were mixed and shuffled for test. The participant was not informed of the exact number of synthetic and real images included in the test.
Figure S14: Screenshot of the developed website for the CT clinical plausibility test. The expert rated the clinical plausibility of generated kidney CT images on a 1–5 scale.
Figure S15: Screenshot of the developed website for the fundus image editing clinical plausibility test. The expert rated edited fundus images on a 0–5 scale across two tasks involving different types of image editing: imaging artifact editing and tessellated-pattern editing.
Figure S16: Screenshot of the developed website for the carotid ultrasound reader study. The study included three tasks: age comparison, BMI comparison, and carotid stenosis grading with Doppler images. The expert also rated the clinical plausibility of the ultrasound images on a 1–5 scale.
Figure S17: Representative raw and cropped images generated by Nano Banana across four imaging modalities. For each modality, the top row (A, C, E) shows raw images and the bottom row (B, D, F) shows the corresponding cropped images. Rows A–B (chest X-ray): healthy subjects (columns 1–3) and subjects with pneumonia (columns 4–6), illustrating diverse acquisition characteristics, including acquisition-related pose tilt (column 1), breast prostheses (column 2), and suboptimal inspiratory effort (column 3). Rows C–D (fundus photograph): normal eyes (columns 1–2), mild RD (columns 3–4), and severe RD (columns 5–6). Rows E–F (carotid ultrasound): subjects aged ≥60 years (column 1), subjects aged <45 years (column 2), and individuals with BMI 17≤BMI≤25 (column 3). Row G (abdominal CT): examples with renal calculi (columns 1–2) and without renal calculi (columns 3–4).
Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.
University of Notre Dame, Notre Dame, IN, USA. · IBM Research, Yorktown Heights, NY, USA. · Department of Cardiology, Boston Children’s Hospital, Boston, MA, USA. +2