Can General-Purpose Generative AI Supply Training Data for Medical Vision Models? A Multimodal Retrospective Study
Authors: Jiachuan Peng, Kyle Lam, Jierui Qu, Lin Li, Haining Zheng, Bang Zheng, Aniruddha Agarwal, Zaki Almallah, Min Chen, Vicente Grau, Hagai Rossman, Eran Segal, Jianing Qiu
Organizations: Biological and Life Sciences Division, MBZUAI, UAE. · Department of Engineering Science, University of Oxford, UK. · Department of Surgery and Cancer, Imperial College London, UK. · OATML, Department of Computer Science, University of Oxford, UK. · Department of Ultrasound, Peking University International Hospital, China. · School of Public Health, Peking University, China. · Eye Institute, Cleveland Clinic Abu Dhabi, UAE. · Cleveland Clinic Lerner College of Medicine, Case Western Reserve University, USA. · Integrated Surgical Institute, Cleveland Clinic Abu Dhabi, UAE. · Pheno.AI, Israel. · Department of Computer Science and Applied Mathematics, Weizmann Institute of Science, Israel.
Background: Well-annotated, diverse medical imaging datasets remain a major constraint for vision-based medical AI. We evaluated Nano Banana, a general-purpose image generation and editing model, as a source of synthetic medical images for model training, subgroup augmentation, and model auditing. Methods: In this retrospective study across four imaging modalities, chest radiography, fundus photography, and CT models were trained only on synthetic images and externally validated on real-world data against real-data-trained models. Carotid ultrasound models were trained on a private Human Phenotype Project cohort (aged 40-75 years) with and without synthetic augmentation. Clinicians rated image plausibility. Evaluation used AUROC, AUPR, calibration, and decision curve analysis. Findings: Synthetic-only models achieved AUROCs of 0.877 (95% CI 0.843-0.910) for pneumonia (n=14863), 0.963 (0.948-0.975) for retinal detachment (n=1149), and 0.711 (0.600-0.823) for renal calculus (n=4122), comparable to real-data-trained models, but their calibration and net benefit were not consistently more favorable. With mixed synthetic-real training, a carotid ultrasound model produced an ordinal vascular age index that stratified 633 external subjects into subgroups differing in BMI and glucose (both p<0.01), linking vascular aging to cardiometabolic health. To stress-test shortcut learning, editing a moderate tessellated pattern into fundus photographs reduced the true-class probability for central serous chorioretinopathy from 0.996 to 0.822. Clinician-guided editing raised low-quality CT plausibility from 3.10/5 to 4.30/5. Interpretation: Synthetic data can complement real data in model training, subgroup representation, and model testing via controlled editing, but its clinical value should be assessed with calibration, decision curve analysis, and clinician review, not discrimination alone.
Figures & tables
Figure 1: Overview of the proposed pipeline for developing and validating vision-based medical AI models with synthetic data generation and image editing using Nano Banana. (A) Synthetic Data Generation and Image Editing: prompts are used to generate synthetic medical images with variations in demographics, anatomical structures, modalities, and medical artifacts, and to edit real medical images by removing non-anatomical cues, modifying clinical attributes, or adding imaging artifacts for shortcut and robustness testing. Generated images undergo quality checks and cropping. (B) Model Training: models pre-trained on ImageNet are fine-tuned using two strategies: synthetic medical data only and a mix of synthetic and real-world data. (C) Model Evaluation: models are assessed quantitatively using discrimination metrics, calibration, clinical utility metrics, and qualitatively using heatmap visualization, latent space analysis, and clinical plausibility inspection. Experiments include both private and public datasets to ensure a fair evaluation of model generalization.
Figure 2: Comprehensive evaluation of models fine-tuned with Nano Banana synthetic data across multiple medical imaging tasks. (A-D) AUROC and AUPR curves for disease recognition on the Chest X-ray Dataset (A-B) and RSNA Pneumonia Detection Dataset (C-D), comparing cropped and raw Nano Banana synthetic images with real-image and RoentGen-v2 baselines. (E-F) Performance comparison on the CT-Kidney Dataset between models fine-tuned using generated CT images and an equal number of real CT images for renal calculus detection. (G-J) RD detection performance on the Eye Disease Image Dataset (G-H) and HPP Dataset (I-J), comparing cropped synthetic images, raw synthetic images, and VisionFM fine-tuned on JSIEC. (K-L) Results for HPP age prediction using HPP data alone or augmented with cropped/raw synthetic data across different age groups. (M–P) Age predictor external evaluation on the CUBS dataset. (M–N) Carotid ultrasound-derived ordinal age index grouping and glucose distributions using the HPP-only model. (O–P) Corresponding results using the synthetic data-augmented model. Bottom and top 10% subgroups were selected. Brackets denote paired statistical comparisons, and ns indicates not significant.
Figure S1: Uniform Manifold Approximation and Projection (UMAP) visualization of features extracted from the classifiers fine-tuned on different datasets. For each feature set, we performed K-means clustering and reported the resulting classification Accuracy (Acc) and adjusted Rand index (ARI). We conducted evaluation separately on the Chest X-Ray dataset (n=5856) (A-D) and the 2018 RSNA Pneumonia Detection Challenge dataset (n=14 863) (E-H).
Figure S2: Grad-CAM visualization of the feature map of the last convolutional layer before the classification head. A: original image; B: fine-tuned on cropped CXRs generated by Nano Banana; C: on raw Nano Banana-generated CXRs; and D: on CXRs generated by RoentGen-v2.
Figure S3: Examples of RSNA chest X-ray image editing. The top row shows two examples of removing image markers and labels from the original images, while the bottom row shows two examples of adding a mild calcified granuloma. In each pair, the left image is the original image and the right image is the edited version.
Figure S4: Calibration and decision-curve analyses for CXR and CT diagnostic tasks. In the calibration plots, solid curves represent locally smoothed calibration estimates, open triangles represent grouped estimates from ten approximately equal-sized probability groups, and the grey dashed diagonal denotes perfect calibration. Smoothed calibration plots (A and C) compare model estimated probabilities with observed event proportions. In the decision curves, colored curves represent smoothed model net benefit, while the grey horizontal line and black curve denote the treat-none and treat-all strategies, respectively. A model provides greater decision-analytic utility where its curve lies above both reference strategies. Decision curves (B and D) show the corresponding net benefit across decision thresholds. Panels A-B show pneumonia recognition on the Chest X-Ray dataset, comparing classifiers trained on cropped Nano Banana-generated CXRs, raw Nano Banana-generated CXRs, and size matched real CXRs from CheXpert. Panels C-D show renal calculus detection on the CT-Kidney dataset, comparing classifiers trained on generated CT images and real CT images. For each training condition, predicted probabilities were averaged across repeated seed runs before plotting.
Figure S5: Additional calibration and decision-curve analyses on external chest-radiograph and fundus datasets. Smoothed calibration plots (A, C, and E) and decision curves (B, D, and F) are shown for the RSNA Pneumonia Detection dataset (A-B), the Eye Disease Image Dataset (C-D), and the HPP retinal detachment cohort (E-F).
Figure S6: Grad-CAM visualization of the feature map. A: original image; B: fine-tuned on processed CT scans generated by Nano Banana; and C: on real CT scans.
Figure S7: Grad-CAM visualization of the feature map of the last convolutional layer before the classification head. A: original image; B: fine-tuned on cropped Nano Banana-generated fundus images; and C: on raw Nano Banana-generated fundus images.
Figure S8: Example images demonstrating Nano Banana editing function. From left to right: the original image sampled from JSIEC , edited shadow artifact image, edited mild tessellated fundus image, and edited moderate tessellated fundus image.
Figure S9: Results of three-class age prediction ( ≤44 , 45–59, and ≥60 ). (A) AUROC and (B) AUPR evaluated across the three age groups and the overall mean. Each bar represents a different training configuration, including the baseline model trained with HPP data alone and models augmented with either cropped or raw synthetic images generated for specific age groups (40-44, 60-75, or both). Additional replacement experiments were performed by randomly deleting real images from each of the ≤44 and ≥60 groups and adding an equal number of cropped or raw synthetic images, thereby keeping the total training set size unchanged. Error bars indicate 95% CI. (C) shows the BMI prediction results for HPP data added with cropped synthetic data (left), added with raw synthetic data (middle), and HPP data alone (right).
Figure S10: Clinical characteristics of participants with low and high carotid ultrasound-derived ordinal age index. (A) Model trained using HPP data alone. (B) Model trained using HPP data supplemented with synthetic images generated for the 40–44 age group. Results are presented as box plots with individual participant values, with the center line indicating the median and boxes indicating the interquartile range. Welch t-test was used for statistical significance test, and the corresponding P values are shown.
Figure S11: Grad-CAM visualizations of the feature maps for models trained with: (B) mixed data (HPP data and cropped synthetic images); (C) mixed data (HPP data and raw synthetic images); and (D) HPP data alone.
Figure S12: Example of a generated carotid Doppler ultrasound image showing B-mode anatomy, color-flow mapping, and the corresponding spectral Doppler waveform in moderate carotid stenosis.
Figure S13: Screenshot of the developed website for CXR Turing Test. 50 synthetic CXRs and 50 real CXRs randomly selected from Chest X-ray Dataset and RSNA Pneumonia Dataset were mixed and shuffled for test. The participant was not informed of the exact number of synthetic and real images included in the test.
Figure S14: Screenshot of the developed website for the CT clinical plausibility test. The expert rated the clinical plausibility of generated kidney CT images on a 1–5 scale.
Figure S15: Screenshot of the developed website for the fundus image editing clinical plausibility test. The expert rated edited fundus images on a 0–5 scale across two tasks involving different types of image editing: imaging artifact editing and tessellated-pattern editing.
Figure S16: Screenshot of the developed website for the carotid ultrasound reader study. The study included three tasks: age comparison, BMI comparison, and carotid stenosis grading with Doppler images. The expert also rated the clinical plausibility of the ultrasound images on a 1–5 scale.
Figure S17: Representative raw and cropped images generated by Nano Banana across four imaging modalities. For each modality, the top row (A, C, E) shows raw images and the bottom row (B, D, F) shows the corresponding cropped images. Rows A–B (chest X-ray): healthy subjects (columns 1–3) and subjects with pneumonia (columns 4–6), illustrating diverse acquisition characteristics, including acquisition-related pose tilt (column 1), breast prostheses (column 2), and suboptimal inspiratory effort (column 3). Rows C–D (fundus photograph): normal eyes (columns 1–2), mild RD (columns 3–4), and severe RD (columns 5–6). Rows E–F (carotid ultrasound): subjects aged ≥60 years (column 1), subjects aged <45 years (column 2), and individuals with BMI 17≤BMI≤25 (column 3). Row G (abdominal CT): examples with renal calculi (columns 1–2) and without renal calculi (columns 3–4).
Synthetic data evaluation measures whether generated examples preserve the properties needed for a stated use, including fidelity, privacy, and utility.
With the rapid adoption of generative AI, synthetic medical images pose growing risks, including diagnostic deception and insurance fraud. Although prior work has explored vision-language model (VLM)-based synthetic image detection, these evaluations typically consider images in isolation. In clinical practice, however, images are interpreted alongside structured records and metadata, and VLMs are increasingly deployed under joint image-record inputs. We uncover a previously underexamined multimodal vulnerability: when given both modalities, VLMs may overweight record context in authenticity judgments, such that the same image receives different predictions solely due to changes in its accompanying text. This raises concerns about robustness in real-world deployment. To systematically characterize this effect, we reformulate synthetic medical image detection as an audit of multimodal robustness at the image-record interface and introduce a paired benchmark that holds the image fixed while swapping controlled metadata variants. Across multiple imaging modalities, we evaluate diverse open-weight and frontier API VLMs and find that changing the metadata context alone can flip authenticity judgments, with accuracy on authentic images dropping by 61.1% on average under an explicit AI-origin tag. We further propose an inference-time mitigation pipeline that detects and neutralizes provenance shortcuts without model retraining, substantially outperforming direct prompt-based suppression on the affected subset. Our benchmark provides a standardized tool for assessing and improving multimodal robustness beyond image-only settings. Code and data will be released upon acceptance.
Ching-Hao Chiu, Hao-Wei Chung, Gelei Xu +7
University of Notre Dame, Notre Dame, IN, USA. · IBM Research, Yorktown Heights, NY, USA. · Department of Cardiology, Boston Children’s Hospital, Boston, MA, USA. +2
Medical foundation models learn latent representations of clinically meaningful phenotypes, yet their ability to support controllable image generation remains largely unexplored. We evaluate four retinal foundation models within the representation tokenizer framework and examine whether demographic and clinical information encoded in latent representations from foundation models is preserved during synthetic image generation. We show that generated representations and images faithfully inherit phenotype information when evaluated within their originating foundation models, consistently outperforming conventional latent diffusion on multiple downstream prediction tasks. However, these gains largely disappear when evaluated using classifiers trained on real images, revealing a previously uncharacterised synthetic-to-real representation gap. These findings demonstrate that foundation-model latent spaces provide a powerful substrate for controllable retinal synthesis while highlighting the need to better align synthetic representations with real-image distributions.
Zuzanna A. Wakefield-Skórniewska, Bartłomiej W. Papież
Big Data Institute, Nuffield Department of Population Health, University of Oxford, United Kingdom
Generative augmentation is often proposed as a remedy for small medical-image datasets, but synthetic images are only useful when they improve downstream task performance. "Augmentation" here means synthetic supplementation: GAN-generated samples added to the real training pool, not geometric or photometric transforms of existing images. Twelve class-plane StyleGAN2-ADA generators were trained on constrained BRISC 2025 partitions to test whether their output, with or without InceptionV3 feature-space filtering, improves held-out tumour classification across three classifier families: a random forest (RF) on InceptionV3 features, a compact two-headed convolutional neural network (CNN), and MobileViTV2, a mobile hybrid convolutional-transformer. Each was evaluated at 1:1 and 1:2 real-to-synthetic ratios. An independent GPT-5.5 blind test placed gated real-versus-synthetic discrimination at 57.73% (95% CI: 54.48--60.92%) on the model-legible subset -- modestly above chance. The RF classifier did not benefit from the synthetic MRIs. The CNN showed consistent mean gains that did not survive Holm correction. MobileViTV2 showed the clearest benefit: filtered 1:1 augmentation improved tumour classification accuracy by 1.02% absolute (95% CI: 0.54--1.54%; Holm-corrected p = 0.0104). A secondary efficiency analysis found that every augmented CNN condition selected its checkpoint 42--64% earlier than baseline, while compute-matched MobileViTV2 runs reached selection after 50--67% fewer real-data epochs. Overall, augmentation utility was found to be architecture- and ratio-dependent, not guaranteed by visual fidelity alone.