cs.CVNov 3, 2026

From Natural Images to Medical Scans: Transferability of Diffusion Models for Medical Image Generation

Authors: Efthymios Georgiou, Ioannis Kakogeorgiou, Sarah Brüningk

Organizations: Department of Digital Medicine, University of Bern, Switzerland · Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Switzerland · IIT, National Centre for Scientific Research “Demokritos”, Greece

Abstract

Transfer learning from natural images is standard in supervised and increasingly adopted in self-supervised medical image analysis, but its role in generative modelling, and in particular diffusion denoisers, remains underexplored. Medical diffusion models typically train the denoiser from scratch, with no systematic evaluation of natural-image transfer. We analyse this across latent and pixel paradigms, model sizes (131M-952M) and two radiology datasets (RadImageNet, BraTS), and ask which aspects of transfer support generation. We find that: i) Natural-image initialisation consistently improves generation, by 21-34% relative gFID in pixel space against 14% in latent space, where the tokeniser supplies much of the prior. gFID gains mirror downstream pathology classification and radiomic-feature FD, both proxies for clinical fidelity. ii) Benefits reflect spatial structure, not semantics: semantic probing does not predict generation quality, whereas a spatial-structure metric adapted to internal denoiser representations tracks gFID at ∣r∣=0.85|r|=0.85, positioning spatial organisation as a channel through which pretraining manifests. iii) Mixed-domain denoiser training shows that natural and medical images share similar layer-wise representations up to the late blocks, and mixing natural images into medical training improves fidelity. iv) Transfer benefit is tied to natural image initialisation, outperforming in-domain medical pretraining and several reinitialisation schemes. Overall, natural image denoisers transfer positively to medical ones in our setup. Code is available at https://github.com/CAIROLab-Bern/rad-dit

Figures & tables

Explore similar work

CardsList
  1. 3D Diffusion Models for Medical Imaging

    Oct 19, 2026Frank-Peter Schilling, Daniel Barco, Philipp Denzel +13D Diffusion ModelsMedical Image Synthesis

  2. CompDiff enables fair and zero shot medical image generation across demographic intersections through compositional diffusion

    Mar 17, 2026Mahmoud K. Ibrahim, Bart Elen, Chang Sun +5HealthcareCompositional Image Generation

  3. Domain-Specific Latent Representations Improve the Fidelity of Diffusion-Based Medical Image Super-Resolution

    Apr 14, 2026Sebastian Cajas, Ashaba Judith, Rahul Gorijavolu +8Medical Image Super-ResolutionDiffusion Models for Image Restoration