Med-RADIO: Reducing All Medical Domains Into One via Multi-Teacher Distillation

Chu Zhang; Haoyu Jiang; Hongyuan Zhang; Hongbin Liu; Dong Yi

cs.CVSep 29, 2026

Med-RADIO: Reducing All Medical Domains Into One via Multi-Teacher Distillation

Authors: Chu Zhang, Haoyu Jiang, Hongyuan Zhang, Hongbin Liu, Dong Yi

Organizations: Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China

Abstract

The rapid expansion of large-scale medical datasets and computational resources has driven significant progress in medical foundation models. Given the inherent heterogeneity of medical imaging modalities, current research mainly follows two paths: specialized models optimized for specific modalities, and generalist models designed to handle multiple modalities. However, medical generalist models suffer from both insufficient training data scale relative to natural image generalists and inadequate domain-specific depth relative to medical specialists. Empirically, generalist models establish a cross-modality performance baseline, while specialists define the performance ceiling within their respective domains. To elevate this baseline toward these ceilings, we propose Med-RADIO, a medical multi-teacher distillation framework that Reduces All Domains Into One by compressing complementary expertise from multiple domain-specific teachers into a unified medical vision foundation model. Our method curates both generalist and specialist teachers, allocates modality-aligned distillation streams to reorganize generalist pretraining data so it matches specialist domains, and uses a balanced loss to prevent any single teacher from dominating the distillation process. On internal and external classification benchmarks spanning five modalities, Med-RADIO improves over strong medical generalists under linear probing and remains competitive with representative specialists on most evaluated modalities. Code is available at https://github.com/CAIR-HKISI/Med-RADIO.

Figures & tables

Figure 1 : Motivation and overview. (a) Generalist–specialist performance gap under limited medical data scale. (b) Med-RADIO compresses heterogeneous teachers into one student encoder. (c) Linear-probing average versus medical generalist baselines (85.28% for Med-RADIO).

Figure 2 : Med-RADIO training pipeline: modality-aligned streams, frozen generalist/specialist teachers, translator heads, and a dispersion-balanced distillation objective feeding a shared student encoder.

Model	US		MRI		CT			Histo	Derm	Avg
Model	Thyroid	Breast	ACL	Meniscus	Axial	Coronal	Sagittal	PCam	HAM	Avg
Biomed-CLIP [ 54 ]	78.25	81.79	89.47	86.99	67.64	57.93	54.05	83.40	83.71	75.91
Pubmed-CLIP [ 8 ]	78.36	84.92	90.91	88.68	72.33	59.39	54.85	82.43	82.62	77.17
PMC-CLIP [ 24 ]	56.96	66.72	68.75	74.61	37.86	34.14	34.95	75.35	52.25	55.73
BMCA-CLIP [ 27 ]	82.57	82.10	93.42	92.06	69.90	61.65	51.78	85.56	81.88	77.88
MMKD-CLIP [ 47 ]	63.63	76.52	97.30	94.91	72.01	62.30	56.63	85.69	85.37	77.15

Table 1 : Linear probing (%) on nine classification benchmarks. Best in bold . AUC for US/MRI; ACC for CT/Histo./Derm.

Figure 3 : Linear probing versus the strongest dedicated specialist per modality. Bars show absolute AUC/ACC; $\Delta$ above each group is Med-RADIO minus the best specialist.

Figure 4 : Fine-tuning $\Delta$ (Med-RADIO $-$ best specialist per modality; pp). Positive favors Med-RADIO.

Type	Model	Internal					External					Avg
		US	MRI	CT	Histo.	Derm.	US	MRI	CT	Histo.	Derm.
MTKD	MMKD-CLIP [ 47 ]	70.08	96.11	63.65	85.72	87.87	78.17	99.96	99.69	77.00	70.02	80.95
Balancing	Uniform	83.56	95.93	73.25	86.92	90.18	79.66	99.98	98.97	71.00	75.11	82.73
	Uncertainty [ 18 ]	84.73	96.49	74.38	88.64	93.76	81.39	100.00	99.61	73.00	77.21	84.14
	AdaLoss [ 12 ]	82.89	95.87	73.30	89.15	94.21	82.52	99.99	99.63	75.00	77.61	83.95
Ours	Med-RADIO	87.11	96.69	72.98	89.64	93.44	81.04	100.00	99.69	76.00	78.25	84.66

Table 2 : Internal/external linear probing (%). External tasks are held out from teacher-pool construction; balancing variants share the Med-RADIO recipe and differ only in loss weighting. Best Avg in bold .

Figure 5 : Linear-probing ablations on nine benchmarks. Each panel contrasts two settings (shaded regions mark where the second legend entry is higher/lower). (a) w/ vs. w/o generalist teacher; (b) ViT-Large vs. ViT-Base student; (c) w/ vs. w/o dispersion-balanced loss.

Stream	#Pairs	Sources	Teacher
US	390k	RadImageNet (US subset) [ 31 ]	URFM [ 17 ]
MRI	673k	RadImageNet (MRI subset) [ 31 ]	Curia [ 4 ]
CT	290k	RadImageNet (CT subset) [ 31 ]	Curia [ 4 ]
Histo.	196k	Quilt-1M [ 13 ]	GPFM [ 30 ]
Derm.	401k	ISIC2024 archive [ 22 ]	PanDerm [ 49 ]
Mix	2.59M	PMC-OA [ 24 ] , ROCOv2 [ 43 ] , LLaVA-Med [ 23 ]	UniMed-CLIP [ 19 ]

Table S1 : Pretraining stream composition (4.54M pairs). Specialist streams are single-modality; Mix is supervised by the generalist.

Modality	Dataset	#Images	#Classes	Metric	Split
US	Thyroid	349	2	AUC	75:10:15
US	Breast	780	2	AUC	75:10:15
MRI	ACL Tear	1,022	2	AUC	75:10:15
MRI	Meniscus Tear	4,201	2	AUC	75:10:15
CT	MediMeTA Axial	12,026	11	ACC	Official
	MediMeTA Coronal	12,026	11	ACC	Official

Table S2 : Internal downstream benchmark statistics.

Modality	Model	Benchmarks		Avg
		Thyroid (AUC)	Breast (AUC)
US	USFM [ 16 ]	59.30	64.73	62.01
	EchoCare [ 53 ]	70.29	77.94	74.12
	URFM [ 17 ]	74.15	89.32	81.74
	UniMed-CLIP (base) [ 19 ]	76.96	79.34	78.15
	UniMed-CLIP (large) [ 19 ]	79.18	77.12	78.15

Table S3 : Specialist-teacher candidates under linear probing. Bold : selected modality-specific specialist (UniMed-CLIP excluded from selection).

Modality	UniMed-CLIP	URFM	Curia	GPFM	PanDerm
Mixed	$0.673$	–	–	–	–
Ultrasound	$0.136$	$0.003$	–	–	–
MRI	$0.209$	–	–	–	–
CT	$0.218$	–	$1.264$	–	–
Histopathology	$1.057$	–	–	$0.807$	–
Dermatology	$0.173$	–	–	–	$0.127$

Table S4 : Offline angular dispersion

\mathrm{Disp}(\Theta)

10{,}000

samples per native stream (Eq. 4 ). – : not applicable.

Parameter	Value
Pretraining (Student Distillation)
Architecture	ViT-L/16 (304M params)
Input resolution	$224{\times}224$
Global batch size	192 (32 per teacher stream; 6 $\times$ A100 80GB, Accelerate fp16)
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ , WD $=0.05$ )
Learning rate schedule	Cosine (peak $1{\times}10^{-4}$ , warmup 1 epoch)

Table S5 : Hyperparameter settings for distillation, linear probing, and full fine-tuning.

Explore similar work

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging
Jul 15, 2026Shengchao Chen, Ting ShuMultimodal Foundation Model Cross-Modality
School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia
Deep Reprogramming Distillation for Medical Foundation Models
May 6, 2026Siyuan Du, Yuhang Zhou, Haolin Li +5Knowledge Distillation Wireless Foundation Models
School of Computer Science, Fudan University and with the Shanghai AI Laboratory, Shanghai, China · Shanghai AI Laboratory, Shanghai, China · Cooperative Medianet Innovation Center, Shanghai Jiao Tong University and with the Shanghai AI Laboratory, Shanghai, China +3
MultiMedVision: Multi-Modal Medical Vision Framework
May 9, 2026Frank Li, Bardia Khosravi, Mohammadreza Chavoshi +5Multimodal Clinical Data Self-Supervised Vision Transformers
Emory University · Yale University

Jul 15, 2026, cs.CVJ/K move · Enter open · S save

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Shengchao Chen, Ting Shu

School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia

Multimodal Foundation Model Cross-Modality Medical Imaging Wireless Foundation Models Medical Visual Question Answering

Open paper AskPDF arXiv

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM

^2

, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM

^2

equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable

O(1/\sqrt{T})

convergence and generalization guarantees. FM

^2

further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.

Abstract

Explore similar work

FM2^22: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Deep Reprogramming Distillation for Medical Foundation Models

MultiMedVision: Multi-Modal Medical Vision Framework

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging