M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model

Yihang Liu; Longzhen Yang; Jiaxiong Yang; Ying Wen; Lianghua He; Heng Tao Shen

cs.CVApr 10, 2026

M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model

Authors: Yihang Liu, Longzhen Yang, Jiaxiong Yang, Ying Wen, Lianghua He, Heng Tao Shen

Organizations: School of Computer Science and Technique, Tongji University, Shanghai 201804, China. · School of Communications and Electronic Engineering, East China Normal University, Shanghai 200241, China.

Abstract

Medical foundation models (MFMs) aim to learn universal representations from multimodal medical images that can generalize effectively to diverse downstream clinical tasks. However, most existing MFMs suffer from information ambiguity that blends multimodal representations in a single embedding space, leading to the degradation of modality specificity and diversity. In this paper, we propose M-IDoL, a self-supervised MFM that introduces Information Decomposition for multimodal representation Learning via two objectives: i) maximizing inter-modality entropy by dispersing multimodal representations into separable Mixture-of-Experts (MoE) subspaces to achieve representation specificity across modalities; and ii) minimizing intra-modality uncertainty by performing fine-grained semantic discrimination within each MoE subspace to enrich representation diversity per modality. By pre-training on 1.15 million medical images, M-IDoL i) delivers superior generalization across 21 downstream clinical tasks, outperforming 20 foundation models on five imaging modalities (e.g., X-ray, fundus, OCT, dermoscopy and pathology), and ii) learns modality-specific and diverse representations, showing clearer separation of feature clusters across modalities and finer-grained feature discrimination within each modality.

Figures & tables

Figure 1: (a) Medical images exhibit inter-modality specificity and intra-modality diversity. (b) Modality-specific models excel in intra-modality diversity by focusing on stable imaging statistics. (c) MFMs suffer from information ambiguity due to uniform maximization of redundancy (Rdncy.) information across modalities. (d) M-IDoL mitigates ambiguity via information decomposition, enhancing modality-specific and diverse representations. $X$ and $Y$ denote representations of two augmented views, $Z$ denotes representations from modalities distinct from $X$ and $Y$ . $\mathcal{I}(\cdot;\cdot)$ is mutual information, $H(\cdot)$ is information entropy. Shapes (e.g., star, circle) indicate modalities; colors indicate intra-modality semantics (e.g., diseases). t-SNE clusters for (b)–(d) are in Fig. 5 .

Figure 2: Left: Overview of M-IDoL. Via information decomposition, M-IDoL optimizes two objectives: (a) the routing-consistency loss $\mathcal{L}_{\text{route}}$ , which learns modality-separable MoE subspaces to maximize $H(X|Z)$ for inter-modality specificity, and (b) the intra-modality contrastive loss $\mathcal{L}_{\text{cst}}$ , which promotes fine-grained discrimination within each modality to minimize $H(X|Y,Z)$ for intra-modality diversity. Right: (a) Inter-modality Entropy Maximization ( $H(X|Z)$ ) and (b) Intra-modality Uncertainty Minimization ( $H(X|Y,Z)$ ).

Pre-train		Downstream
Dataset	Images	Dataset	Train	Val	Test
Retinology (Fundus)		Retinology (Fundus)
		APTOS ( Karthik et al., 2019 )	2,048	514	1,100
EYEPACS ( Gulshan et al., 2016 )	88,702	Glaucoma ( Kim, 2018 )	861	218	465
		PAPILA 1 1 1 PAPILA (2022), a retinal dataset for glaucomatous diagnosis. URL https://doi.org/10.6084/m9.figshare.14798004.v1.	311	79	98
AIROGS ( De Vente et al., 2023 )	101,442	Retina ( Ret, )	336	84	181

Table 1: Summary of the datasets used in M-IDoL.

MoE	$\mathcal{L}_{\text{route}}$	$\mathcal{L}_{\text{cst}}$	RSNA	APTOS	OCTID	Mitosis	HAM10000
✗	✗	✗	87.42 ± 0.57	90.36 ± 0.22	96.98 ± 0.47	86.72 ± 0.34	92.77 ± 0.19
✔	✗	✗	88.31 ± 0.34	87.97 ± 0.52	96.77 ± 0.59	88.64 ± 0.60	90.86 ± 0.28
✔	✔	✗	91.45 ± 0.16	93.46 ± 0.32	98.84 ± 0.31	91.75 ± 0.27	96.66 ± 0.19
✔	✔	✔	93.23 ± 0.22	95.19 ± 0.62	99.34 ± 0.35	95.62 ± 0.24	97.96 ± 0.36

Table 2: Impact of different components in M-IDoL.

Figure 3: (a) Visualization of routing assignments for 1,000 images per modality. (b) Impact of expert number on downstream tasks.

	Retinology (Fundus)
	APTOS		Glaucoma		PAPILA		Retina
Methods	ACC ( $\%,\uparrow$ )	AUC ( $\%,\uparrow$ )	ACC ( $\%,\uparrow$ )	AUC ( $\%,\uparrow$ )	ACC ( $\%,\uparrow$ )	AUC ( $\%,\uparrow$ )	ACC ( $\%,\uparrow$ )	AUC ( $\%,\uparrow$ )
RETFound ( Zhou et al., 2023b )	92.17 ± 0.35 ‡	94.36 ± 0.14 ‡	90.18 ± 0.38 ‡	94.36 ± 0.19 ‡	87.25 ± 0.26	84.04 ± 0.16	84.98 ± 0.13 ‡	85.27 ± 0.53
KeepFIT ( Wu et al., 2024 )	92.05 ± 0.63	94.26 ± 0.61	87.62 ± 1.01	91.95 ± 0.41	86.64 ± 0.19	84.88 ± 0.39 ‡	84.06 ± 0.46	87.02 ± 0.31 ‡
FLAIR ( Silva-Rodriguez et al., 2025 )	91.99 ± 0.31	93.08 ± 0.84	89.05 ± 0.34	93.99 ± 0.73	87.07 ± 0.59 ‡	83.08 ± 0.42	85.31 ± 0.39	84.28 ± 0.39
M-IDoL(ours)	93.43 ± 0.85	95.19 ± 0.62	90.97 ± 0.58	95.05 ± 0.29	88.30 ± 0.19	86.61 ± 0.34	88.4 ± 0.26	88.89 ± 0.47

Table 3: Comparison with Modality-specific Models. The best performance is bolded and the second best is underlined . ‡ denotes a statistically significant improvement (p

<

0.05) over the second-best results and n.s. denotes no significant improvement.

Figure 4: Comparison with unified MFMs. Error bars denote standard deviation, the center position reflects the mean performance score ( $\uparrow$ , %). p $<$ 0.05 indicates that our M-IDoL significantly outperforms the second-best method. Statistical results are in Appendix D.

Figure 5: t-SNE clusters of representations across five modalities. (a–e) modality-specific models, (f–h) unified MFMs and (i-j) M-IDoL.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Loss Curves from Training Log.

	APTOS	Glaucoma	PAPILA	Retina	OCTDL	OCTID	TVHL-DME
lr	5e-4	5e-4	1e-3	8e-4	1e-4	1e-4	1e-4
epoch	50	50	50	50	50	50	50
shape	224 × 224	224 × 224	224 × 224	224 × 224	224 × 224	224 × 224	224 × 224
batch size	64	64	64	64	64	64	64
task	Multi-class cls.	Multi-class cls.	Multi-class cls.	Multi-class cls.	Multi-class cls.	Multi-class cls.	Multi-class cls.
	TVHL-DR	BreakHis-Binary	BreakHis-8C	Mitosis	MHIST	ISIC2016	ISIC2017

Appendix

Table 4: Detailed configurations for each downstream task

Figure 7: Confusion matrices on classification datasets of our proposed M-IDoL.

Figure 8: Visualization of gradient-weighted class activation mapping (Grad-CAM) across modalities.

	Retinology (Fundus)
	APTOS		Glaucoma		PAPILA		Retina
Methods	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
SwinViT	89.84 ± 0.57	92.30 ± 0.76	86.27 ± 0.53	91.64 ± 1.41	81.37 ± 0.31	77.24 ± 0.97	77.88 ± 0.14	73.17 ± 0.10
DINO	90.13 ± 0.62	92.71 ± 0.29	89.54 ± 0.35	89.79 ± 0.68	81.66 ± 0.14	79.81 ± 0.19	82.32 ± 0.29	76.43 ± 0.29
MAE	91.21 ± 0.35	92.22 ± 0.44	88.41 ± 0.72	92.14 ± 0.24	79.52 ± 0.04	73.64 ± 0.07	78.31 ± 0.17	72.61 ± 0.08
Unimiss+	87.52 ± 0.88	89.52 ± 0.19	86.72 ± 0.37	90.26 ± 0.29	82.39 ± 0.14	73.24 ± 0.85	81.32 ± 0.48	76.19 ± 0.38

Appendix

Table 5: Comparison with unified pre-training MFMs. The best performance is bolded and the second best is underlined .

Medical Imaging Foundation Models

Medical imaging foundation models learn reusable representations for tasks across clinical images, modalities, and institutions.

See all 234 papers

Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models
May 21, 2026Yuting He, Chenyu You, Shuo LiMedical Imaging Foundation Models Multimodal Pretraining
Case Western Reserve University · Stony Brook University
FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging
Jul 15, 2026Shengchao Chen, Ting ShuMultimodal Federated Learning Cross-Modal Alignment
School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia
CoM $^3$ eT: A foundation model for medical image analysis through federated, multidimensional context integration
Aug 17, 2026J. Raphael Schäfer, Kai Geissler, Till Nicke +27Medical Imaging Foundation Models Multi-Task Learning
Fraunhofer Institute for Digital Medicine MEVIS · Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany · Institute of Diagnostic and Interventional Radiology, Medizinische Hochschule Hannover, Hannover, Germany +11

May 21, 2026J/K move · Enter open · S save

Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models

Yuting He, Chenyu You, Shuo Li

Case Western Reserve University · Stony Brook University

Medical Imaging Foundation Models Multimodal Pretraining Mixture-of-Experts Models Medical Image Representation Learning Multimodal Learning Modular Neural Networks

Open paper AskPDF arXiv

Multi-modality medical vision (MV) foundation models (FM) are fundamentally challenged by pronounced Non-IID feature statistics across heterogeneous imaging modalities. Monolithic self-supervised optimization on such data induces conflicting gradients, driving representations to collapse toward modality-dominant shortcuts. This work reframes this failure as an imbalance between specialization and coordination in emergent modularity, and proposes Director-Experts (DEX), a modular network that explicitly regulates these dynamics in stacked modules. Each DEX module comprises a pool of experts, dynamically adapted by our image-wise activation strategy, autonomously specializing in modality-dominant statistics, together with a director, updated via our group exponential moving average, which distills multi-expert knowledge into a shared space for semantic integration across modalities, thus driving the emergence of modular representations. We curate a new benchmark, Medical Vision Universe, over 4 million images across 10 modalities, which provides a FM-level pre-training with the broadest coverage of distinct imaging modalities to our DEX. Extensive evaluations on 26 downstream tasks demonstrate improved optimization behavior and transferability, indicating DEX as a principled step toward general-purpose multi-modality medical AI. Our code and dataset will be opened at https://github.com/YutingHe-list/DEX.

Abstract

Medical Imaging Foundation Models

Explore similar work

Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Models

FM2^22: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

CoM3^33eT: A foundation model for medical image analysis through federated, multidimensional context integration

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

CoM $^3$ eT: A foundation model for medical image analysis through federated, multidimensional context integration