Seeking Necessary and Sufficient Information from Multimodal Medical Data

Boyu Chen; Weiye Bao; Junjie Liu; Michael Shen; Bo Peng; Paul Taylor; Zhu Li; Mengyue Yang

cs.CVFeb 27, 2026

Seeking Necessary and Sufficient Information from Multimodal Medical Data

Authors: Boyu Chen, Weiye Bao, Junjie Liu, Michael Shen, Bo Peng, Paul Taylor, Zhu Li, Mengyue Yang

Organizations: University College London, London, UK · Imperial College London, London, UK · Mingdu Tech, China · University of Bristol, Bristol, UK

Abstract

Learning multimodal representations from medical images and other data sources can provide richer information for decision-making. While various multimodal models have been developed for this, they overlook learning features that are both necessary (must be present for the outcome to occur) and sufficient (enough to determine the outcome). We argue learning such features is crucial as they can improve model performance by capturing essential predictive information, and enhance model robustness to missing modalities as each modality can provide adequate predictive signals. Such features can be learned by leveraging the Probability of Necessity and Sufficiency (PNS) as a learning objective, an approach that has proven effective in unimodal settings. However, extending PNS to multimodal scenarios remains underexplored and is non-trivial as key conditions of PNS estimation are violated. We address this by decomposing multimodal representations into modality-invariant and modality-specific components, then deriving tractable PNS objectives for each. Experiments on synthetic and real-world medical datasets demonstrate our method's effectiveness. Code will be available on GitHub.

Figures & tables

Figure 1: Overview of the proposed MPNS framework. The figure shows the multimodal data generation process, a baseline decoupling model, and our proposed complement branch with PNS-based optimization.

	Mode	Modality 1				Modality 2
	Mode	NS	SF	NC	SC	NS	SF	NC	SC
s = 0.0	w/o PNS	0.626	0.667	0.658	0.291	0.519	0.607	0.646	0.312
	w/o $\mathcal{L}_{M,I}^{pns}$	0.645	0.681	0.665	0.294	0.536	0.597	0.633	0.323
	w/o $\mathcal{L}_{M,S}^{pns}$	0.663	0.673	0.644	0.305	0.627	0.577	0.636	0.349
	MPNS	0.682	0.658	0.578	0.298	0.649	0.553	0.629	0.364
s = 0.3	w/o PNS	0.613	0.682	0.645	0.322	0.520	0.618	0.614	0.347

Table 1: Distance Correlation based on

s

for modality 1 and 2

Subregions	F	T1	T1c	T2	RobustSeg	RFNet	mmFormer	M 3 AE	ShaSpec	ShaSpec (MPNS)	DC-Seg	DC-Seg (MPNS)
Whole	$\circ$	$\circ$	$\circ$	$\bullet$	82.20	86.05	85.51	86.10	83.98	83.86	85.16	85.09
	$\circ$	$\circ$	$\bullet$	$\circ$	71.39	76.77	78.04	78.90	76.18	76.91 †	78.57	79.45 †
	$\circ$	$\bullet$	$\circ$	$\circ$	71.41	77.16	76.24	79.00	74.27	74.65 †	77.33	76.96
	$\bullet$	$\circ$	$\circ$	$\circ$	82.87	87.32	86.54	88.00	89.25	89.25	88.86	89.01 †
	$\circ$	$\circ$	$\bullet$	$\bullet$	85.97	87.74	87.52	87.10	86.57	86.21	87.94	88.07 †
	$\circ$	$\bullet$	$\bullet$	$\circ$	76.84	81.12	80.70	80.10	78.92	78.90	81.99	82.00 †

Table 2: Dice coefficients on BraTS2020 with missing modalities. Notes:

\bullet

\circ

: available/missing modality. Bold / underlined : best / second-best result.

\dagger

: MPNS outperforms its base decoupling model.

Medical Image Representation Learning

Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.

See all 260 papers

Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling
Feb 19, 2026Divyam Madaan, Sumit Chopra, Kyunghyun ChoMissing-Modality Learning Multimodal Learning
New York University. · New York University Grossman School of Medicine · Genentech +1
Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data
May 25, 2026Tianling Liu, Lequan Yu, Tong Han +1Missing-Modality Learning Multimodal Robustness
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China. · Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong, Hong Kong. · Department of Radiology, Tianjin Huanhu Hospital, Tianjin 300350, China. +2
InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision
Sep 28, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +9Multimodal Pretraining Multimodal QA
The Hong Kong Polytechnic University · PolyU-Daya Bay Technology and Innovation Research Institute

Feb 19, 2026J/K move · Enter open · S save

Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling

Divyam Madaan, Sumit Chopra, Kyunghyun Cho

New York University. · New York University Grossman School of Medicine · Genentech+1

Missing-Modality Learning Multimodal Learning Multimodal Classification Latent Variable Models Multimodal Model Interpretability

Open paper AskPDF arXiv

Despite the recent success of Multimodal Large Language Models (MLLMs), existing approaches predominantly assume the availability of multiple modalities during training and inference. In practice, multimodal data is often incomplete because modalities may be missing, collected asynchronously, or available only for a subset of examples. In this work, we propose PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting. PRIMO enables the use of all available training examples, whether modalities are complete or partial. Specifically, it models the missing modality through a latent variable that captures its relationship with the observed modality in the context of prediction. During inference, we draw many samples from the learned distribution over the missing modality to both obtain the marginal predictive distribution (for the purpose of prediction) and analyze the impact of the missing modalities on the prediction for each instance. We evaluate PRIMO on a synthetic XOR dataset, Audio-Vision MNIST, and MIMIC-III for mortality and ICD-9 prediction. Across all datasets, PRIMO obtains performance comparable to unimodal baselines when a modality is fully missing and to multimodal baselines when all modalities are available. PRIMO quantifies the predictive impact of a modality at the instance level using a variance-based metric computed from predictions across latent completions. We visually demonstrate how varying completions of the missing modality result in a set of plausible labels.

Abstract

Medical Image Representation Learning

Explore similar work

Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling

Context-driven Missing-Modality Learning for Robust Medical Diagnosis with Image-Tabular Data

InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision