Despite the recent success of Multimodal Large Language Models (MLLMs), existing approaches predominantly assume the availability of multiple modalities during training and inference. In practice, multimodal data is often incomplete because modalities may be missing, collected asynchronously, or available only for a subset of examples. In this work, we propose PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting. PRIMO enables the use of all available training examples, whether modalities are complete or partial. Specifically, it models the missing modality through a latent variable that captures its relationship with the observed modality in the context of prediction. During inference, we draw many samples from the learned distribution over the missing modality to both obtain the marginal predictive distribution (for the purpose of prediction) and analyze the impact of the missing modalities on the prediction for each instance. We evaluate PRIMO on a synthetic XOR dataset, Audio-Vision MNIST, and MIMIC-III for mortality and ICD-9 prediction. Across all datasets, PRIMO obtains performance comparable to unimodal baselines when a modality is fully missing and to multimodal baselines when all modalities are available. PRIMO quantifies the predictive impact of a modality at the instance level using a variance-based metric computed from predictions across latent completions. We visually demonstrate how varying completions of the missing modality result in a set of plausible labels.
Figures & tables
Figure 1: Overview of PRIMO. Given an observed modality xo and an additional modality xm that may be missing, PRIMO samples a latent variable z conditioned on the available modalities. The classifier maps (xo,z) to predictions, and the conditional variance Vz[p(y∣xo,z)] quantifies how changes in z affect the prediction. When both modalities are observed (orange), V is lower. When a modality is missing (red), V is higher. PRIMO then clusters the output logits across latent samples to visualize plausible labels under each availability scenario.
Figure 2: DGP for missing modalities. The dashed line denotes an a priori correlation between the two modalities.
Figure 3: Evaluation on the XOR dataset. (Left.) Accuracy under complete and missing-modality inputs. PRIMO matches the unimodal baseline (xo) when xm is missing and matches the multimodal baseline (xo,xm) when both modalities are observed, outperforming the remaining baselines. (Right.) Scatter plot of the predictive impact gap Vmissing−Vcomplete . The gap is small for examples with xo>0 , where the label can be determined by xo only, and larger for xo<0 , where xm affects the label.
Figure 6: Qualitative analysis of modality impact on AV-MNIST under audio-missing (top) and vision-missing (bottom) . We visualize plausible label outcomes induced by varying the latent completion z . High- V examples yield multiple plausible label clusters under missingness, while low- V examples concentrate on a single dominant label.
Figure 5
Figure 9: Patient-level analysis on MIMIC-III under missing time-series modality. We report Vmissing as V and visualize the fraction of clusters assigned to each label. High- V examples yield clusters spread across multiple labels, indicating ambiguity for (a) mortality risk and (c) respiratory disease diagnosis, whereas (b) neoplasm prediction concentrates on a single label and remains stable.
Figure 10: Bias analysis with vision missing. PRIMO (xo) stays close to the unimodal oracle, while PRIMO (xo,xm) stays close to the multimodal oracle. The dashed arc shows the unimodal–multimodal oracle gap.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: XOR predictions under missing and complete modalities. Each column shows a method and each row corresponds to the modality-availability setting (top: xm missing, bottom: complete). For each input xo , we sample latent completions and visualize the induced distribution over predicted labels; points are colored by the predicted class. Methods that capture label-relevant uncertainty produce multiple plausible labels in regions where the XOR label depends on xm (e.g., xo<0 ), while predictions concentrate on a single label when xo is sufficient (e.g., xo>0 ). Accuracies are shown in the column headers.
Figure 13: Patient-level mortality results on MIMIC-III under missing time-series inputs. Each panel shows the fraction of clusters assigned to each mortality class, with Vmissing reported as V . The top row shows high- V examples with clusters spread across multiple risk classes, while the bottom row shows low- V examples dominated by a single class.
Figure 14: Patient-level ICD-9 (140–239) results on MIMIC-III under missing time-series inputs. Each panel shows the fraction of clusters assigned to each label, with Vmissing reported as V . The top row shows high- V examples and the bottom row shows low- V examples; in both cases, clusters are largely dominated by a single label, indicating stable predictions.
Figure 15: Patient-level ICD-9 (460–519) results on MIMIC-III under missing time-series inputs. Each panel shows the fraction of clusters assigned to each label, with Vmissing reported as V . The top row shows high- V examples with clusters spread across multiple labels, indicating ambiguity when time-series is missing, while the bottom row shows low- V examples dominated by a single label.
This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction. We propose modality-specific normalization and a novel partial-modality distance metric to enable fair clustering of incomplete observations, capturing cross-modal structures while preserving scale-invariance across varying dimensionalities and modality counts. Cluster centers from this unsupervised stage guide an iterative greedy imputation process for any missing modalities during training or inference, supporting arbitrary numbers of modalities and arbitrary missing patterns per sample. The imputation module is lightweight, uses frozen encoders, and decouples from the downstream task, allowing easy integration with any fusion/prediction architecture. Extensive experiments under diverse and highly incomplete regimes demonstrate UL4M4's robustness, achieving, to the best of our knowledge, the first consistent F1-Micro scores above 0.7 on challenging missing configurations even when more than 50% of modality slots are missing. Results are also stable across cluster sizes and significantly outperform state-of-the-art baselines. Code is available here: https://github.com/h-ismkhan/Multimodal-Learning-with-Missing-Modalities-via-Unsupervised-Learning.
Real-world multimodal learning is often hindered by missing modalities. While Incomplete Multimodal Learning (IML) has gained traction, existing methods typically rely on the unrealistic assumption of full-modal availability during training to provide reconstruction supervision or cross-modal priors. This paper tackles the more challenging setting of IML under training-time incomplete observations, which precludes reliance on a ``God's eye view'' of complete data. We propose LIMSSR (LLM-Driven Incomplete Multimodal Sequence-to-Score Reasoning), a framework that reformulates this challenge as a conditional sequence reasoning task. LIMSSR leverages the semantic reasoning capabilities of Large Language Models via Prompt-Guided Context-Aware Modality Imputation and Multidimensional Representation Fusion to infer latent semantics from available contexts without direct reconstruction. To mitigate hallucinations, we introduce a Mask-Aware Dual-Path Aggregation to dynamically calibrate inference uncertainty. Extensive experiments on three Action Quality Assessment datasets demonstrate that LIMSSR significantly outperforms state-of-the-art baselines without relying on complete training data, establishing a new paradigm for data-efficient multimodal learning. Code is available at https://github.com/XuHuangbiao/LIMSSR.
Huangbiao Xu, Huanqi Wu, Xiao Ke +1
Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing, College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China · Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou 350108, China · Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China
We study multimodal learning under missing modalities, with particular motivation from bioscience applications in which heterogeneous modalities are often only partially available when decisions need to be made. We propose Latent World Recovery (LWR), a framework built on two key ideas: (i) modality-specific embeddings from different modalities are aligned in a shared latent space, and (ii) a unified representation is constructed by fusing only the embeddings of the modalities that are actually available at both training and inference time. Rather than imputing missing modalities or requiring a fixed modality set, LWR treats each modality as a partial perception of an underlying latent state and performs availability-aware representation learning directly from the observed modalities. This combination of neighbor-based latent alignment and availability-aware modality fusion enables robust multimodal prediction under partial observation, while avoiding error propagation from explicit reconstruction of missing modalities. We evaluate the proposed framework on real-world incomplete multi-omics benchmarks and demonstrate that it provides an effective approach to downstream tasks such as cancer phenotype classification and survival prediction.
Hui Wang, Tianyu Ren, Joseph Butler +3
Queen’s University Belfast, Belfast, United Kingdom