Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Songyuan Sui; Zhen Tan; Mohan Zhang; Rana Muhammad Shahroz Khan; Xia Hu; Tianlong Chen

cs.AIOct 3, 2026

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Authors: Songyuan Sui, Zhen Tan, Mohan Zhang, Rana Muhammad Shahroz Khan, Xia Hu, Tianlong Chen

Organizations: Rice University · Stevens Institute of Technology · University of North Carolina at Chapel Hill

Abstract

Missing modality remains a longstanding challenge in multimodal learning. Existing methods typically address this issue through modality recovery or adaptive strategies. However, they overlook models' internal cross-modal dependencies formed during multimodal training, which later impair robustness. We systematically characterize a counterintuitive deployment-time failure mode: models trained on full modalities can underperform unimodal models when one modality is missing at inference time. This pattern appears across diverse architectures, such as fusion models, CLIP-style two-tower models, and vision-language models. We show that such degradation is closely associated with learned cross-modal dependencies in the principal parameter subspaces. Multimodal training induces structured rotations of these subspaces, particularly in cross-modal interaction layers. These rotations are associated with reduced task-aligned margins and larger task-aware representation harm under missing-modality inputs. We propose Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness. It rotates the principal input subspace toward a unimodal reference along a geodesic path. We prove that this correction minimizes the distance to the reference within a fixed subspace-distance budget. Experiments across architectures and datasets show that GU improves performance under missing-modality inference while preserving full-modality accuracy, outperforming strong missing-modality robustness baselines. These findings support a geometric view of deployment-time missing-modality degradation and suggest localized subspace editing as a practical route for robustness correction.

Figures & tables

Figure 1: Missing-modality degradation across architectures and datasets. F1 scores are reported for TI-TI, T-T, and TI-T settings. In 26 of the 27 settings, we observe

\mathrm{TI\text{-}TI}>\mathrm{T\text{-}T}>\mathrm{TI\text{-}T}

, revealing a recurring deployment-time missing-modality degradation pattern.

Model Family	Backbone(s)	Variants
Fusion	RoBERTa [ 33 ] + ResNet50 [ 28 ]	Concat, Gate, Cross-Attention
Fusion (strong encoders)	Qwen [ 62 ] + OpenCLIP [ 24 ]	Concat, Gate, Cross-Attention
Two-tower	OpenCLIP	Mean Pooling, Late Concat
VLM	BLIP-2 [ 31 ]	Classifier Head

Table 1: Representative model families used in our analysis. We cover distinct multimodal integration mechanisms, including explicit fusion architectures, aligned two-tower encoders, and VLMs.

Panel of Figure 2 — Figure 2: Principal subspace rotation correlates with missing-modality degradation. (a) Within-family z-score normalization reveals a global positive association. (b) Raw within-family measurements show consistent positive associations without normalization.

Figure 3: Stage-wise distribution of task-aware representation harm. The final-error versus final-correct harm gap appears at cross-modal interaction stages and often persists near the classifier.

Dataset	Null-token	SMIL	Flex-MoE	DyMo	Interpolation	GU (Ours)
HM	57.63	56.60	47.58	48.82	58.50	$\mathbf{60.88}$ (+2.38)
IU-XRay	80.00	79.21	71.02	79.19	80.14	$\mathbf{81.41}$ (+1.27)
SNLI-VE	66.35	72.20	70.74	61.30	67.29	$\mathbf{74.41}$ (+2.21)

Table 2: Main missing-image performance (TI-T, F1

\uparrow

). Values are three-run means. Underline and bold denote the second-best and best results, and parentheses show gains over the second best.

Panel of Figure 4 — Figure 4: Representation structure and ablation analysis. (a) On SNLI-VE, GU and layer interpolation preserve similar broad UMAP layouts. (b) On IU-XRay, target-layer selection has the largest effect on robustness recovery.

Family	#Var	TI-T		TI-TI		$\Delta$ T		$\Delta$ TI
		Interp.	GU	Interp.	GU	Interp.	GU	Interp.	GU
Fusion	3	71.34	$\mathbf{73.86}$	86.33	$\mathbf{86.37}$	0.98	$\mathbf{3.49}$	$-0.13$	$\mathbf{-0.09}$
Fusion (str.)	3	72.34	$\mathbf{73.06}$	75.16	$\mathbf{75.95}$	0.96	$\mathbf{1.69}$	$-1.13$	$\mathbf{-0.34}$
CLIP	2	74.50	$\mathbf{74.68}$	76.13	$\mathbf{76.70}$	1.04	$\mathbf{1.22}$	$-1.21$	$\mathbf{-0.64}$
VLM	1	70.65	$\mathbf{70.65}$	81.72	$\mathbf{81.73}$	0.01	$\mathbf{0.01}$	0.00	$\mathbf{+0.01}$

Table 3: Family-level robustness–preservation trade-off on SNLI-VE.

\Delta T

and

\Delta TI

denote changes from the original TI model in TI-T and TI-TI, respectively.

Architecture	Method / Phase	Wall-clock (s)	Peak Mem. (GiB)	$\Delta T$	$\Delta TI$
Fusion CA	Post-hoc LoRA	1812.30	2.090	+6.02	-1.62
	Dual CE	2592.03	13.954	+13.57	-0.94
	T Reference	1849.09	5.241	–	–
	GU Edit	297.19	1.744	–	–
	GU (Total)	2146.28	5.241	+8.06	-0.11
CLIP MP	Post-hoc LoRA	1910.39	4.423	+1.03	-3.66

Table 4: Offline cost and robustness–preservation trade-off on SNLI-VE. Including reference training, GU uses less time and peak GPU memory than Dual CE.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Dataset	Variant	TI-TI	T-T	TI-T	$\Delta_{\mathrm{miss}}$
Fusion	HM	Concat	$69.46\pm 0.32$	$68.99\pm 0.25$	$68.96\pm 0.44$	$0.50\pm 0.14$
	HM	Gate	$67.98\pm 0.28$	$67.86\pm 0.23$	$67.57\pm 0.38$	$0.41\pm 0.12$
	HM	Cross-Attn	$68.82\pm 0.48$	$68.41\pm 0.40$	$57.63\pm 0.88$	$11.19\pm 0.45$
	IX	Concat	$86.41\pm 0.22$	$81.94\pm 0.18$	$80.70\pm 0.38$	$5.71\pm 0.18$
	IX	Gate	$82.79\pm 0.20$	$81.18\pm 0.17$	$80.76\pm 0.33$	$2.03\pm 0.15$
	IX	Cross-Attn	$84.57\pm 0.28$	$80.97\pm 0.23$	$80.00\pm 0.47$	$4.57\pm 0.22$

Appendix

Table 5: F1 performance under three training–testing regimes. TI-T consistently underperforms T-T across model families and datasets.

Variant	Layer	$\Delta_{\mathrm{harm}}$	$\Delta_{\mathrm{viol}}$
Concat	text_proj_out	0.5187	0.6170
	image_proj_out	0.0000	0.0000
	fusion_mlp0_concat_input	0.5187	0.6170
	fusion_mlp3_output	0.5617	0.7308
	head_last_linear_input	0.5617	0.7308
Gate	text_proj_out	0.3881	0.5735

Appendix

Table 6: Layer-wise task-aware representational harm for fusion models on SNLI-VE. Harm gaps appear at cross-modal interaction stages and often persist near the classifier.

Variant	Layer	Overlap (In / Out)	Mean Angle (In / Out)	Fro. Dist.
Concat	text_proj	0.9666 / 0.9666	8.3333 / 8.3150	1.4884
	image_proj	0.8368 / 0.8409	19.2640 / 18.7282	2.5745
	fusion_mlp0	0.9307 / 0.9314	11.4472 / 11.3110	1.8050
	fusion_mlp3	0.9578 / 0.9573	4.2398 / 4.2895	1.3692
	head_last_linear	0.9991 / 1.0000	1.7072 / 0.0000	0.0511
Gate	text_proj	0.9715 / 0.9714	7.7292 / 7.7325	1.4133

Appendix

Table 7: Layer-wise parameter subspace rotation for fusion models on SNLI-VE. Projection and cross-modal interaction layers generally exhibit larger rotations than the final classifier.

Split	Hateful Memes	SNLI-VE	IU-XRay
Train	8,500	353,482	2,928
Dev	500	11,898	366
Test	1,000	11,937	366

Appendix

Table 8: Dataset split statistics used in our experiments.

Dataset	GU	SMIL Outer	SMIL Inner	Flex-MoE	DyMo
Hateful Memes	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$1.0{\times}10^{-3}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
SNLI-VE	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$1.0{\times}10^{-3}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
IU-XRay	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$1.0{\times}10^{-3}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$

Appendix

Table 9: Learning rates for missing-modality baselines. Unspecified hyperparameters follow the official codebase defaults.

Dataset	Concat	Cross-Attention	Gate
Hateful Memes	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
SNLI-VE	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
IU-XRay	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$

Appendix

Table 10: Learning rates for basic fusion models. A single global learning rate is used for all trainable parameters.

Dataset	Variant	Text/Image	Fusion	Bottleneck/Head
Hateful Memes	Concat	$2.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
	Cross-Attention	$2.0{\times}10^{-5}$	$2.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
	Gate	$1.0{\times}10^{-5}$	$2.0{\times}10^{-5}$	$2.0{\times}10^{-5}$
SNLI-VE	Concat	$2.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
	Cross-Attention	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$
	Gate	$2.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$

Appendix

Table 11: Module-wise learning rates for fusion models with stronger encoders. Text/image denotes both text_encoder.proj and image_encoder.proj ; bottleneck/head denotes both modules.

Dataset	Variant	Text/Image	Fusion	Bottleneck/Head
Hateful Memes	Late Concat	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$
Hateful Memes	Mean Pooling	$1.0{\times}10^{-5}$	–	$1.0{\times}10^{-5}$
SNLI-VE	Late Concat	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$
SNLI-VE	Mean Pooling	$1.0{\times}10^{-5}$	–	$1.0{\times}10^{-5}$
IU-XRay	Late Concat	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$	$1.0{\times}10^{-5}$
IU-XRay	Mean Pooling	$1.0{\times}10^{-5}$	–	$1.0{\times}10^{-5}$

Appendix

Table 12: Module-wise learning rates for CLIP-style two-tower models. Text/image denotes both projection modules; bottleneck/head denotes both modules.

Dataset	Q-Former	Projection	Bottleneck	Head
Hateful Memes	$5.0{\times}10^{-6}$	$5.0{\times}10^{-6}$	$2.0{\times}10^{-5}$	$2.0{\times}10^{-5}$
SNLI-VE	$1.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$
IU-XRay	$1.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$	$3.0{\times}10^{-5}$

Appendix

Table 13: Module-wise learning rates for BLIP-2 based VLM models. Q-Former denotes both vlm_encoder.qformer_embeddings and vlm_encoder.backbone.qformer ; projection denotes both vlm_encoder.backbone.language_projection and vlm_encoder.proj .

Model Family	Variant	Target Layer
Fusion / Fusion (strong)	Concat	fusion_mlp0
Fusion / Fusion (strong)	Gate	gate_mlp0
Fusion / Fusion (strong)	Cross-Attention	attn_v_from_t_in_proj
Two-tower	Late Concat	fusion_mlp0
Two-tower	Mean Pooling	text_proj
VLM	BLIP-2 Classifier Head	qformer_crossattn0_out

Appendix

Table 14: Target layers edited by Geodesic Unlearning. We use

\eta=0.75

and

k=8

for all configurations.

Method	HM	IU-XRay	SNLI-VE
Null-token	$57.63\pm 0.88$	$80.00\pm 0.47$	$66.35\pm 0.38$
SMIL	$56.60\pm 0.74$	$79.21\pm 0.68$	$\underline{72.20}\pm 0.52$
Flex-MoE	$47.58\pm 1.38$	$71.02\pm 0.82$	$70.74\pm 0.64$
DyMo	$48.82\pm 0.93$	$79.19\pm 0.77$	$61.30\pm 0.70$
Interpolation	$\underline{58.50}\pm 0.91$	$\underline{80.14}\pm 0.49$	$67.29\pm 0.61$
GU (Ours)	$\mathbf{60.88}\pm 0.73$	$\mathbf{81.41}\pm 0.63$	$\mathbf{74.41}\pm 0.77$

Appendix

Table 15: Supplementary results for Table 2 . We report mean

\pm

sample standard deviation over three independent runs. Underline and bold denote the second-best and best mean results, respectively.

Metric	Method	Fusion	Fusion (str.)	CLIP	VLM
TI-T	Interp.	$71.34\pm 0.46$	$72.34\pm 0.38$	$74.50\pm 0.44$	$70.65\pm 0.79$
	GU	$\mathbf{73.86}\pm 0.56$	$\mathbf{73.06}\pm 0.43$	$\mathbf{74.68}\pm 0.52$	$\mathbf{70.65}\pm 0.68$
TI-TI	Interp.	$86.33\pm 0.18$	$75.16\pm 0.37$	$76.13\pm 0.38$	$81.72\pm 0.30$
	GU	$\mathbf{86.37}\pm 0.14$	$\mathbf{75.95}\pm 0.31$	$\mathbf{76.70}\pm 0.31$	$\mathbf{81.73}\pm 0.18$
$\Delta$ T	Interp.	$0.98\pm 0.30$	$0.96\pm 0.22$	$1.04\pm 0.35$	$0.01\pm 0.10$
	GU	$\mathbf{3.49}\pm 0.26$	$\mathbf{1.69}\pm 0.20$	$\mathbf{1.22}\pm 0.33$	$\mathbf{0.01}\pm 0.04$

Appendix

Table 16: Supplementary robustness–preservation results for Table 3 on SNLI-VE. Results are mean

\pm

sample standard deviation over three independent runs.

Dataset	Comparison	10-seed paired $\Delta$ F1	GU wins	Raw $p$	Holm $p$
HM	GU $-$ Null	$+3.19\pm 1.14$	9/10	0.003906	0.011719
HM	GU $-$ Interp.	$+2.41\pm 0.77$	10/10	0.001953	0.011719
IX	GU $-$ Null	$+1.51\pm 1.71$	9/10	0.033203	0.033203
IX	GU $-$ Interp.	$+1.34\pm 0.45$	10/10	0.001953	0.011719
SV	GU $-$ Null	$+8.09\pm 0.22$	10/10	0.001953	0.011719
SV	GU $-$ SMIL	$+2.21\pm 0.68$	10/10	0.001953	0.011719

Appendix

Table 17: Ten-seed paired tests for GU against Null and the strongest competing method in Table 2 . All six comparisons remain significant after Holm correction.

Method	TI-T	CKA to orig.	Error-margin gain	Corr./regr. rate (%)
Orig. TI-T	$66.35\pm 0.38$	$1.000$	$0$	$0/0$
Interp.	$67.29\pm 0.61$	$0.988\pm 0.001$	$+0.06\pm 0.01$	$11.8\pm 0.6\,/\,4.5\pm 0.2$
GU (Ours)	$74.41\pm 0.77$	$0.991\pm 0.001$	$+0.30\pm 0.02$	$34.0\pm 1.3\,/\,5.1\pm 0.3$

Appendix

Table 18: Quantitative analysis of Figure 4 (a) on SNLI-VE Fusion Cross-Attention. Both edits preserve high representation similarity, while GU yields larger task-level improvements.

Family	Variant	Target Layer	Method	TI-T	TI-TI	$\Delta T$	$\Delta TI$
Fusion	Concat	fusion_mlp0	Interp.	$74.86\pm.68$	$86.10\pm.16$	$1.82\pm.48$	$-0.12\pm.10$
	Concat	fusion_mlp0	GU	$\mathbf{75.11}\pm.56$	$\mathbf{86.14}\pm.12$	$\mathbf{2.07}\pm.32$	$\mathbf{-0.08}\pm.05$
	Gate	gate_mlp0	Interp.	$71.88\pm.47$	$86.40\pm.18$	$0.18\pm.24$	$-0.13\pm.11$
	Gate	gate_mlp0	GU	$\mathbf{72.05}\pm.35$	$\mathbf{86.44}\pm.12$	$\mathbf{0.35}\pm.07$	$\mathbf{-0.09}\pm.04$
	Cross-Attn	attn_v $\leftarrow$ t_in	Interp.	$67.29\pm.61$	$86.50\pm.24$	$0.94\pm.29$	$-0.14\pm.15$
	Cross-Attn	attn_v $\leftarrow$ t_in	GU	$\mathbf{74.41}\pm.77$	$\mathbf{86.53}\pm.17$	$\mathbf{8.06}\pm.40$	$\mathbf{-0.11}\pm.08$

Appendix

Table 19: Per-variant robustness–preservation results on SNLI-VE. GU generally provides stronger recovery with comparable or better full-modality preservation than interpolation.

Hateful Memes
Method	Fusion Concat	CLIP Late Concat	CLIP Mean Pooling
Null-token	$68.96\pm 0.44\,/\,69.46\pm 0.32$	$66.58\pm 0.41\,/\,67.31\pm 0.27$	$66.87\pm 0.24\,/\,67.14\pm 0.20$
MoRA	– / –	$65.64\pm 0.28\,/\,65.39\pm 0.83$	$60.45\pm 6.17\,/\,64.98\pm 0.87$
OGM-GE	$66.71\pm 0.56\,/\,67.07\pm 0.78$	$66.07\pm 0.47\,/\,65.85\pm 0.41$	$64.51\pm 0.66\,/\,65.68\pm 1.10$
GU	$69.01\pm 0.24\,/\,69.32\pm 0.30$	$67.03\pm 0.17\,/\,67.26\pm 0.31$	$66.96\pm 0.34\,/\,67.09\pm 0.22$
GU $\Delta$ TI	$-0.14$	$-0.05$	$-0.05$

Appendix

Table 20: Cross-architecture comparison with MoRA and OGM-GE. GU achieves the highest mean TI-T in all nine evaluated settings. Entries report TI-T / TI-TI F1.

\Delta

TI denotes GU’s mean TI-TI change relative to Null-token (GU minus Null-token).

Architecture	Method	TI-T	TI-TI
Fusion CA	Null-token	$66.35\pm 0.38$	$86.64\pm 0.10$
	LoRA	$72.37\pm 0.25$	$85.02\pm 0.35$
	Dual CE	$79.92\pm 0.23$	$85.70\pm 0.22$
	GU	$74.41\pm 0.77$	$86.53\pm 0.17$
	Dual CE+GU	$80.09\pm 0.30$	$86.51\pm 0.19$
CLIP MP	Null-token	$74.15\pm 0.18$	$76.62\pm 0.06$

Appendix

Table 21: Comparison with post-hoc LoRA and Dual CE on SNLI-VE. Entries report F1 as mean

\pm

sample standard deviation over three independent runs.

	Angle only	Harm only	Angle+harm
Selected layer	attn_t_from_v_out_proj	text_proj	attn_v_from_t_in_proj
TI-T	$80.34\pm 0.66$	$80.72\pm 0.40$	$\mathbf{81.41}\pm 0.63$
TI-TI	$83.96\pm 0.37$	$84.21\pm 0.32$	$\mathbf{84.44}\pm 0.28$
$\Delta T$	$+0.34$	$+0.72$	$\mathbf{+1.41}$
$\Delta TI$	$-0.61$	$-0.36$	$\mathbf{-0.13}$

Appendix

Table 22: Target-layer selection on IU-XRay Fusion Cross-Attention. Joint angle–harm selection yields the strongest recovery and preservation.

Dataset	Weight	Linear	GU (Ours)
HM	$58.50\pm 0.91\,/\,66.52\pm 0.33$	$58.07\pm 1.03\,/\,66.62\pm 0.21$	$\mathbf{60.88}\pm 0.73\,/\,\mathbf{67.55}\pm 0.26$
IX	$80.14\pm 0.49\,/\,82.67\pm 0.56$	$79.83\pm 0.75\,/\,82.39\pm 0.36$	$\mathbf{81.41}\pm 0.63\,/\,\mathbf{84.44}\pm 0.28$
SV	$67.29\pm 0.61\,/\,86.50\pm 0.24$	$66.84\pm 1.29\,/\,86.50\pm 0.07$	$\mathbf{74.41}\pm 0.77\,/\,\mathbf{86.53}\pm 0.17$

Appendix

Table 23: Controlled comparison of editing strategies on Fusion Cross-Attention. GU achieves the highest mean TI-T and TI-TI across all three datasets. Entries report TI-T / TI-TI F1.

Method	TI-T Acc.	TI-TI Acc.
T Refer. (T-T)	$47.17\pm 0.31$	–
Shuffle-I	–	$39.85\pm 1.97$
Orig. TI	$38.94\pm 1.92$	$50.79\pm 0.71$
Interp.	$39.80\pm 1.21$	$43.21\pm 1.44$
LoRA	$\mathbf{43.52}\pm 0.60$	$45.01\pm 1.72$
GU (Ours)	$41.07\pm 1.79$	$\mathbf{47.40}\pm 1.19$

Appendix

Table 24: A-OKVQA multiple-choice accuracy (%) over ten runs. GU provides the strongest full-modality preservation among the evaluated interventions. Bold marks the best result among Interpolation, LoRA, and GU.

Method	Eval.	HM	IU-XRay	SNLI-VE
Orig. TI	TI-TI	$68.82\pm 0.48$	$84.57\pm 0.28$	$86.64\pm 0.10$
I Refer.	I-I	$66.58\pm 0.15$	$75.32\pm 0.41$	$68.74\pm 0.00$
Orig. TI	TI-I	$63.32\pm 0.88$	$73.58\pm 0.85$	$64.48\pm 0.97$
Interp.	TI-I	$64.10\pm 0.24$	$73.70\pm 0.76$	$65.87\pm 0.46$
Interp.	TI-TI	$68.30\pm 0.28$	$84.39\pm 0.36$	$86.01\pm 0.15$
GU (Ours)	TI-I	$64.34\pm 0.20$	$74.42\pm 0.22$	$66.61\pm 0.27$

Appendix

Table 25: Missing-text evaluation with Fusion Cross-Attention. GU improves TI-I while largely preserving full-modality performance across all three datasets.

Explore similar work

Unsupervised Learning for Missing Modalities in Multimodal Learning
Jun 14, 2026Hassan Ismkhan, Hamid BouchahciaMissing-Modality Learning Unsupervised Learning
Bournemouth University Bournemouth, UK
General Incomplete Multimodal Learning via Dynamic Quality Perception
Jul 8, 2026Xiangyu Meng, Shicai WeiMissing-Modality Learning Multimodal Robustness
University of Electronic Science and Technology of China
ModalImmune: Immunity Driven Unlearning via Self Destructive Training
Feb 18, 2026Rong Fu, WeiZhi Tang, Ziming Wang +6Multimodal Robustness Missing-Modality Learning
University of Macau · Zhejiang University · Renmin University of China +3

Jun 14, 2026J/K move · Enter open · S save

Unsupervised Learning for Missing Modalities in Multimodal Learning

Hassan Ismkhan, Hamid Bouchahcia

Bournemouth University Bournemouth, UK

Missing-Modality Learning Unsupervised Learning Learning with Missing Data Multimodal Learning

Open paper AskPDF arXiv

This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction. We propose modality-specific normalization and a novel partial-modality distance metric to enable fair clustering of incomplete observations, capturing cross-modal structures while preserving scale-invariance across varying dimensionalities and modality counts. Cluster centers from this unsupervised stage guide an iterative greedy imputation process for any missing modalities during training or inference, supporting arbitrary numbers of modalities and arbitrary missing patterns per sample. The imputation module is lightweight, uses frozen encoders, and decouples from the downstream task, allowing easy integration with any fusion/prediction architecture. Extensive experiments under diverse and highly incomplete regimes demonstrate UL4M4's robustness, achieving, to the best of our knowledge, the first consistent F1-Micro scores above 0.7 on challenging missing configurations even when more than 50% of modality slots are missing. Results are also stable across cluster sizes and significantly outperform state-of-the-art baselines. Code is available here: https://github.com/h-ismkhan/Multimodal-Learning-with-Missing-Modalities-via-Unsupervised-Learning.

Abstract

Explore similar work

Unsupervised Learning for Missing Modalities in Multimodal Learning

General Incomplete Multimodal Learning via Dynamic Quality Perception

ModalImmune: Immunity Driven Unlearning via Self Destructive Training