TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning

Liang Nathan Zheng; Jiyuan Liu; Wenhao Liang; Wei Emma Zhang; Weitong Chen; Lin Yue

cs.LGOct 15, 2026

TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning

Authors: Liang Nathan Zheng, Jiyuan Liu, Wenhao Liang, Wei Emma Zhang, Weitong Chen, Lin Yue

Organizations: School of Computer Science and Information Technology, Adelaide University, Adelaide, SA 5000, Australia

Abstract

Recent Mixture-of-Experts (MoE) approaches address missing modalities either by routing each missingness pattern to a dedicated expert, which scales poorly as the number of modalities grows, or by imputing the missing modality from representations contributed by multiple experts, which requires a costly retrieval mechanism. We show that routing induces expert-conditioned representation neighborhoods and derive an imputation-error bound controlled by their local feature dispersion. When these expert-conditioned neighborhoods are more compact than the global representation space, intra-expert aggregation admits a tighter worst-case reconstruction bound than unconstrained global aggregation. Guided by this analysis, we propose TWIN, a two-stage intra-expert imputation framework. Pre-imputation generates a context-conditioned estimate in the encoder embedding space and radially soft-clips it using modality-specific statistics maintained in an observed FIFO queue, this ensures that the pre-imputed modality stays in empirically calibrated region of the target modality space. Post-imputation refines the missing representation after MoE by aggregating expert-conditioned observed representations within the same sample. Across four missing-modality benchmarks under natural, random, and asymmetric dropout protocols, TWIN achieves state-of-the-art performance while using substantially fewer parameters than competing MoE-based imputation methods.

Figures & tables

Figure 1: Illustration of the intuition behind TWIN-MoE.

Figure 2: Overview of Proposed Method: (1) Starting at the encoding of modalities, where each modality will be assigned a unique encoder and implement pre-imputation if any modality is needed. Pre-imputation learns the imputed coefficient derived from existing modalities to impute the missing modality from its modality distribution. (2) Inspired by the theoretical motivation, post-imputation conducted intra-expert imputation before each token is forwarded through the expert. An SWD loss is applied to enforce the semantic alignment of tokens in the slots.

Task	Metric	SMIL	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
48-IHM	F1	40.36 $\pm$ 2.10	38.06 $\pm$ 0.51	11.55 $\pm$ 0.65	45.08 $\pm$ 0.71	41.34 $\pm$ 0.33	42.70 $\pm$ 0.61	41.04 $\pm$ 0.96	44.12 $\pm$ 0.33	46.95 $\pm$ 0.81	49.82 $\pm$ 0.82
48-IHM	AUC	81.25 $\pm$ 0.87	79.16 $\pm$ 0.15	72.92 $\pm$ 0.55	83.96 $\pm$ 0.10	82.93 $\pm$ 0.76	81.82 $\pm$ 1.02	83.17 $\pm$ 0.46	83.53 $\pm$ 0.20	85.11 $\pm$ 0.38	85.56 $\pm$ 0.11
LOS	F1	64.87 $\pm$ 0.46	37.97 $\pm$ 2.18	49.87 $\pm$ 1.22	62.67 $\pm$ 0.89	62.64 $\pm$ 0.15	62.76 $\pm$ 0.13	61.26 $\pm$ 0.72	63.72 $\pm$ 0.24	65.93 $\pm$ 0.30	66.42 $\pm$ 0.49
LOS	AUC	79.80 $\pm$ 0.56	77.17 $\pm$ 0.30	69.94 $\pm$ 0.57	79.71 $\pm$ 0.21	78.35 $\pm$ 0.33	78.31 $\pm$ 0.45	77.23 $\pm$ 0.69	80.27 $\pm$ 0.82	81.66 $\pm$ 0.38	81.49 $\pm$ 0.32
25-PHE	F1	28.70 $\pm$ 0.90	19.77 $\pm$ 0.72	5.19 $\pm$ 0.35	34.41 $\pm$ 0.28	31.16 $\pm$ 0.20	32.53 $\pm$ 0.90	36.61 $\pm$ 0.77	36.74 $\pm$ 1.59	38.63 $\pm$ 0.57	38.11 $\pm$ 0.45
25-PHE	AUC	75.75 $\pm$ 0.26	71.66 $\pm$ 1.33	66.00 $\pm$ 0.12	78.19 $\pm$ 0.41	78.78 $\pm$ 0.16	78.55 $\pm$ 0.10	78.01 $\pm$ 0.11	78.52 $\pm$ 0.49	80.20 $\pm$ 0.10	79.81 $\pm$ 0.29

Table 1: Experiment setting I: Main results on MIMIC-III

Missing Rate	Metric	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
0%	F1	47.48 $\pm$ 0.27	38.89 $\pm$ 0.54	59.13 $\pm$ 0.52	57.04 $\pm$ 0.15	58.71 $\pm$ 0.23	60.62 $\pm$ 0.96	61.31 $\pm$ 0.14	61.75 $\pm$ 0.13	62.17 $\pm$ 0.44
0%	AUC	67.93 $\pm$ 0.56	64.26 $\pm$ 0.61	80.29 $\pm$ 0.31	78.54 $\pm$ 0.42	78.13 $\pm$ 0.34	81.34 $\pm$ 0.20	82.87 $\pm$ 0.10	82.68 $\pm$ 0.16	83.48 $\pm$ 0.28
10%	F1	47.41 $\pm$ 0.18	34.94 $\pm$ 1.10	57.46 $\pm$ 1.10	55.90 $\pm$ 0.55	56.68 $\pm$ 0.31	58.66 $\pm$ 1.10	59.81 $\pm$ 0.34	59.40 $\pm$ 0.66	60.65 $\pm$ 0.22
10%	AUC	69.26 $\pm$ 0.22	61.05 $\pm$ 0.34	79.38 $\pm$ 0.94	76.38 $\pm$ 0.39	76.44 $\pm$ 0.12	79.48 $\pm$ 0.23	81.00 $\pm$ 0.10	80.78 $\pm$ 0.15	81.90 $\pm$ 0.05
20%	F1	46.54 $\pm$ 1.04	35.37 $\pm$ 0.94	55.35 $\pm$ 0.88	54.44 $\pm$ 0.27	54.60 $\pm$ 0.19	55.96 $\pm$ 1.67	58.06 $\pm$ 0.21	56.81 $\pm$ 0.60	58.29 $\pm$ 0.64
20%	AUC	68.05 $\pm$ 0.56	59.54 $\pm$ 0.22	77.29 $\pm$ 0.47	74.32 $\pm$ 0.16	74.92 $\pm$ 0.64	78.05 $\pm$ 0.10	78.85 $\pm$ 0.28	78.63 $\pm$ 0.28	78.73 $\pm$ 0.21

Table 2: Experiment setting II: Main results on CMU-MOSEI

Figure 3: Imputation Error of different post-imputation method. We take two post imputation method cross-attention and average, and compare their combination of inter- and intra-expert imputation.

Post Imput.	Sparse MoE		Soft MoE
Post Imput.	Average	Cross-Attention	Average	Cross-Attention
Inter-Expert	45.37 $\pm$ 2.09	45.41 $\pm$ 2.20	44.59 $\pm$ 2.37	44.55 $\pm$ 2.39
Intra-Expert	46.18 $\pm$ 3.23	46.26 $\pm$ 2.97	45.98 $\pm$ 1.70	46.35 $\pm$ 0.66

Table 3: F1 score comparison of different backbones and imputation strategy on CMU-MOSI (Experiment Setting II, missing ratio

0.5

)

Variants	Metrics	Sparse MoE				Soft MoE
Variants	Metrics	w/o pre	w/o post	w/o impute.	TWIN	w/o pre	w/o post	w/o impute.	TWIN
MIMIC-III	F1	48.24 $\pm$ 0.38	48.65 $\pm$ 0.25	44.90 $\pm$ 1.61	49.82 $\pm$ 0.82	46.28 $\pm$ 0.99	46.01 $\pm$ 0.60	44.78 $\pm$ 1.09	46.95 $\pm$ 0.81
MIMIC-III	AUC	84.37 $\pm$ 0.26	85.38 $\pm$ 0.28	84.30 $\pm$ 0.84	85.56 $\pm$ 0.11	84.06 $\pm$ 0.75	83.89 $\pm$ 0.39	83.90 $\pm$ 0.26	85.11 $\pm$ 0.38
CMU-MOSI	F1	45.59 $\pm$ 1.83	45.01 $\pm$ 2.03	45.01 $\pm$ 0.16	46.62 $\pm$ 4.43	44.42 $\pm$ 2.02	45.25 $\pm$ 3.33	44.03 $\pm$ 1.89	44.29 $\pm$ 1.92
CMU-MOSI	AUC	69.94 $\pm$ 0.75	70.30 $\pm$ 1.52	69.39 $\pm$ 0.31	70.50 $\pm$ 1.30	69.59 $\pm$ 0.45	70.47 $\pm$ 0.39	67.86 $\pm$ 0.80	71.03 $\pm$ 2.43

Table 4: Module Ablation on MIMIC-III IHM task and CMU-MOSI (Experiment Setting II, missing ratio

0.5

)

Scale	Metrics	ShaSpec	TF	LlMoE	FuseMoE	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
Small	MFLOPs	21.73	39.69	1245.03	30.18	40.32	39.62	39.34	39.88
	#Params	2,177,406	9,921,903	2,130,227	9,636,174	2,184,963	2,184,730	335,691	336.376
	F1 score	39.12 $\pm$ 1.21	35.81 $\pm$ 0.85	41.11 $\pm$ 0.90	33.85 $\pm$ 0.76	42.12 $\pm$ 1.18	44.34 $\pm$ 0.77	43.69 $\pm$ 0.60	44.72 $\pm$ 0.73
Medium	MFLOPs	111.27	263.53	6803,84	286.02	263.94	268.62	267.04	267.45
	#Params	11,137,046	65,878,703	11,469,437	11,524,392	11,371,473	11,330,062	2,789,275	2,794,759
	F1 score	41.56 $\pm$ 0.67	38.62 $\pm$ 0.38	43.21 $\pm$ 0.56	39.44 $\pm$ 0.81	43.78 $\pm$ 0.36	45.37 $\pm$ 1.73	44.88 $\pm$ 1.10	45.78 $\pm$ 2.22

Table 5: Control Complexity Study: We control the complexity of the model to evaluate performance

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Task	#Samples	Modality	#Classes	Is Imbalanced?	Missing Ratio
MIMIC-III	48-IHM	21,139	Time Series, Text	2	Yes	23.60%
	LOS	21,139	Time Series, Text	2	Yes	23.60%
	25-PHE	41,902	Time Series, Text	25	Yes	23.70%
MIMIC-IV	48-IHM	22,733	Image, ECG, Time Series, Text	2	Yes	47.10%
	LOS	22,733	Image, ECG, Time Series, Text	2	Yes	47.10%
	25-PHE	47,027	Image, ECG, Time Series, Text	25	Yes	50.32%

Appendix

Table 6: Dataset statistics

Parameter	Value
#Expert	8
#Top-K Expert	1
Learning Rate	3e-4
Hidden Size	128
#MoE layers	1
Training Epoch	50

Appendix

Table 7: Hyperparameter setting

Figure 4: Ablation study on parameter sensitivity

Task	Metric	SMIL	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
48-IHM	F1	39.58 $\pm$ 1.12	32.97 $\pm$ 1.08	11.60 $\pm$ 0.54	43.76 $\pm$ 0.44	30.14 $\pm$ 1.15	40.21 $\pm$ 1.29	35.29 $\pm$ 0.30	43.36 $\pm$ 0.94	47.05 $\pm$ 0.48	45.42 $\pm$ 0.39
48-IHM	AUC	76.60 $\pm$ 0.98	78.29 $\pm$ 0.25	67.08 $\pm$ 0.77	82.97 $\pm$ 0.61	71.34 $\pm$ 0.88	78.05 $\pm$ 0.72	80.45 $\pm$ 0.44	84.50 $\pm$ 0.33	85.55 $\pm$ 0.41	84.38 $\pm$ 0.34
LOS	F1	58.52 $\pm$ 0.44	56.22 $\pm$ 0.28	42.01 $\pm$ 1.12	59.03 $\pm$ 0.39	57.59 $\pm$ 0.76	58.31 $\pm$ 0.46	56.96 $\pm$ 0.53	60.82 $\pm$ 0.56	61.61 $\pm$ 0.21	61.14 $\pm$ 0.33
LOS	AUC	75.86 $\pm$ 0.66	72.22 $\pm$ 0.18	64.11 $\pm$ 0.40	76.17 $\pm$ 0.17	72.59 $\pm$ 1.07	72.59 $\pm$ 0.61	74.81 $\pm$ 0.37	76.80 $\pm$ 0.36	77.30 $\pm$ 0.78	77.98 $\pm$ 0.48
25-PHE	F1	27.77 $\pm$ 0.91	25.53 $\pm$ 0.11	26.52 $\pm$ 0.29	25.38 $\pm$ 0.52	12.45 $\pm$ 0.54	12.25 $\pm$ 0.55	24.61 $\pm$ 0.71	26.84 $\pm$ 0.56	28.52 $\pm$ 0.67	28.51 $\pm$ 0.47
25-PHE	AUC	63.90 $\pm$ 1.31	62.57 $\pm$ 0.23	56.28 $\pm$ 0.16	72.50 $\pm$ 0.67	55.48 $\pm$ 0.39	58.52 $\pm$ 0.27	71.57 $\pm$ 0.47	74.73 $\pm$ 0.71	74.82 $\pm$ 0.77	74.49 $\pm$ 0.98

Appendix

Table 8: Experiment setting I: Main results on MIMIC-IV

Missing Rate	Metric	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
0%	F1	41.78 $\pm$ 0.64	43.28 $\pm$ 0.36	50.91 $\pm$ 0.19	47.69 $\pm$ 0.39	48.15 $\pm$ 0.57	51.50 $\pm$ 0.55	51.83 $\pm$ 0.27	53.35 $\pm$ 0.10	53.61 $\pm$ 0.39
0%	AUC	65.08 $\pm$ 0.39	64.48 $\pm$ 0.51	78.12 $\pm$ 0.31	75.26 $\pm$ 0.20	75.96 $\pm$ 0.29	79.38 $\pm$ 0.23	80.56 $\pm$ 0.10	81.23 $\pm$ 0.36	80.64 $\pm$ 0.10
10%	F1	43.40 $\pm$ 0.74	40.14 $\pm$ 0.13	49.91 $\pm$ 0.33	46.81 $\pm$ 0.45	48.81 $\pm$ 0.42	49.92 $\pm$ 0.28	49.99 $\pm$ 0.85	51.66 $\pm$ 0.85	51.73 $\pm$ 1.40
10%	AUC	65.45 $\pm$ 0.56	61.54 $\pm$ 0.66	77.12 $\pm$ 0.14	70.25 $\pm$ 0.93	72.36 $\pm$ 0.50	77.07 $\pm$ 0.21	78.06 $\pm$ 0.64	79.60 $\pm$ 0.40	78.40 $\pm$ 0.77
20%	F1	39.15 $\pm$ 0.13	42.64 $\pm$ 0.30	48.84 $\pm$ 0.61	47.92 $\pm$ 0.58	45.00 $\pm$ 0.44	49.16 $\pm$ 0.24	50.58 $\pm$ 0.73	51.17 $\pm$ 0.42	50.66 $\pm$ 0.84
20%	AUC	60.87 $\pm$ 0.29	65.52 $\pm$ 0.98	76.64 $\pm$ 0.23	71.05 $\pm$ 0.36	72.15 $\pm$ 0.37	73.80 $\pm$ 1.43	77.17 $\pm$ 0.55	78.26 $\pm$ 0.49	76.33 $\pm$ 0.82

Appendix

Table 9: Experiment setting II: Main results on CMU-MOSI

Test Modality			Dataset: CMU-MOSI / Evaluation Metric: AUC
Video	Text	Audio	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
✓	✓		62.18 $\pm$ 0.88	59.37 $\pm$ 1.12	74.63 $\pm$ 0.13	55.35 $\pm$ 1.11	56.44 $\pm$ 1.48	74.59 $\pm$ 1.50	75.78 $\pm$ 1.03	78.75 $\pm$ 0.25	77.10 $\pm$ 0.57
✓		✓	55.79 $\pm$ 2.53	57.87 $\pm$ 0.32	56.84 $\pm$ 0.34	56.24 $\pm$ 1.96	55.11 $\pm$ 0.77	52.28 $\pm$ 1.23	57.61 $\pm$ 0.62	69.90 $\pm$ 1.03	59.06 $\pm$ 0.89
	✓	✓	65.22 $\pm$ 1.01	57.89 $\pm$ 0.65	75.59 $\pm$ 0.39	54.36 $\pm$ 1.67	56.10 $\pm$ 1.76	75.55 $\pm$ 1.64	76.53 $\pm$ 0.74	77.87 $\pm$ 0.46	77.69 $\pm$ 0.22
✓			55.10 $\pm$ 1.59	58.57 $\pm$ 0.44	74.73 $\pm$ 0.51	55.94 $\pm$ 1.51	55.66 $\pm$ 1.66	74.35 $\pm$ 1.60	75.06 $\pm$ 0.22	77.76 $\pm$ 0.46	77.06 $\pm$ 0.65
	✓		61.60 $\pm$ 4.80	60.28 $\pm$ 1.42	75.73 $\pm$ 0.33	55.45 $\pm$ 0.95	56.58 $\pm$ 1.29	74.42 $\pm$ 1.39	74.88 $\pm$ 0.46	75.59 $\pm$ 0.81	77.05 $\pm$ 0.72

Appendix

Table 10: Experiment setting III: Main results on CMU-MOSI

Test Modality			Dataset: CMU-MOSI / Evaluation Metric: F1
Video	Text	Audio	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
✓	✓		41.27 $\pm$ 0.81	34.96 $\pm$ 0.72	44.45 $\pm$ 0.33	30.51 $\pm$ 2.53	29.90 $\pm$ 2.58	50.54 $\pm$ 0.73	50.26 $\pm$ 0.28	51.35 $\pm$ 0.51	53.17 $\pm$ 0.64
✓		✓	36.99 $\pm$ 2.03	34.83 $\pm$ 0.54	35.20 $\pm$ 0.64	24.29 $\pm$ 4.23	27.89 $\pm$ 6.23	30.02 $\pm$ 7.82	37.37 $\pm$ 1.00	45.10 $\pm$ 2.82	37.99 $\pm$ 0.33
	✓	✓	43.72 $\pm$ 1.64	37.25 $\pm$ 0.53	47.79 $\pm$ 0.46	20.68 $\pm$ 2.14	29.88 $\pm$ 4.55	50.41 $\pm$ 1.55	50.77 $\pm$ 0.48	51.11 $\pm$ 0.28	53.86 $\pm$ 3.42
✓			42.98 $\pm$ 0.21	38.11 $\pm$ 0.35	44.86 $\pm$ 0.94	25.67 $\pm$ 2.82	27.34 $\pm$ 2.94	50.39 $\pm$ 0.87	50.41 $\pm$ 0.24	51.55 $\pm$ 0.32	53.10 $\pm$ 0.65
	✓		41.36 $\pm$ 2.27	37.99 $\pm$ 0.40	43.81 $\pm$ 0.42	25.70 $\pm$ 2.82	27.30 $\pm$ 2.95	50.65 $\pm$ 0.90	50.35 $\pm$ 0.18	50.72 $\pm$ 0.34	52.78 $\pm$ 1.92

Appendix

Table 11: Experiment setting III: Main results on CMU-MOSI

Test Modality			Dataset: CMU-MOSEI / Evaluation Metric: AUC
Video	Text	Audio	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
✓	✓		64.47 $\pm$ 0.10	56.98 $\pm$ 0.20	81.23 $\pm$ 0.07	75.81 $\pm$ 0.82	76.62 $\pm$ 0.37	80.63 $\pm$ 0.41	81.52 $\pm$ 0.21	81.67 $\pm$ 0.61	81.50 $\pm$ 0.17
✓		✓	58.41 $\pm$ 0.10	58.41 $\pm$ 0.17	59.64 $\pm$ 0.56	57.32 $\pm$ 0.62	57.22 $\pm$ 0.12	58.28 $\pm$ 1.51	60.07 $\pm$ 0.22	60.53 $\pm$ 1.15	60.64 $\pm$ 0.17
	✓	✓	64.46 $\pm$ 0.14	58.64 $\pm$ 0.13	80.00 $\pm$ 0.29	77.24 $\pm$ 0.60	77.31 $\pm$ 0.16	80.00 $\pm$ 0.10	81.26 $\pm$ 0.10	81.24 $\pm$ 0.13	81.23 $\pm$ 0.16
✓			63.52 $\pm$ 0.19	58.71 $\pm$ 0.35	80.69 $\pm$ 0.34	76.84 $\pm$ 0.14	76.27 $\pm$ 0.70	81.06 $\pm$ 0.20	81.53 $\pm$ 0.02	81.62 $\pm$ 0.37	81.51 $\pm$ 0.16
	✓		64.40 $\pm$ 0.24	58.89 $\pm$ 0.40	80.96 $\pm$ 0.26	76.85 $\pm$ 0.14	76.51 $\pm$ 0.50	80.48 $\pm$ 0.21	81.50 $\pm$ 0.16	81.67 $\pm$ 0.45	81.61 $\pm$ 0.17

Appendix

Table 12: Experiment setting III: Main results on CMU-MOSEI

Test Modality			Dataset: CMU-MOSEI / Evaluation Metric: F1
Video	Text	Audio	ShaSpec	TF	LlMoE	FuseMoE-S	FuseMoE-L	FlexMoE	ConfSMoE	TWIN-SF	TWIN-SP
✓	✓		40.75 $\pm$ 0.14	31.38 $\pm$ 0.61	58.97 $\pm$ 0.53	55.85 $\pm$ 0.69	53.59 $\pm$ 1.07	60.10 $\pm$ 0.76	61.01 $\pm$ 0.58	61.97 $\pm$ 0.54	61.06 $\pm$ 1.00
✓		✓	38.15 $\pm$ 0.92	31.73 $\pm$ 0.16	38.26 $\pm$ 2.40	29.40 $\pm$ 1.67	33.29 $\pm$ 2.89	36.28 $\pm$ 3.51	39.20 $\pm$ 0.93	40.64 $\pm$ 0.14	40.18 $\pm$ 0.11
	✓	✓	40.82 $\pm$ 0.12	31.86 $\pm$ 0.23	58.58 $\pm$ 0.31	57.02 $\pm$ 1.60	53.38 $\pm$ 0.65	58.70 $\pm$ 2.10	61.37 $\pm$ 0.28	61.98 $\pm$ 0.66	61.54 $\pm$ 0.57
✓			40.37 $\pm$ 0.46	31.59 $\pm$ 0.23	61.20 $\pm$ 0.46	54.02 $\pm$ 2.88	53.08 $\pm$ 0.40	58.58 $\pm$ 0.71	61.71 $\pm$ 1.00	61.05 $\pm$ 1.40	61.05 $\pm$ 1.00
	✓		42.16 $\pm$ 0.22	30.84 $\pm$ 0.26	59.35 $\pm$ 0.12	55.87 $\pm$ 0.68	54.11 $\pm$ 0.49	58.42 $\pm$ 0.27	61.34 $\pm$ 0.83	61.97 $\pm$ 0.54	61.00 $\pm$ 0.97

Appendix

Table 13: Experiment setting III: Main results on CMU-MOSEI

$\mathcal{B}_{m}$	AUC	F1
arcsinh	$69.61\pm 2.12$	$45.95\pm 0.97$
tanh	$69.52\pm 2.02$	$45.76\pm 1.18$
softsign	$69.63\pm 1.72$	$46.19\pm 1.61$
gated	$70.28\pm 1.47$	$44.24\pm 1.10$
Ours	$\mathbf{70.50\pm 1.30}$	$\mathbf{46.62\pm 4.43}$

Appendix

Table 14: Comparison of pre-imputation coefficient operator

\mathcal{B}_{m}

on CMU-MOSI (Experiment Setting II, missing ratio

0.5

). The coefficient

\Delta_{m}

is plugged into

\hat{h}_{m}^{pre}=\mu_{m}+\sigma_{m}\odot\Delta_{m}

Figure 5: Embedding space of MoE has been split into different clusters

Missing-Modality Learning

Missing-modality learning develops multimodal models that remain useful when input streams are absent, corrupted, or available only during training.

See all 170 papers

Residual-Guided Expert Specialization for Incomplete Multimodal Learning
Jun 29, 2026Seunghun Baek, Jihwan Park, Jaeyoon Sim +3Missing-Modality Learning Multimodal Robustness
Pohang University of Science and Technology (POSTECH), South Korea
Unsupervised Learning for Missing Modalities in Multimodal Learning
Jun 14, 2026Hassan Ismkhan, Hamid BouchahciaMissing-Modality Learning Unsupervised Learning
Bournemouth University Bournemouth, UK
Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
May 22, 2026Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel +2Efficient Multimodal Inference Cross-Modal Representation Learning
Adelaide University

Jun 29, 2026J/K move · Enter open · S save

Residual-Guided Expert Specialization for Incomplete Multimodal Learning

Seunghun Baek, Jihwan Park, Jaeyoon Sim, Minjae Jeong+2

Pohang University of Science and Technology (POSTECH), South Korea

Missing-Modality Learning Multimodal Robustness Learning with Missing Data Multimodal Classification Mixture-of-Experts Models

Open paper AskPDF arXiv

As real-world prediction systems often face missing modalities at inference, incomplete multimodal learning (IML) remains a practical challenge. While prior methods aim to learn representations robust to missing inputs, representations from incomplete modalities inevitably deviate from their full-modality counterparts due to missing evidence. To explicitly leverage these deviations, we propose MARS (Missingness-Aware Residual-guided Specialization), a mixture-of-experts framework that guides expert specialization based on how representations are reshaped by missingness. By contrasting task representations derived from incomplete inputs with their complete counterparts during training, we derive a privileged residual signal that captures this representational gap. The residual signal guides a residual router to assign samples to experts specialized for the corresponding deviation patterns. In parallel, a feature router learns to imitate this routing behavior using only incomplete inputs, enabling deployment without access to full modalities. To mitigate this train-test router gap, we develop a discrepancy-aware noise regularization that adaptively perturbs the residual router's decisions when the feature router deviates, enhancing expert robustness under imperfect imitation. Experiments on multimodal classification (CASIA-SURF, CREMA-D, UPMC Food-101) and segmentation (MCubeS) under missing scenarios show that MARS consistently surpasses baselines while remaining efficient and extensible to diverse backbones and tasks.

Abstract

Missing-Modality Learning

Explore similar work

Residual-Guided Expert Specialization for Incomplete Multimodal Learning

Unsupervised Learning for Missing Modalities in Multimodal Learning

Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey