Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data

Sijie Mai; Shiqin Han; Haifeng Hu

cs.LGMar 3, 2026

Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data

Authors: Sijie Mai, Shiqin Han, Haifeng Hu

Organizations: Department of Computer Science South China Normal University · School of Electronics and Information Technology Sun Yat-sen University

Abstract

Multimodal data encountered in real-world scenarios are typically of low quality, with noisy modalities and missing modalities being typical forms that severely hinder model performance and robustness. However, prior works often handle noisy and missing modalities separately. In contrast, we jointly address missing and noisy modalities to enhance model robustness in low-quality data scenarios. We regard both noisy and missing modalities as a unified low-quality modality problem, and propose a unified modality-quality (UMQ) framework to enhance low-quality representations for multimodal affective computing. Firstly, we train a quality estimator with explicit supervised signals via a rank-guided training strategy that compares the relative quality of different representations by adding a ranking constraint, avoiding training noise caused by inaccurate absolute quality labels. Then, a quality enhancer for each modality is constructed, which uses the sample-specific information provided by other modalities and the modality-specific information provided by the defined modality baseline representation to enhance the quality of unimodal representations. Finally, we propose a quality-aware mixture-of-experts module with particular routing mechanism to enable multiple modality-quality problems to be addressed more specifically. UMQ consistently outperforms state-of-the-art baselines on multiple datasets under the settings of complete, missing, and noisy modalities.

Figures & tables

Figure 1: An example of missing and noisy modalities.

Figure 2: Diagram of UMQ and the training objectives of the proposed components.


	CMU-MOSI					CMU-MOSEI
	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
MFM ( Tsai et al., 2019 )	33.3	80.0	80.1	0.948	0.664	50.8	83.4	83.4	0.580	0.722
DEVA † ( Wu et al., 2025 )	46.3	86.3	86.3	0.730	0.787	52.3	86.1	86.2	0.541	0.769
AtCAF † ( Huang et al., 2025 )	46.5	88.6	88.5	0.650	0.831	55.9	87.0	86.8	0.508	0.785
DLF † ( Wang et al., 2025 )	47.1	85.1	85.0	0.731	0.781	53.9	85.4	85.3	0.536	0.764

Table 1: The results on CMU-MOSI and CMU-MOSEI under complete modalities. The results labeled with † are obtained from original papers, and other results are obtained from our experiments.


Model	Acc	Parameters
HKT † ( Hasan et al., 2021 )	77.4	-
DMD+SuCI † ( Yang et al., 2024 )	70.8	-
AtCAF † ( Huang et al., 2025 )	72.1	-
HKT ( Hasan et al., 2021 )	76.5	17,066,564
MCL ( Mai et al., 2023a )	77.7	13,762,973

Table 2: The results on UR-FUNNY dataset.


Model	Acc	Parameters
HKT † ( Hasan et al., 2021 )	79.4	-
MO-Sarcation † ( Tomar et al., 2023 )	79.7	-
HKT ( Hasan et al., 2021 )	76.5	17,101,372
MCL ( Mai et al., 2023a )	77.9	13,828,449
MGCL ( Mai et al., 2023b )	77.9	14,282,000

Table 3: The results on MUStARD dataset.


		CIDer	MMIN	GCNet	IMDer	MoMKE	UMQ
	MR	Acc2 / F1 / Acc7	Acc2 / F1 / Acc7	Acc2 / F1 / Acc7	Acc2 / F1 / Acc7	Acc2 / F1 / Acc7	Acc2 / F1 / Acc7
MOSI	0.1	81.1 / 79.7 / 39.4	81.8 / 81.8 / 41.2	82.2 / 82.3 / 35.4	83.5 / 83.4 / 42.1	82.5 / 81.6 / 35.1	85.1 / 85.1 / 48.9
	0.2	78.5 / 75.6 / 36.7	79.0 / 79.1 / 38.9	79.4 / 79.5 / 34.6	80.5 / 80.5 /41.6	78.5 / 76.6 / 32.9	82.5 / 82.5 / 44.5
	0.3	76.0 / 71.6 / 34.0	76.1 / 76.2 / 36.9	77.1 / 77.2 / 32.5	77.4 / 77.6 / 37.4	74.4 / 71.7 / 30.6	78.2 / 78.2 / 41.4
	0.4	73.4 / 67.4 / 31.3	71.7 / 71.6 / 34.9	75.3 / 75.4 / 30.3	66.5 / 66.3 / 35.2	70.7 / 67.5 / 28.4	74.1 / 74.2 / 38.3

Table 4: The results under missing modalities on the CMU-MOSI and CMU-MOSEI datasets.


		C-MIB	Multimodal Boosting	UMQ
	NR	Acc2/MAE	Acc2/MAE	Acc2/MAE
MOSI	0.1	87.8 / 0.670	86.7 / 0.678	88.2 / 0.627
	0.2	87.5 / 0.726	86.1 / 0.738	87.8 / 0.652
	0.3	86.4 / 0.912	86.4 / 0.785	87.6 / 0.644
	0.4	83.2 / 1.366	85.5 / 0.841	86.9 / 0.643

Table 5: Discussion on noisy modalities.


	CMU-MOSI
Model	Acc7 $\uparrow$	Acc2 $\uparrow$	MAE $\uparrow$
W/O Quality Estimation	44.8	86.4	0.676
W/O Rank-Guided Training	46.8	86.6	0.670
W/O Quality Enhancement	47.1	88.4	0.640
W/O Modality Decoupling	48.6	88.3	0.639

Table 6: Ablation experiments.

Figure 3: Visualization of the original and reconstructed language features. ‘UMQ’ and ‘W/O MSI’ denote the reconstructed features obtained by our method and obtained without modality-specific information, respectively.

Figure 4: Analysis on estimated quality and selected experts. The modalities enclosed by the red bounding box are replaced with Gaussian noise.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix


	MAE $\downarrow$	Corr $\uparrow$	Acc2 $\uparrow$	F1-score $\uparrow$
MFM ( Tsai et al., 2019 )	0.493	0.473	71.7	70.5
Self-MM ( Yu et al., 2021 )	0.425	0.592	80.0	80.4
ConFEDE ( Yang et al., 2023 )	0.394	0.637	81.0	81.0
DEVA † ( Wu et al., 2025 )	0.424	0.583	79.6	80.3
ITHP ( Xiao et al., 2024 )	0.425	0.581	79.4	79.4

Appendix

Table 7: The results on CH-SIMS dataset. The results labeled with † are obtain in their original paper.


	CMU-MOSI					CMU-MOSEI
	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
MFM ( Tsai et al., 2019 )	33.3	80.0	80.1	0.948	0.664	50.8	83.4	83.4	0.580	0.722
Self-MM ( Yu et al., 2021 )	45.8	84.9	84.8	0.731	0.785	53.0	85.2	85.2	0.540	0.763
AtCAF † ( Huang et al., 2025 )	46.5	88.6	88.5	0.650	0.831	55.9	87.0	86.8	0.508	0.785
DLF † ( Wang et al., 2025 )	47.1	85.1	85.0	0.731	0.781	53.9	85.4	85.3	0.536	0.764

Appendix

Table 8: Complete results on CMU-MOSI and CMU-MOSEI under complete modalities. The results labeled with † are obtained from original papers, and other results are obtained from our experiments. The best results are highlighted in bold and the second best results are marked with underlines.


		Gaussian Noise				Other Noises (Laplace Noise and Random Erasing Noise)
		NIAT	C-MIB	MuBo	UMQ	NIAT	C-MIB	MuBo	UMQ	UMQ (OOD)
	NR	Acc2/MAE	Acc2/MAE	Acc2/MAE	Acc2/MAE	Acc2/MAE	Acc2/MAE	Acc2/MAE	Acc2/MAE	Acc2/MAE
MOSI	0.1	83.6 / 0.672	87.8 / 0.670	86.7 / 0.678	88.2 / 0.627	85.5 / 0.622	87.6 / 0.681	87.3/0.660	88.3 / 0.625	89.2 / 0.624
	0.2	83.0 / 0.689	87.5 / 0.726	86.1 / 0.738	87.8 / 0.652	84.1 / 0.684	87.3 / 0.695	85.8/0.726	87.4 / 0.659	87.7 / 0.679
	0.3	82.3 / 0.741	86.4 / 0.912	86.4 / 0.785	87.6 / 0.644	83.4 / 0.777	85.0 / 0.890	85.6/0.757	87.6 / 0.655	85.9 / 0.681

Appendix

Table 9: Discussion on noisy modalities on the CMU-MOSI and CMU-MOSEI datasets. MuBo denotes Multimodal Boosting ( Mai et al., 2024 ) .

Panel of Figure 5 — Figure 5: The MAE of UMQ with respect to the change of loss weights on CMU-MOSEI.

Panel of Figure 6 — Figure 6: The MAE of UMQ with respect to the change of the number of total experts $h$ and the number of selected experts $k$ on CMU-MOSEI.


	CMU-MOSI	CMU-MOSEI	CH-SIMS	UR-FUNNY	MUStARD
Loss Function	MSE	MSE	MSE	BCE	BCE
Batch Size	64	64	32	48	48
Learning Rate	1e-5	1e-5	1e-5	5e-6	5e-6
Shared Dimensionality $d$	100	100	100	100	64
Variance Margin $\beta$	0.1	0.1	0.1	0.1	0.1

Appendix

Table 10: Hyperparameter Settings of UMQ. MSE and BCE denotes mean square error and binary cross-entropy, respectively.

Figure 7: The structures of modality decouple network, modality couple network, fusion network and predictor.

Figure 8: The structures of quality estimator, quality enhancer, and experts.

Multimodal Robustness

Multimodal robustness studies how systems combining different data types remain reliable when inputs are missing, noisy, shifted, or misleading.

See all 279 papers

Jul 8, 2026

General Incomplete Multimodal Learning via Dynamic Quality Perception

Multimodal learning robust to missing modalities is essential for real-world applications. Existing methods mainly focus on inter-modality missing, where entire modalities are absent, while overlooking intra-modality degradation, where modalities are present but severely corrupted. In practice, these two types of missing often coexist, making existing approaches ineffective. To address this limitation, we propose General Incomplete Multimodal Learning (GIML), a unified framework that simultaneously handles both inter-modality missing and intra-modality degradation through dynamic quality perception. Specifically, GIML models heterogeneous missing patterns as continuous modality information degradation, enabling degradation-aware adaptive fusion. To achieve reliable quality perception, we introduce a Noise-aware Quality Estimator that learns the mapping from corrupted features to noise intensity through controlled noise injection. Furthermore, we propose a Noise-Semantic Decoupled module that separates semantic information from noise interference. This improves robustness and generalization to unseen corruption patterns. Extensive experiments across datasets with diverse modality types demonstrate the effectiveness and generality of GIML. Code is available at: https://github.com/Yu-Five/GIML.

Xiangyu Meng, Shicai Wei

University of Electronic Science and Technology of China

Missing-Modality Learning Multimodal Robustness Learning with Missing Data Multimodal Fusion

Ask PDF

Apr 7, 2026

QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) aims to infer human sentiment from textual, acoustic, and visual signals. In real-world scenarios, however, multimodal inputs are often compromised by dynamic noise or modality missingness. Existing methods typically treat these imperfections as discrete cases or assume fixed corruption ratios, which limits their adaptability to continuously varying reliability conditions. To address this, we first introduce a Continuous Reliability Spectrum to unify missingness and quality degradation into a single framework. Building on this, we propose QA-MoE, a Quality-Aware Mixture-of-Experts framework that quantifies modality reliability via self-supervised aleatoric uncertainty. This mechanism explicitly guides expert routing, enabling the model to suppress error propagation from unreliable signals while preserving task-relevant information. Extensive experiments indicate that QA-MoE achieves competitive or state-of-the-art performance across diverse degradation scenarios and exhibits a promising One-Checkpoint-for-All property in practice.

Yitong Zhu, Yuxuan Jiang, Guanxuan Jiang +4

The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Aarhus University

Multimodal Sentiment Analysis Multimodal Robustness Uncertainty-Aware Multimodal Fusion Mixture-of-Experts Models Missing-Modality Learning

Ask PDF

Sep 24, 2026

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.

Menghua Jiang, Haokai Gao, Xiangui Kang +2

School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University

Multimodal Sentiment Analysis Multimodal Robustness Uncertainty-Aware Multimodal Fusion Missing-Modality Learning

Ask PDF

Abstract

Multimodal Robustness

Explore similar work

General Incomplete Multimodal Learning via Dynamic Quality Perception

QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis