QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis

Yitong Zhu; Yuxuan Jiang; Guanxuan Jiang; Bojing Hou; Peng Yuan Zhou; Ge Lin Kan; Yuyang Wang

cs.AIApr 7, 2026

QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis

Authors: Yitong Zhu, Yuxuan Jiang, Guanxuan Jiang, Bojing Hou, Peng Yuan Zhou, Ge Lin Kan, Yuyang Wang

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Aarhus University

Abstract

Multimodal Sentiment Analysis (MSA) aims to infer human sentiment from textual, acoustic, and visual signals. In real-world scenarios, however, multimodal inputs are often compromised by dynamic noise or modality missingness. Existing methods typically treat these imperfections as discrete cases or assume fixed corruption ratios, which limits their adaptability to continuously varying reliability conditions. To address this, we first introduce a Continuous Reliability Spectrum to unify missingness and quality degradation into a single framework. Building on this, we propose QA-MoE, a Quality-Aware Mixture-of-Experts framework that quantifies modality reliability via self-supervised aleatoric uncertainty. This mechanism explicitly guides expert routing, enabling the model to suppress error propagation from unreliable signals while preserving task-relevant information. Extensive experiments indicate that QA-MoE achieves competitive or state-of-the-art performance across diverse degradation scenarios and exhibits a promising One-Checkpoint-for-All property in practice.

Figures & tables

Figure 1: The Continuous Reliability Spectrum unifies three evaluation protocols defined by noise intensity ( $\lambda$ ) and missing rate ( $\eta$ ), and inputs from Text, Audio, and Vision are processed as imperfect multimodal inputs.

Figure 2: Overview of QA-MoE. (A) Probabilistic Feature Modeling encodes inputs as distributions to capture the uncertainty. (B) Quality-Aware Routing calculates a quality score $r_{m}$ to guide the expert selection. (C) Dual-Branch Prediction outputs both the prediction and uncertainty to optimize the model via heteroscedastic regression.

Models	CMU-MOSI					CMU-MOSEI					MIntRec
Models	$\text{ACC}_{7}$ $\uparrow$	$\text{ACC}_{2}$ $\uparrow$	$\text{F}_{1}$ $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	$\text{ACC}_{7}$ $\uparrow$	$\text{ACC}_{2}$ $\uparrow$	$\text{F}_{1}$ $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	ACC $\uparrow$	$\text{F}_{1}$ $\uparrow$
TFN †	31.9	78.8	78.9	0.953	0.698	50.9	80.4	80.7	0.574	0.700	-	-
LMF †	36.9	78.7	78.7	0.931	0.695	52.3	84.7	84.5	0.564	0.677	-	-
MulT †	35.1	80.0	80.1	0.936	0.711	52.3	82.7	82.8	0.572	-	72.6	69.5
MISA †	41.8	84.2	84.2	0.754	0.761	52.3	85.3	85.1	0.543	0.756	72.4	70.8
MMIM †	45.8	84.6	84.5	0.717	-	50.1	83.6	83.5	0.580	-	-	-

Table 1: Experimental results on CMU-MOSI, CMU-MOSEI, and MIntRec datasets. The results marked with † are retrieved from Fang et al. (2025) , and those with ‡ are cited from Chen et al. (2025) .

Datasets	Models	Testing Condition (Available Modalities)							Avg.
Datasets	Models	$\{t\}$	$\{a\}$	$\{v\}$	$\{t,a\}$	$\{t,v\}$	$\{a,v\}$	$\{t,a,v\}$	Avg.
IEMOCAP	MulT	62.4 / 63.7	49.7 / 51.6	48.9 / 45.7	68.3 / 69.4	67.8 / 68.3	56.3 / 55.8	70.1 / 70.5	60.5 / 60.7
	MISA	66.5 / 68.0	56.5 / 59.0	52.5 / 51.6	72.9 / 75.1	72.6 / 73.6	63.9 / 65.4	74.2 / 74.5	65.5 / 66.7
	MMIM	67.0 / 68.2	55.0 / 53.2	51.9 / 50.4	74.0 / 75.4	72.6 / 73.6	65.3 / 66.5	75.5 / 75.8	65.9 / 66.1
	Ours	71.2 / 72.3	58.2 / 59.1	54.6 / 53.8	75.1 / 75.3	73.8 / 74.1	66.9 / 66.8	77.1 / 77.3	68.1 / 68.4
CMU-MOSI	MCTN	79.10 / 79.20	56.10 / 54.50	55.00 / 54.40	81.00 / 81.00	81.10 / 81.20	57.50 / 57.40	81.40 / 81.50	68.30 / 67.95

Table 2: Performance comparison under various modality missingness scenarios. The values denote

\text{ACC}_{2}/\text{F}_{1}

. The testing conditions indicate the available modalities (e.g.,

\{t\}

means only Textual is available).

Dataset	Models	Random Missing Rate ( $\eta$ )							Avg.
Dataset	Models	10%	20%	30%	40%	50%	60%	70%	Avg.
CMU-MOSI	MCTN	39.8 / 78.5	38.5 / 75.7	35.5 / 71.2	32.9 / 67.6	31.2 / 64.8	29.7 / 62.5	27.5 / 59.0	33.6 / 68.5
	MMIN	41.2 / 81.8	38.9 / 79.1	36.9 / 76.2	34.9 / 71.6	34.2 / 66.5	29.1 / 64.0	28.4 / 61.0	34.5 / 71.5
	IMDer	42.1 / 83.4	41.6 / 80.5	37.4 / 77.6	35.2 / 66.3	29.5 / 65.4	27.0 / 65.5	26.5 / 60.4	34.2 / 71.3
	MoMKE	35.1 / 81.6	32.9 / 76.6	30.6 / 71.7	28.4 / 67.5	26.2 / 63.2	23.9 / 58.9	22.4 / 55.9	28.5 / 67.9
	SAM-LML	45.6 / 84.7	42.9 / 81.2	37.5 / 78.1	37.8 / 74.7	32.8 / 70.9	28.1 / 66.6	26.4 / 65.6	35.9 / 74.5

Table 3: Robustness comparison under Random Missing Protocol (

\text{ACC}_{7}/\text{F}_{1})

Dataset	NR	C-MIB	MM-Boosting	SAM-LML	QA-MoE (Ours)
Dataset	NR	$\text{ACC}_{2}$ / MAE	$\text{ACC}_{2}$ / MAE	$\text{ACC}_{2}$ / MAE	$\text{ACC}_{2}$ / MAE
CMU-MOSI	0.1	87.8 / 0.670	86.7 / 0.678	88.4 / 0.636	89.4 / 0.616
	0.2	87.5 / 0.726	86.1 / 0.738	88.1 / 0.665	88.9 / 0.636
	0.3	86.4 / 0.912	86.4 / 0.785	87.8 / 0.663	88.6 / 0.639
	0.4	83.2 / 1.366	85.5 / 0.841	87.6 / 0.666	88.4 / 0.641
	0.5	84.9 / 1.660	86.1 / 1.172	88.1 / 0.666	88.1 / 0.649

Table 4: Comparison under varying NI

(\lambda)

.The results are cited from Mai et al. (2025) .

Model	Training Strategy	Mixed Test Set (Protocol III)
Model	Training Strategy	MAE $\downarrow$	$\text{ACC}_{7}$ $\uparrow$	$\text{ACC}_{2}\uparrow$	$\text{F}_{1}\uparrow$
MMA	Clean Train	0.693	46.9	86.4	86.4
MMA	Spectrum Train	0.688	51.8	87.2	88.4
SAM-LML	Clean Train	0.628	49.4	89.2	89.1
SAM-LML	Spectrum Train	0.599	52.5	89.6	89.7
QA-MoE (Ours)	Clean Train	0.589	53.6	87.3	87.6

Table 5: Decoupling analysis under Protocol III.

Figure 3: Continuous Reliability Landscape. The smooth performance gradient (from warm to cool colors) demonstrates that QA-MoE exhibits graceful degradation rather than abrupt failure. The star ( $\star$ ) marks the compound defect scenario ( $\lambda=0.3,\eta=0.2$ ).

Model Variants	Spectrum Training Setting
Model Variants	MAE $\downarrow$	$\text{ACC}_{7}\uparrow$	$\text{ACC}_{2}\uparrow$	$\text{F}_{1}\uparrow$
QA-MoE (Full)	0.525	54.5	88.8	89.1
w/o Quality Gating	0.884	52.1	76.2	77.4
w/o Variance ( $\boldsymbol{\sigma}^{2}$ )	0.795	50.7	79.1	51.2
w/o Universal Fallback ( $\mathbf{y}_{prior}$ )	0.780	48.6	78.5	45.9

Table 6: Ablation studies on CMU-MOSI. We evaluate the contribution of the key parts of our QA-MoE.

Figure 4: Parameter $k$ Sensitivity Analysis.

Figure 5: Visualization of Adaptive Routing Patterns.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Models	CMU-MOSI					CMU-MOSEI
Models	ACC-7 $\uparrow$	ACC-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	ACC-7 $\uparrow$	ACC-2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
TFN †	35.3	76.5	76.6	0.995	0.698	50.2	84.2	84.0	0.573	0.700
LMF †	31.1	79.1	79.1	0.963	0.695	51.9	83.8	83.9	0.565	0.677
MulT †	33.2	80.3	80.3	0.933	0.711	53.2	84.0	84.0	0.556	-
MISA †	43.6	83.8	83.9	0.742	0.761	51.0	84.8	84.8	0.557	0.756
MMIM †	45.9	83.4	83.4	0.777	-	52.6	81.5	81.3	0.578	-

Appendix

Table 7: Experimental results on CMU-MOSI and CMU-MOSEI datasets. The results marked with † are retrieved from Fang et al. (2025) , and those with ‡ are cited from Chen et al. (2025) .

MR ( $\eta$ )	Noise Intensity ( $\lambda$ )
MR ( $\eta$ )	0.0	0.1	0.2	0.3	0.4	0.5	0.6	0.7
0%	54.50	54.27	51.31	46.65	41.98	38.63	32.36	31.92
10%	54.20	53.98	51.60	46.79	42.27	39.07	32.94	33.38
20%	52.60	51.79	49.27	44.61	40.67	37.76	31.63	32.36
30%	49.25	48.31	46.65	43.15	38.92	35.71	32.22	31.05
40%	43.06	42.38	41.44	40.38	37.17	34.11	30.32	30.03

Appendix

Table 8: Discrete Evaluation Grid (ACC-7 %). This table presents the exact performance metrics of the single QA-MoE checkpoint across varying degrees of degradation. The gray cell marks the compound defect scenario (

\lambda=0.3,\eta=20\%

) analyzed in Figure 3 .

Figure 6: Reliability Landscape of Baseline (SAM-LML). The visualization reveals a sharp performance decay, forming a reliability cliff.” While the model achieves peak performance at the clean origin, its accuracy plummets rapidly as degradation intensity increases. The star ( $\star$ ) marks the compound defect scenario ( $\lambda=0.3,\eta=0.2$ ), where accuracy has already degraded to 35.5%, demonstrating the lack of robustness in the One-Checkpoint” setting.

Model	Parameters (M)		Comp.	Speed	Mem.
Model	Total	Active	GFLOPs $\downarrow$	Latency (ms) $\downarrow$	VRAM (MB) $\downarrow$
MulT	113.1	113.1	6.82	14.5	1350
MISA	114.4	114.4	5.95	12.8	1600
MMA	113.5	113.5	6.10	13.2	1420
SAM-LML	112.8	112.8	5.15	11.9	1100
QA-MoE (Ours)	113.2	38.4 ∗	4.33	10.59	950

Appendix

Table 9: Computational Efficiency Analysis. We compare QA-MoE against strong baselines on a single RTX 4090 GPU (Batch=16). "Active" denotes the number of parameters actually used during a single inference pass.

Multimodal Sentiment Analysis

Multimodal sentiment analysis combines language, voice, and visual cues to infer sentiment, polarity, and emotional intensity.

See all 54 papers

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis
Sep 24, 2026Menghua Jiang, Haokai Gao, Xiangui Kang +2Multimodal Sentiment Analysis Multimodal Robustness
School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University
Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations
Aug 4, 2026Chunlei Meng, Jacqueline J. Pang, Pengbin Feng +3Missing-Modality Learning Multimodal Sentiment Analysis
The College of Intelligent Robotics and Advanced Manufacturing (CIRAM), Fudan University · Cornell University · University of Southern California (USC) +1
Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis
Oct 20, 2026Xiaohua Huang, Bingrui Lei, Xiuyan Shao +4Multimodal Sentiment Analysis Modality Reliance in Multimodal Models
the School of Communication and Information Engineering, Nanjing Institute of Technology, Nanjing 210096, China. · the School of Economics and Management, Southeast University, Nanjing 211196, China. · the School of Computer Science and Engineering, Macau University of Science and Technology, Macau SAR, 999078, China. +2

Sep 24, 2026J/K move · Enter open · S save

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Menghua Jiang, Haokai Gao, Xiangui Kang, Haifeng Hu+1

School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University

Multimodal Sentiment Analysis Multimodal Robustness Uncertainty-Aware Multimodal Fusion Missing-Modality Learning

Open paper AskPDF arXiv

Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.

Abstract

Multimodal Sentiment Analysis

Explore similar work

Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis

Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations

Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis