MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities

Tien Anh Pham; Phuong-Anh Nguyen; Duc-Trong Le; Cam-Van Thi Nguyen

cs.CVMar 10, 2026

MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities

Authors: Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le, Cam-Van Thi Nguyen

Organizations: VNU University of Engineering and Technology, Hanoi, Vietnam

Abstract

Multimodal affective computing underpins key tasks such as sentiment analysis and emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a benchmark and framework for multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality settings.For reproducibility, we release our code at: https://anonymous.4open.science/r/MissBench-4098/

Figures & tables

Figure 1. Sampled modality-specific missing rate under mean-match SMR=

0.5

and IMR=

(0.4,0.5,0.6)

. Under SMR

\hat{r_{m}}\approx r_{\mathrm{sh}}

, while under IMR

r_{m}

can receive any predefined value

\in[0,1)

Figure 2. Decoupled modality parameters ( Wang et al., 2023b ) of RedCore ( Sun et al., 2024 ) on IEMOCAP under different missing-rate configurations, illustrating how a single dominant modality can drive most parameter updates under imbalanced missing rates.

Work	Affective	Missing	IMR	Mod.-level	Grad.-level	Public
	focus	protocol	(imb.)	diagnostics	diagnostics	benchmark
MMIN ( Zhao et al., 2021 )	✓	SMR / pattern	×	×	×	×
GCNet ( Lian et al., 2023 )	✓	SMR / fixed	×	×	×	×
RedCore ( Sun et al., 2024 )	✓	IMR (custom)	✓	limited	×	×
MCE ( Zhao et al., 2025 )	general	IMR (general)	✓	utility metric	gradient-aware	×
MultiBench ( Liang et al., 2021 )	mixed	noise / drop	×	×	×	✓

Table 1. Comparison with prior work on missing modalities, multimodal imbalance, and benchmarks. “Missing protocol” shows whether SMR or IMR is explicitly modeled. “Mod.-level” and “Grad.-level” indicate modality-wise and optimization-level diagnostics comparable to odality Equity Index (MEI) and Modality Learning Index (MLI).

Figure 3. MissBench benchmark pipeline for multimodal affective tasks, illustrating data preparation, SMR/IMR missingness protocols, unified training with model plugins, and evaluation with both task metrics and modality-aware diagnostics.

Method

Setting

IEMOCAP

CMU-MOSI

0.3 / (0.1,0.2,0.6)

0.5 / (0.2,0.5,0.8)

0.7 / (0.4,0.8,0.9)

0.3 / (0.1,0.2,0.6)

0.5 / (0.2,0.5,0.8)

0.7 / (0.4,0.8,0.9)

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

GCNet

SMR

8.21_{\pm 5.40}

31.56_{\pm 3.08}

11.41_{\pm 8.31}

38.83_{\pm 6.99}

20.57_{\pm 17.78}

41.15_{\pm 6.45}

79.67_{\pm 14.41}

34.77_{\pm 7.29}

76.67_{\pm 7.15}

43.12_{\pm 8.51}

48.75_{\pm 30.61}

62.42_{\pm 5.19}

IMR

7.33_{\pm 2.48}

32.89_{\pm 3.55}

30.71_{\pm 34.82}

42.52_{\pm 8.69}

25.06_{\pm 28.70}

41.97_{\pm 4.62}

70.83_{\pm 10.94}

41.06_{\pm 9.81}

71.24_{\pm 18.21}

55.85_{\pm 10.38}

79.90_{\pm 15.55}

58.02_{\pm 9.19}

RedCore

SMR

5.92_{\pm 2.71}

34.77_{\pm 7.29}

13.45_{\pm 8.89}

44.03_{\pm 2.64}

11.48_{\pm 10.00}

43.12_{\pm 8.51}

88.08_{\pm 2.50}

34.28_{\pm 2.56}

85.04_{\pm 3.57}

41.76_{\pm 5.30}

83.57_{\pm 8.65}

49.20_{\pm 5.40}

IMR

7.06_{\pm 1.73}

41.06_{\pm 9.81}

8.39_{\pm 10.67}

46.81_{\pm 4.34}

5.63_{\pm 2.50}

55.85_{\pm 10.38}

86.85_{\pm 2.03}

40.39_{\pm 5.54}

87.16_{\pm 2.13}

40.86_{\pm 7.70}

80.58_{\pm 6.82}

51.01_{\pm 2.50}

Table 2.

\text{MEI}_{\text{UA}}

/MLI(%) of representatives from each method-family.

Figure 4. Performance of multimodal methods under shared missing rates (SMR) on IEMOCAP (WA) and CMU-MOSI/MOSEI/CH-SIMS (Acc-2). IMR-aware methods are shown in red, missing-modality handling models in blue, and gradient-based or generic baselines in green.

Method	Setting	0.3 / (0.1, 0.2, 0.6)			0.5 / (0.2, 0.5, 0.8)			0.7 / (0.4, 0.8, 0.9)
Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	SMR	$67.20_{\pm 1.48}$	$1.180_{\pm 0.085}$	$0.543_{\pm 0.025}$	$63.57_{\pm 2.82}$	$1.249_{\pm 0.008}$	$0.472_{\pm 0.015}$	$55.09_{\pm 6.66}$	$1.374_{\pm 0.064}$	$0.331_{\pm 0.092}$
MMIN	IMR	$61.80_{\pm 1.43}$	$1.288_{\pm 0.017}$	$0.425_{\pm 0.020}$	$55.61_{\pm 3.86}$	$1.398_{\pm 0.042}$	$0.305_{\pm 0.064}$	$52.59_{\pm 4.11}$	$1.425_{\pm 0.027}$	$0.293_{\pm 0.027}$
GCNet	SMR	$67.59_{\pm 1.60}$	$1.227_{\pm 0.025}$	$0.453_{\pm 0.014}$	$65.15_{\pm 2.77}$	$1.283_{\pm 0.057}$	$0.428_{\pm 0.020}$	$60.79_{\pm 4.01}$	$1.387_{\pm 0.043}$	$0.284_{\pm 0.070}$
GCNet	IMR	$64.33_{\pm 3.04}$	$1.316_{\pm 0.055}$	$0.346_{\pm 0.073}$	$55.58_{\pm 3.78}$	$1.436_{\pm 0.043}$	$0.167_{\pm 0.066}$	$55.95_{\pm 4.01}$	$1.425_{\pm 0.058}$	$0.208_{\pm 0.043}$
RedCore	SMR	$68.75_{\pm 1.59}$	$1.188_{\pm 0.032}$	$0.509_{\pm 0.013}$	$65.21_{\pm 1.73}$	$1.256_{\pm 0.008}$	$0.433_{\pm 0.023}$	$61.43_{\pm 2.70}$	$1.347_{\pm 0.047}$	$0.352_{\pm 0.047}$

Table 3. Task performance on CMU-MOSI under SMR and mean-matched IMR across low, medium, and high missing regimes. For IMR, modality-specific rates follow the order

(r_{A},r_{V},r_{L})

Method	Setting	0.3 / (0.1, 0.2, 0.6)		0.5 / (0.2, 0.5, 0.8)		0.7 / (0.4, 0.8, 0.9)
Method	Setting	WA	F1	WA	F1	WA	F1
MMIN	SMR	$63.98_{\pm 1.90}$	$64.30_{\pm 1.88}$	$55.10_{\pm 2.60}$	$55.36_{\pm 2.67}$	$49.43_{\pm 1.96}$	$49.51_{\pm 1.94}$
MMIN	IMR	$61.00_{\pm 2.53}$	$61.19_{\pm 2.57}$	$55.55_{\pm 1.95}$	$55.91_{\pm 1.91}$	$50.25_{\pm 1.72}$	$50.18_{\pm 2.03}$
GCNet	SMR	$76.07_{\pm 2.06}$	$76.23_{\pm 1.95}$	$75.33_{\pm 2.04}$	$75.49_{\pm 1.99}$	$73.52_{\pm 2.95}$	$73.54_{\pm 2.89}$
GCNet	IMR	$73.70_{\pm 3.58}$	$73.96_{\pm 3.26}$	$65.21_{\pm 4.47}$	$65.35_{\pm 4.67}$	$65.61_{\pm 3.70}$	$66.09_{\pm 3.69}$
RedCore	SMR	$64.90_{\pm 0.88}$	$65.15_{\pm 0.92}$	$58.58_{\pm 1.69}$	$58.82_{\pm 1.66}$	$51.36_{\pm 1.30}$	$51.71_{\pm 1.28}$

Table 4. Task performance on IEMOCAP under SMR and mean-matched IMR across low, medium, and high missing regimes. IMR configurations are reported using the modality order

(r_{A},r_{V},r_{L})

Panel of Figure 5 — Figure 5. Statistics of GCNet ( Lian et al., 2023 ) during training on IEMOCAP under different missing configurations.

Method

Setting (AV)

0.1 / (0.1, 0.1)

0.3 / (0.1, 0.5)

0.5 / (0.6, 0.4)

\mathrm{MEI}_{\mathrm{UA}}

MLI

\mathrm{MEI}_{\mathrm{UA}}

MLI

\mathrm{MEI}_{\mathrm{UA}}

MLI

GCNet

SMR

68.07_{\pm 3.55}

68.57_{\pm 3.43}

38.45_{\pm 16.94}

13.20_{\pm 3.21}

64.88_{\pm 3.01}

65.17_{\pm 3.30}

52.18_{\pm 34.67}

15.87_{\pm 4.57}

64.56_{\pm 3.08}

65.11_{\pm 2.70}

41.05_{\pm 32.08}

18.19_{\pm 5.21}

IMR

68.99_{\pm 3.54}

69.35_{\pm 3.47}

32.71_{\pm 14.74}

11.27_{\pm 3.69}

66.24_{\pm 1.79}

66.69_{\pm 1.56}

64.24_{\pm 19.16}

12.15_{\pm 3.38}

65.00_{\pm 3.53}

65.41_{\pm 3.27}

77.01_{\pm 9.12}

16.99_{\pm 4.04}

RedCore

SMR

60.35_{\pm 1.47}

60.60_{\pm 1.32}

10.42_{\pm 6.40}

15.73_{\pm 1.34}

54.68_{\pm 0.73}

54.79_{\pm 0.74}

3.60_{\pm 4.78}

22.50_{\pm 1.53}

48.62_{\pm 2.04}

47.33_{\pm 3.74}

2.77_{\pm 3.59}

28.95_{\pm 3.57}

IMR

58.03_{\pm 2.02}

58.32_{\pm 1.91}

5.24_{\pm 5.12}

14.86_{\pm 2.40}

54.18_{\pm 1.13}

54.22_{\pm 1.29}

9.78_{\pm 9.00}

22.03_{\pm 2.13}

48.88_{\pm 1.66}

48.07_{\pm 2.54}

15.30_{\pm 9.15}

26.20_{\pm 1.34}

Ada2I

SMR

72.07_{\pm 1.00}

72.01_{\pm 1.08}

17.45_{\pm 1.54}

19.69_{\pm 4.27}

71.39_{\pm 1.39}

71.44_{\pm 1.41}

17.05_{\pm 4.25}

27.07_{\pm 5.26}

71.75_{\pm 1.84}

71.78_{\pm 1.77}

22.02_{\pm 7.26}

32.42_{\pm 6.46}

Table 5. Performance comparison under different bimodal combinations (A: Audio, V: Visual, L: Language) and missing-rate settings, including Shared Missing Rate (SMR) and Imbalanced Missing Rate (IMR).

Figure 6. Performance of baselines on IEMOCAP under extreme IMR=(0.4,0.8,0.9).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Dialogues			Utterances
Dataset	train	valid	test	train	valid	test
IEMOCAP	120		31	5810		1623
CMU-MOSI	52	10	31	1284	229	686
CMU-MOSEI	2249	300	676	16326	1871	4659
CH-SIMS	60	59	59	1,368	456	457

Appendix

Table 6. Statistical overview of the MER and MSA datasets.

Method	Setting	0.3 / (0.1, 0.2, 0.6)			0.5 / (0.2, 0.5, 0.8)			0.7 / (0.4, 0.8, 0.9)
Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	SMR	$77.72_{\pm 0.47}$	$0.748_{\pm 0.022}$	$0.641_{\pm 0.014}$	$73.64_{\pm 2.82}$	$0.814_{\pm 0.015}$	$0.558_{\pm 0.004}$	$69.60_{\pm 1.42}$	$0.891_{\pm 0.032}$	$0.447_{\pm 0.041}$
MMIN	IMR	$71.80_{\pm 0.50}$	$0.855_{\pm 0.021}$	$0.497_{\pm 0.026}$	$68.97_{\pm 0.27}$	$0.890_{\pm 0.005}$	$0.427_{\pm 0.010}$	$68.00_{\pm 0.36}$	$0.916_{\pm 0.002}$	$0.392_{\pm 0.011}$
GCNet	SMR	$80.22_{\pm 1.10}$	$0.706_{\pm 0.024}$	$0.692_{\pm 0.023}$	$76.84_{\pm 0.67}$	$0.773_{\pm 0.016}$	$0.621_{\pm 0.018}$	$76.34_{\pm 0.96}$	$0.781_{\pm 0.014}$	$0.599_{\pm 0.016}$
GCNet	IMR	$76.74_{\pm 0.86}$	$0.776_{\pm 0.016}$	$0.616_{\pm 0.016}$	$73.89_{\pm 0.45}$	$0.828_{\pm 0.010}$	$0.541_{\pm 0.019}$	$73.19_{\pm 0.63}$	$0.839_{\pm 0.013}$	$0.525_{\pm 0.021}$
RedCore	SMR	$77.20_{\pm 0.52}$	$0.760_{\pm 0.007}$	$0.631_{\pm 0.009}$	$72.57_{\pm 0.79}$	$0.845_{\pm 0.012}$	$0.535_{\pm 0.015}$	$68.10_{\pm 0.44}$	$0.902_{\pm 0.012}$	$0.443_{\pm 0.014}$

Appendix

Table 7. Performance comparison on CMU-MOSEI

Method	Setting	0.3 / (0.1, 0.2, 0.6)			0.5 / (0.2, 0.5, 0.8)			0.7 / (0.4, 0.8, 0.9)
Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	SMR	$78.19_{\pm 0.26}$	$0.480_{\pm 0.012}$	$0.596_{\pm 0.026}$	$72.21_{\pm 1.86}$	$0.544_{\pm 0.009}$	$0.485_{\pm 0.028}$	$67.26_{\pm 2.28}$	$0.594_{\pm 0.034}$	$0.330_{\pm 0.097}$
MMIN	IMR	$71.80_{\pm 0.50}$	$0.509_{\pm 0.010}$	$0.542_{\pm 0.031}$	$68.97_{\pm 0.27}$	$0.621_{\pm 0.046}$	$0.212_{\pm 0.190}$	$68.00_{\pm 0.36}$	$0.603_{\pm 0.013}$	$0.318_{\pm 0.036}$
GCNet	SMR	$66.03_{\pm 0.59}$	$0.628_{\pm 0.008}$	$0.234_{\pm 0.024}$	$65.97_{\pm 1.45}$	$0.645_{\pm 0.010}$	$0.168_{\pm 0.041}$	$65.10_{\pm 0.66}$	$0.644_{\pm 0.005}$	$0.163_{\pm 0.022}$
GCNet	IMR	$64.94_{\pm 0.96}$	$0.648_{\pm 0.009}$	$0.161_{\pm 0.055}$	$64.77_{\pm 0.67}$	$0.648_{\pm 0.002}$	$0.152_{\pm 0.026}$	$65.55_{\pm 0.43}$	$0.649_{\pm 0.002}$	$0.140_{\pm 0.024}$
RedCore	SMR	$75.25_{\pm 1.57}$	$0.517_{\pm 0.024}$	$0.550_{\pm 0.053}$	$70.82_{\pm 1.49}$	$0.561_{\pm 0.012}$	$0.445_{\pm 0.023}$	$66.95_{\pm 1.39}$	$0.596_{\pm 0.016}$	$0.354_{\pm 0.040}$

Appendix

Table 8. Performance comparison on CH-SIMS

Method

(0.1, 0.6, 0.2)

(0.6, 0.1, 0.2)

(0.2, 0.8, 0.5)

(0.8, 0.2, 0.5)

(0.4, 0.9, 0.8)

(0.9, 0.4, 0.8)

MMIN

66.04_{\pm 2.63}

66.34_{\pm 2.60}

68.95_{\pm 0.65}

69.05_{\pm 0.69}

57.31_{\pm 0.88}

57.46_{\pm 0.89}

60.35_{\pm 1.61}

60.22_{\pm 1.26}

52.32_{\pm 2.17}

52.38_{\pm 2.09}

53.72_{\pm 0.50}

53.74_{\pm 0.46}

GCNet

74.61_{\pm 2.28}

74.91_{\pm 2.04}

77.14_{\pm 2.63}

76.87_{\pm 2.94}

68.03_{\pm 4.52}

68.31_{\pm 4.45}

76.52_{\pm 2.09}

76.33_{\pm 2.12}

65.24_{\pm 0.56}

65.58_{\pm 0.43}

72.52_{\pm 3.67}

72.42_{\pm 3.43}

RedCore

65.94_{\pm 0.27}

66.11_{\pm 0.25}

65.94_{\pm 0.56}

66.12_{\pm 0.57}

60.05_{\pm 0.74}

60.27_{\pm 0.72}

59.97_{\pm 2.73}

60.06_{\pm 2.40}

52.61_{\pm 1.42}

52.52_{\pm 1.49}

53.34_{\pm 2.46}

53.56_{\pm 2.35}

MCE

68.73_{\pm 0.84}

68.84_{\pm 0.81}

69.54_{\pm 0.62}

69.49_{\pm 0.55}

59.54_{\pm 1.20}

59.67_{\pm 1.25}

62.47_{\pm 0.89}

62.46_{\pm 0.77}

53.93_{\pm 0.81}

53.92_{\pm 0.86}

55.43_{\pm 1.55}

55.18_{\pm 1.47}

Ada2I

80.12_{\pm 1.03}

80.31_{\pm 0.98}

77.89_{\pm 0.53}

78.00_{\pm 0.47}

76.92_{\pm 1.73}

77.14_{\pm 1.69}

74.48_{\pm 1.22}

74.36_{\pm 1.38}

73.70_{\pm 2.12}

73.62_{\pm 2.74}

71.60_{\pm 2.68}

71.32_{\pm 3.10}

Appendix

Table 9. Performance comparison on IEMOCAP under different imbalanced missing rates

Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	(0.1,0.6,0.2)	$74.13_{\pm 0.72}$	$1.009_{\pm 0.019}$	$0.635_{\pm 0.010}$
	(0.6,0.1,0.2)	$73.73_{\pm 0.96}$	$1.076_{\pm 0.097}$	$0.622_{\pm 0.022}$
	(0.2,0.8,0.5)	$67.02_{\pm 0.94}$	$1.191_{\pm 0.024}$	$0.514_{\pm 0.001}$
	(0.8,0.2,0.5)	$65.65_{\pm 0.31}$	$1.210_{\pm 0.028}$	$0.493_{\pm 0.006}$
	(0.4,0.9,0.8)	$52.03_{\pm 7.26}$	$1.397_{\pm 0.077}$	$0.263_{\pm 0.166}$
	(0.9,0.4,0.8)	$52.38_{\pm 7.18}$	$1.417_{\pm 0.060}$	$0.248_{\pm 0.156}$

Appendix

Table 10. Performance comparison on CMU-MOSI under different imbalanced missing rates

Method	Setting	0.3 / (0.1, 0.2, 0.6)		0.5 / (0.2, 0.5, 0.8)		0.7 / (0.4, 0.8, 0.9)
Method	Setting	$\mathrm{MEI}_{\mathrm{UA}}$	MLI	$\mathrm{MEI}_{\mathrm{UA}}$	MLI	$\mathrm{MEI}_{\mathrm{UA}}$	MLI
GCNet	SMR	$74.25_{\pm 8.58}$	$33.63_{\pm 2.97}$	$82.90_{\pm 7.13}$	$41.05_{\pm 1.84}$	$89.51_{\pm 7.02}$	$44.12_{\pm 3.99}$
GCNet	IMR	$62.83_{\pm 15.95}$	$32.86_{\pm 3.65}$	$63.82_{\pm 6.30}$	$37.65_{\pm 2.67}$	$74.61_{\pm 15.01}$	$39.74_{\pm 2.49}$
RedCore	SMR	$80.19_{\pm 2.73}$	$33.92_{\pm 1.27}$	$80.52_{\pm 4.42}$	$38.63_{\pm 2.29}$	$83.62_{\pm 0.75}$	$43.25_{\pm 2.43}$
RedCore	IMR	$80.65_{\pm 1.39}$	$35.00_{\pm 2.10}$	$84.36_{\pm 3.72}$	$40.40_{\pm 3.51}$	$84.83_{\pm 1.73}$	$42.86_{\pm 4.67}$
Ada2I	SMR	$86.82_{\pm 5.45}$	$15.22_{\pm 1.21}$	$80.65_{\pm 10.39}$	$13.41_{\pm 2.22}$	$89.86_{\pm 7.60}$	$12.79_{\pm 1.48}$

Appendix

Table 11.

\text{MEI}_{\text{UA}}

/MLI(%) of representatives from each method-family on CMU-MOSEI.

Method	AV			AL			VL
Method	WA	F1	MEI UA	WA	F1	MEI UA	WA	F1	MEI UA
GCNet	$68.61_{\pm 1.92}$	$69.19_{\pm 1.56}$	$35.05_{\pm 21.01}$	$74.62_{\pm 2.71}$	$74.83_{\pm 2.75}$	$6.80_{\pm 6.16}$	$78.31_{\pm 1.40}$	$78.04_{\pm 1.26}$	$62.24_{\pm 20.04}$
RedCore	$72.93_{\pm 0.95}$	$72.89_{\pm 0.92}$	$13.75_{\pm 1.39}$	$80.61_{\pm 2.18}$	$80.84_{\pm 2.09}$	$8.62_{\pm 7.84}$	$78.69_{\pm 2.85}$	$78.50_{\pm 2.94}$	$39.77_{\pm 29.06}$
Ada2I	$62.34_{\pm 0.67}$	$62.47_{\pm 0.74}$	$9.55_{\pm 6.20}$	$65.51_{\pm 1.49}$	$65.53_{\pm 1.57}$	$71.70_{\pm 14.28}$	$68.73_{\pm 0.74}$	$68.73_{\pm 0.64}$	$45.48_{\pm 19.01}$

Appendix

Table 12. Performance comparison under different bimodal combinations (A: Audio, V: Visual, L: Language) on non-missing data

Figure 7. Estimated training time (sec/epoch) of the baselines.

Figure 8. MEI evaluated by different task-specific metrics UA/WA/F1 during the training of GCNet on IEMOCAP.

Panel of Figure 9 — Figure 9. Diagnosis of GCNet on a different IEMOCAP train/valid split from Figure 5 .

Panel of Figure 10 — Figure 10. Diagnosis of RedCore on IEMOCAP.

Panel of Figure 11 — Figure 11. Diagnosis of Ada2I on IEMOCAP.

Panel of Figure 12 — Figure 12. Diagnosis of RedCore on CMU-MOSI.

Panel of Figure 13 — Figure 13. Diagnosis of Ada2I on CMU-MOSI.

Panel of Figure 14 — Figure 14. Diagnosis of GCNet on CMU-MOSI.

Explore similar work

Jan 31, 2026

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

Current Multimodal Affective Computing (MAC) systems heavily rely on the completeness of multiple modalities to accurately understand human's affective state. However, in real-world scenarios, the availability of modality data is often dynamic and uncertain, leading to substantial performance fluctuations due to the distribution shifts and semantic deficiencies of the incomplete multimodal inputs. Known as the missing modality issue, this challenge poses a critical barrier to the robustness and practical deployment of MAC models. To systematically quantify this issue, we introduce MissMAC-Bench, a comprehensive benchmark designed to establish fair and unified evaluation standards from the perspective of cross-modal synergy. Two guiding principles are proposed, including no missing prior during training, and one single model capable of handling both complete and incomplete modality scenarios, thereby ensuring better generalization. Moreover, to bridge the gap between academic research and real-world applications, our benchmark integrates evaluation protocols with both fixed and random missing patterns at the dataset and instance levels. Extensive experiments conducted on 3 widely-used language models across 4 datasets validate the effectiveness of diverse MAC approaches in tackling the missing modality issue. Our benchmark provides a solid foundation for advancing robust MAC and promotes the development of multimedia data mining. Our code is released in https://github.com/RH-Lin/MissMAC-Bench.

Ronghao Lin, Honghao Lu, Ruixing Wu +5

Sun Yat-Sen University Guangzhou, China · South China Normal University Guangzhou, China · Sun Yat-Sen University Pazhou Laboratory Guangzhou, Guangdong, China

Missing-Modality Learning Affective Computing Multimodal Robustness Multimodal Model Evaluation

Ask PDF

Sep 10, 2026

The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods

Multimodal Sentiment Analysis (MSA) remains constrained by modality imbalance, yet the field continues to rely on optimization-based balancing methods that promise more than they deliver. We provide three contributions: 1) a unified evaluation framework testing gradient and loss-based balancing strategies under controlled settings; 2) a theoretical diagnosis explaining why these methods fail, as they conflate fitting speed with discriminative contribution; and 3) a research agenda toward held-out discriminative modality valuation. Experiments on CMU-MOSI and CMU-MOSEI reveal three shortcomings: no strategy reliably outperforms Late Concatenation; performance is sensitive to hyperparameters; and even ratio calibration fails to yield consistent gains. The core issue is fundamental: loss is not utility, and gradients are not importance. Modality imbalance remains unresolved, motivating utility estimation from held-out performance.

Ioanna Kaffeza, Efthymios Georgiou, Alexandros Potamianos

PERSEE Center, Mines Paris-PSL University, Sophia-Antipolis, France · National Technical University of Athens, Athens, Greece · University of Bern, Bern, Switzerland +2

Modality Imbalance Multimodal Optimization Multimodal Sentiment Analysis Multimodal Fusion Multimodal Model Evaluation

Ask PDF

Apr 7, 2026

Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities

The missing modality problem poses a fundamental challenge in multimodal sentiment analysis, significantly degrading model accuracy and generalization in real world scenarios. Existing approaches primarily improve robustness through prompt learning and pre trained models. However, two limitations remain. First, the necessity of generating missing modalities lacks rigorous evaluation. Second, the structural dependencies among multimodal prompts and their global coherence are insufficiently explored. To address these issues, a Prompt based Missing Modality Adaptation framework is proposed. A Missing Modality Evaluator is introduced at the input stage to dynamically assess the importance of missing modalities using pretrained models and pseudo labels, thereby avoiding low quality data imputation. Building on this, a Modality invariant Prompt Disentanglement module decomposes shared prompts into modality specific private prompts to capture intrinsic local correlations and improve representation quality. In addition, a Dynamic Prompt Weighting module computes mutual information based weights from cross attention outputs to adaptively suppress interference from missing modalities. To enhance global consistency, a Multi level Prompt Dynamic Connection module integrates shared prompts with self attention outputs through residual connections, leveraging global prompt priors to strengthen key guidance features. Extensive experiments on three public benchmarks, including CMU MOSI, CMU MOSEI, and CH SIMS, demonstrate that the proposed framework achieves state of the art performance and stable results under diverse missing modality settings. The implementation is available at https://github.com/rongfei-chen/ProMMA

Rongfei Chen, Tingting Zhang, Xiaoyu Shen +1

School of Computer Science and Technology, Eastern Institute of Technology, Ningbo, China · School of Mechatronic Engineering and Automation, Shanghai University, China

Multimodal Sentiment Analysis Missing-Modality Learning Multimodal Robustness Multimodal Learning

Ask PDF