MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities

Tien Anh Pham; Phuong-Anh Nguyen; Duc-Trong Le; Cam-Van Thi Nguyen

cs.CVMar 10, 2026

MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities

Authors: Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le, Cam-Van Thi Nguyen

Organizations: VNU University of Engineering and Technology, Hanoi, Vietnam

Abstract

Multimodal affective computing underpins key tasks such as sentiment analysis and emotion recognition. Standard evaluations, however, often assume that textual, acoustic, and visual modalities are equally available. In real applications, some modalities are systematically more fragile or expensive, creating imbalanced missing rates and training biases that task-level metrics alone do not reveal. We introduce MissBench, a benchmark and framework for multimodal affective tasks that standardizes both shared and imbalanced missing-rate protocols on four widely used sentiment and emotion datasets. MissBench also defines two diagnostic metrics. The Modality Equity Index (MEI) measures how fairly different modalities contribute across missing-modality configurations. The Modality Learning Index (MLI) quantifies optimization imbalance by comparing modality-specific gradient norms during training, aggregated across modality-related modules. Experiments on representative method families show that models that appear robust under shared missing rates can still exhibit marked modality inequity and optimization imbalance under imbalanced conditions. These findings position MissBench, together with MEI and MLI, as practical tools for stress-testing and analyzing multimodal affective models in realistic incomplete-modality settings.For reproducibility, we release our code at: https://anonymous.4open.science/r/MissBench-4098/

Figures & tables

Figure 1. Sampled modality-specific missing rate under mean-match SMR=

0.5

and IMR=

(0.4,0.5,0.6)

. Under SMR

\hat{r_{m}}\approx r_{\mathrm{sh}}

, while under IMR

r_{m}

can receive any predefined value

\in[0,1)

Figure 2. Decoupled modality parameters ( Wang et al., 2023b ) of RedCore ( Sun et al., 2024 ) on IEMOCAP under different missing-rate configurations, illustrating how a single dominant modality can drive most parameter updates under imbalanced missing rates.

Work	Affective	Missing	IMR	Mod.-level	Grad.-level	Public
	focus	protocol	(imb.)	diagnostics	diagnostics	benchmark
MMIN ( Zhao et al., 2021 )	✓	SMR / pattern	×	×	×	×
GCNet ( Lian et al., 2023 )	✓	SMR / fixed	×	×	×	×
RedCore ( Sun et al., 2024 )	✓	IMR (custom)	✓	limited	×	×
MCE ( Zhao et al., 2025 )	general	IMR (general)	✓	utility metric	gradient-aware	×
MultiBench ( Liang et al., 2021 )	mixed	noise / drop	×	×	×	✓

Table 1. Comparison with prior work on missing modalities, multimodal imbalance, and benchmarks. “Missing protocol” shows whether SMR or IMR is explicitly modeled. “Mod.-level” and “Grad.-level” indicate modality-wise and optimization-level diagnostics comparable to odality Equity Index (MEI) and Modality Learning Index (MLI).

Figure 3. MissBench benchmark pipeline for multimodal affective tasks, illustrating data preparation, SMR/IMR missingness protocols, unified training with model plugins, and evaluation with both task metrics and modality-aware diagnostics.

Method

Setting

IEMOCAP

CMU-MOSI

0.3 / (0.1,0.2,0.6)

0.5 / (0.2,0.5,0.8)

0.7 / (0.4,0.8,0.9)

0.3 / (0.1,0.2,0.6)

0.5 / (0.2,0.5,0.8)

0.7 / (0.4,0.8,0.9)

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

MEI UA

MLI

GCNet

SMR

8.21_{\pm 5.40}

31.56_{\pm 3.08}

11.41_{\pm 8.31}

38.83_{\pm 6.99}

20.57_{\pm 17.78}

41.15_{\pm 6.45}

79.67_{\pm 14.41}

34.77_{\pm 7.29}

76.67_{\pm 7.15}

43.12_{\pm 8.51}

48.75_{\pm 30.61}

62.42_{\pm 5.19}

IMR

7.33_{\pm 2.48}

32.89_{\pm 3.55}

30.71_{\pm 34.82}

42.52_{\pm 8.69}

25.06_{\pm 28.70}

41.97_{\pm 4.62}

70.83_{\pm 10.94}

41.06_{\pm 9.81}

71.24_{\pm 18.21}

55.85_{\pm 10.38}

79.90_{\pm 15.55}

58.02_{\pm 9.19}

RedCore

SMR

5.92_{\pm 2.71}

34.77_{\pm 7.29}

13.45_{\pm 8.89}

44.03_{\pm 2.64}

11.48_{\pm 10.00}

43.12_{\pm 8.51}

88.08_{\pm 2.50}

34.28_{\pm 2.56}

85.04_{\pm 3.57}

41.76_{\pm 5.30}

83.57_{\pm 8.65}

49.20_{\pm 5.40}

IMR

7.06_{\pm 1.73}

41.06_{\pm 9.81}

8.39_{\pm 10.67}

46.81_{\pm 4.34}

5.63_{\pm 2.50}

55.85_{\pm 10.38}

86.85_{\pm 2.03}

40.39_{\pm 5.54}

87.16_{\pm 2.13}

40.86_{\pm 7.70}

80.58_{\pm 6.82}

51.01_{\pm 2.50}

Table 2.

\text{MEI}_{\text{UA}}

/MLI(%) of representatives from each method-family.

Figure 4. Performance of multimodal methods under shared missing rates (SMR) on IEMOCAP (WA) and CMU-MOSI/MOSEI/CH-SIMS (Acc-2). IMR-aware methods are shown in red, missing-modality handling models in blue, and gradient-based or generic baselines in green.

Method	Setting	0.3 / (0.1, 0.2, 0.6)			0.5 / (0.2, 0.5, 0.8)			0.7 / (0.4, 0.8, 0.9)
Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	SMR	$67.20_{\pm 1.48}$	$1.180_{\pm 0.085}$	$0.543_{\pm 0.025}$	$63.57_{\pm 2.82}$	$1.249_{\pm 0.008}$	$0.472_{\pm 0.015}$	$55.09_{\pm 6.66}$	$1.374_{\pm 0.064}$	$0.331_{\pm 0.092}$
MMIN	IMR	$61.80_{\pm 1.43}$	$1.288_{\pm 0.017}$	$0.425_{\pm 0.020}$	$55.61_{\pm 3.86}$	$1.398_{\pm 0.042}$	$0.305_{\pm 0.064}$	$52.59_{\pm 4.11}$	$1.425_{\pm 0.027}$	$0.293_{\pm 0.027}$
GCNet	SMR	$67.59_{\pm 1.60}$	$1.227_{\pm 0.025}$	$0.453_{\pm 0.014}$	$65.15_{\pm 2.77}$	$1.283_{\pm 0.057}$	$0.428_{\pm 0.020}$	$60.79_{\pm 4.01}$	$1.387_{\pm 0.043}$	$0.284_{\pm 0.070}$
GCNet	IMR	$64.33_{\pm 3.04}$	$1.316_{\pm 0.055}$	$0.346_{\pm 0.073}$	$55.58_{\pm 3.78}$	$1.436_{\pm 0.043}$	$0.167_{\pm 0.066}$	$55.95_{\pm 4.01}$	$1.425_{\pm 0.058}$	$0.208_{\pm 0.043}$
RedCore	SMR	$68.75_{\pm 1.59}$	$1.188_{\pm 0.032}$	$0.509_{\pm 0.013}$	$65.21_{\pm 1.73}$	$1.256_{\pm 0.008}$	$0.433_{\pm 0.023}$	$61.43_{\pm 2.70}$	$1.347_{\pm 0.047}$	$0.352_{\pm 0.047}$

Table 3. Task performance on CMU-MOSI under SMR and mean-matched IMR across low, medium, and high missing regimes. For IMR, modality-specific rates follow the order

(r_{A},r_{V},r_{L})

Method	Setting	0.3 / (0.1, 0.2, 0.6)		0.5 / (0.2, 0.5, 0.8)		0.7 / (0.4, 0.8, 0.9)
Method	Setting	WA	F1	WA	F1	WA	F1
MMIN	SMR	$63.98_{\pm 1.90}$	$64.30_{\pm 1.88}$	$55.10_{\pm 2.60}$	$55.36_{\pm 2.67}$	$49.43_{\pm 1.96}$	$49.51_{\pm 1.94}$
MMIN	IMR	$61.00_{\pm 2.53}$	$61.19_{\pm 2.57}$	$55.55_{\pm 1.95}$	$55.91_{\pm 1.91}$	$50.25_{\pm 1.72}$	$50.18_{\pm 2.03}$
GCNet	SMR	$76.07_{\pm 2.06}$	$76.23_{\pm 1.95}$	$75.33_{\pm 2.04}$	$75.49_{\pm 1.99}$	$73.52_{\pm 2.95}$	$73.54_{\pm 2.89}$
GCNet	IMR	$73.70_{\pm 3.58}$	$73.96_{\pm 3.26}$	$65.21_{\pm 4.47}$	$65.35_{\pm 4.67}$	$65.61_{\pm 3.70}$	$66.09_{\pm 3.69}$
RedCore	SMR	$64.90_{\pm 0.88}$	$65.15_{\pm 0.92}$	$58.58_{\pm 1.69}$	$58.82_{\pm 1.66}$	$51.36_{\pm 1.30}$	$51.71_{\pm 1.28}$

Table 4. Task performance on IEMOCAP under SMR and mean-matched IMR across low, medium, and high missing regimes. IMR configurations are reported using the modality order

(r_{A},r_{V},r_{L})

Panel of Figure 5 — Figure 5. Statistics of GCNet ( Lian et al., 2023 ) during training on IEMOCAP under different missing configurations.

Method

Setting (AV)

0.1 / (0.1, 0.1)

0.3 / (0.1, 0.5)

0.5 / (0.6, 0.4)

\mathrm{MEI}_{\mathrm{UA}}

MLI

\mathrm{MEI}_{\mathrm{UA}}

MLI

\mathrm{MEI}_{\mathrm{UA}}

MLI

GCNet

SMR

68.07_{\pm 3.55}

68.57_{\pm 3.43}

38.45_{\pm 16.94}

13.20_{\pm 3.21}

64.88_{\pm 3.01}

65.17_{\pm 3.30}

52.18_{\pm 34.67}

15.87_{\pm 4.57}

64.56_{\pm 3.08}

65.11_{\pm 2.70}

41.05_{\pm 32.08}

18.19_{\pm 5.21}

IMR

68.99_{\pm 3.54}

69.35_{\pm 3.47}

32.71_{\pm 14.74}

11.27_{\pm 3.69}

66.24_{\pm 1.79}

66.69_{\pm 1.56}

64.24_{\pm 19.16}

12.15_{\pm 3.38}

65.00_{\pm 3.53}

65.41_{\pm 3.27}

77.01_{\pm 9.12}

16.99_{\pm 4.04}

RedCore

SMR

60.35_{\pm 1.47}

60.60_{\pm 1.32}

10.42_{\pm 6.40}

15.73_{\pm 1.34}

54.68_{\pm 0.73}

54.79_{\pm 0.74}

3.60_{\pm 4.78}

22.50_{\pm 1.53}

48.62_{\pm 2.04}

47.33_{\pm 3.74}

2.77_{\pm 3.59}

28.95_{\pm 3.57}

IMR

58.03_{\pm 2.02}

58.32_{\pm 1.91}

5.24_{\pm 5.12}

14.86_{\pm 2.40}

54.18_{\pm 1.13}

54.22_{\pm 1.29}

9.78_{\pm 9.00}

22.03_{\pm 2.13}

48.88_{\pm 1.66}

48.07_{\pm 2.54}

15.30_{\pm 9.15}

26.20_{\pm 1.34}

Ada2I

SMR

72.07_{\pm 1.00}

72.01_{\pm 1.08}

17.45_{\pm 1.54}

19.69_{\pm 4.27}

71.39_{\pm 1.39}

71.44_{\pm 1.41}

17.05_{\pm 4.25}

27.07_{\pm 5.26}

71.75_{\pm 1.84}

71.78_{\pm 1.77}

22.02_{\pm 7.26}

32.42_{\pm 6.46}

Table 5. Performance comparison under different bimodal combinations (A: Audio, V: Visual, L: Language) and missing-rate settings, including Shared Missing Rate (SMR) and Imbalanced Missing Rate (IMR).

Figure 6. Performance of baselines on IEMOCAP under extreme IMR=(0.4,0.8,0.9).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Dialogues			Utterances
Dataset	train	valid	test	train	valid	test
IEMOCAP	120		31	5810		1623
CMU-MOSI	52	10	31	1284	229	686
CMU-MOSEI	2249	300	676	16326	1871	4659
CH-SIMS	60	59	59	1,368	456	457

Appendix

Table 6. Statistical overview of the MER and MSA datasets.

Method	Setting	0.3 / (0.1, 0.2, 0.6)			0.5 / (0.2, 0.5, 0.8)			0.7 / (0.4, 0.8, 0.9)
Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	SMR	$77.72_{\pm 0.47}$	$0.748_{\pm 0.022}$	$0.641_{\pm 0.014}$	$73.64_{\pm 2.82}$	$0.814_{\pm 0.015}$	$0.558_{\pm 0.004}$	$69.60_{\pm 1.42}$	$0.891_{\pm 0.032}$	$0.447_{\pm 0.041}$
MMIN	IMR	$71.80_{\pm 0.50}$	$0.855_{\pm 0.021}$	$0.497_{\pm 0.026}$	$68.97_{\pm 0.27}$	$0.890_{\pm 0.005}$	$0.427_{\pm 0.010}$	$68.00_{\pm 0.36}$	$0.916_{\pm 0.002}$	$0.392_{\pm 0.011}$
GCNet	SMR	$80.22_{\pm 1.10}$	$0.706_{\pm 0.024}$	$0.692_{\pm 0.023}$	$76.84_{\pm 0.67}$	$0.773_{\pm 0.016}$	$0.621_{\pm 0.018}$	$76.34_{\pm 0.96}$	$0.781_{\pm 0.014}$	$0.599_{\pm 0.016}$
GCNet	IMR	$76.74_{\pm 0.86}$	$0.776_{\pm 0.016}$	$0.616_{\pm 0.016}$	$73.89_{\pm 0.45}$	$0.828_{\pm 0.010}$	$0.541_{\pm 0.019}$	$73.19_{\pm 0.63}$	$0.839_{\pm 0.013}$	$0.525_{\pm 0.021}$
RedCore	SMR	$77.20_{\pm 0.52}$	$0.760_{\pm 0.007}$	$0.631_{\pm 0.009}$	$72.57_{\pm 0.79}$	$0.845_{\pm 0.012}$	$0.535_{\pm 0.015}$	$68.10_{\pm 0.44}$	$0.902_{\pm 0.012}$	$0.443_{\pm 0.014}$

Appendix

Table 7. Performance comparison on CMU-MOSEI

Method	Setting	0.3 / (0.1, 0.2, 0.6)			0.5 / (0.2, 0.5, 0.8)			0.7 / (0.4, 0.8, 0.9)
Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	SMR	$78.19_{\pm 0.26}$	$0.480_{\pm 0.012}$	$0.596_{\pm 0.026}$	$72.21_{\pm 1.86}$	$0.544_{\pm 0.009}$	$0.485_{\pm 0.028}$	$67.26_{\pm 2.28}$	$0.594_{\pm 0.034}$	$0.330_{\pm 0.097}$
MMIN	IMR	$71.80_{\pm 0.50}$	$0.509_{\pm 0.010}$	$0.542_{\pm 0.031}$	$68.97_{\pm 0.27}$	$0.621_{\pm 0.046}$	$0.212_{\pm 0.190}$	$68.00_{\pm 0.36}$	$0.603_{\pm 0.013}$	$0.318_{\pm 0.036}$
GCNet	SMR	$66.03_{\pm 0.59}$	$0.628_{\pm 0.008}$	$0.234_{\pm 0.024}$	$65.97_{\pm 1.45}$	$0.645_{\pm 0.010}$	$0.168_{\pm 0.041}$	$65.10_{\pm 0.66}$	$0.644_{\pm 0.005}$	$0.163_{\pm 0.022}$
GCNet	IMR	$64.94_{\pm 0.96}$	$0.648_{\pm 0.009}$	$0.161_{\pm 0.055}$	$64.77_{\pm 0.67}$	$0.648_{\pm 0.002}$	$0.152_{\pm 0.026}$	$65.55_{\pm 0.43}$	$0.649_{\pm 0.002}$	$0.140_{\pm 0.024}$
RedCore	SMR	$75.25_{\pm 1.57}$	$0.517_{\pm 0.024}$	$0.550_{\pm 0.053}$	$70.82_{\pm 1.49}$	$0.561_{\pm 0.012}$	$0.445_{\pm 0.023}$	$66.95_{\pm 1.39}$	$0.596_{\pm 0.016}$	$0.354_{\pm 0.040}$

Appendix

Table 8. Performance comparison on CH-SIMS

Method

(0.1, 0.6, 0.2)

(0.6, 0.1, 0.2)

(0.2, 0.8, 0.5)

(0.8, 0.2, 0.5)

(0.4, 0.9, 0.8)

(0.9, 0.4, 0.8)

MMIN

66.04_{\pm 2.63}

66.34_{\pm 2.60}

68.95_{\pm 0.65}

69.05_{\pm 0.69}

57.31_{\pm 0.88}

57.46_{\pm 0.89}

60.35_{\pm 1.61}

60.22_{\pm 1.26}

52.32_{\pm 2.17}

52.38_{\pm 2.09}

53.72_{\pm 0.50}

53.74_{\pm 0.46}

GCNet

74.61_{\pm 2.28}

74.91_{\pm 2.04}

77.14_{\pm 2.63}

76.87_{\pm 2.94}

68.03_{\pm 4.52}

68.31_{\pm 4.45}

76.52_{\pm 2.09}

76.33_{\pm 2.12}

65.24_{\pm 0.56}

65.58_{\pm 0.43}

72.52_{\pm 3.67}

72.42_{\pm 3.43}

RedCore

65.94_{\pm 0.27}

66.11_{\pm 0.25}

65.94_{\pm 0.56}

66.12_{\pm 0.57}

60.05_{\pm 0.74}

60.27_{\pm 0.72}

59.97_{\pm 2.73}

60.06_{\pm 2.40}

52.61_{\pm 1.42}

52.52_{\pm 1.49}

53.34_{\pm 2.46}

53.56_{\pm 2.35}

MCE

68.73_{\pm 0.84}

68.84_{\pm 0.81}

69.54_{\pm 0.62}

69.49_{\pm 0.55}

59.54_{\pm 1.20}

59.67_{\pm 1.25}

62.47_{\pm 0.89}

62.46_{\pm 0.77}

53.93_{\pm 0.81}

53.92_{\pm 0.86}

55.43_{\pm 1.55}

55.18_{\pm 1.47}

Ada2I

80.12_{\pm 1.03}

80.31_{\pm 0.98}

77.89_{\pm 0.53}

78.00_{\pm 0.47}

76.92_{\pm 1.73}

77.14_{\pm 1.69}

74.48_{\pm 1.22}

74.36_{\pm 1.38}

73.70_{\pm 2.12}

73.62_{\pm 2.74}

71.60_{\pm 2.68}

71.32_{\pm 3.10}

Appendix

Table 9. Performance comparison on IEMOCAP under different imbalanced missing rates

Method	Setting	Acc-2( $\uparrow$ )	MAE( $\downarrow$ )	Corr( $\uparrow$ )
MMIN	(0.1,0.6,0.2)	$74.13_{\pm 0.72}$	$1.009_{\pm 0.019}$	$0.635_{\pm 0.010}$
	(0.6,0.1,0.2)	$73.73_{\pm 0.96}$	$1.076_{\pm 0.097}$	$0.622_{\pm 0.022}$
	(0.2,0.8,0.5)	$67.02_{\pm 0.94}$	$1.191_{\pm 0.024}$	$0.514_{\pm 0.001}$
	(0.8,0.2,0.5)	$65.65_{\pm 0.31}$	$1.210_{\pm 0.028}$	$0.493_{\pm 0.006}$
	(0.4,0.9,0.8)	$52.03_{\pm 7.26}$	$1.397_{\pm 0.077}$	$0.263_{\pm 0.166}$
	(0.9,0.4,0.8)	$52.38_{\pm 7.18}$	$1.417_{\pm 0.060}$	$0.248_{\pm 0.156}$

Appendix

Table 10. Performance comparison on CMU-MOSI under different imbalanced missing rates

Method	Setting	0.3 / (0.1, 0.2, 0.6)		0.5 / (0.2, 0.5, 0.8)		0.7 / (0.4, 0.8, 0.9)
Method	Setting	$\mathrm{MEI}_{\mathrm{UA}}$	MLI	$\mathrm{MEI}_{\mathrm{UA}}$	MLI	$\mathrm{MEI}_{\mathrm{UA}}$	MLI
GCNet	SMR	$74.25_{\pm 8.58}$	$33.63_{\pm 2.97}$	$82.90_{\pm 7.13}$	$41.05_{\pm 1.84}$	$89.51_{\pm 7.02}$	$44.12_{\pm 3.99}$
GCNet	IMR	$62.83_{\pm 15.95}$	$32.86_{\pm 3.65}$	$63.82_{\pm 6.30}$	$37.65_{\pm 2.67}$	$74.61_{\pm 15.01}$	$39.74_{\pm 2.49}$
RedCore	SMR	$80.19_{\pm 2.73}$	$33.92_{\pm 1.27}$	$80.52_{\pm 4.42}$	$38.63_{\pm 2.29}$	$83.62_{\pm 0.75}$	$43.25_{\pm 2.43}$
RedCore	IMR	$80.65_{\pm 1.39}$	$35.00_{\pm 2.10}$	$84.36_{\pm 3.72}$	$40.40_{\pm 3.51}$	$84.83_{\pm 1.73}$	$42.86_{\pm 4.67}$
Ada2I	SMR	$86.82_{\pm 5.45}$	$15.22_{\pm 1.21}$	$80.65_{\pm 10.39}$	$13.41_{\pm 2.22}$	$89.86_{\pm 7.60}$	$12.79_{\pm 1.48}$

Appendix

Table 11.

\text{MEI}_{\text{UA}}

/MLI(%) of representatives from each method-family on CMU-MOSEI.

Method	AV			AL			VL
Method	WA	F1	MEI UA	WA	F1	MEI UA	WA	F1	MEI UA
GCNet	$68.61_{\pm 1.92}$	$69.19_{\pm 1.56}$	$35.05_{\pm 21.01}$	$74.62_{\pm 2.71}$	$74.83_{\pm 2.75}$	$6.80_{\pm 6.16}$	$78.31_{\pm 1.40}$	$78.04_{\pm 1.26}$	$62.24_{\pm 20.04}$
RedCore	$72.93_{\pm 0.95}$	$72.89_{\pm 0.92}$	$13.75_{\pm 1.39}$	$80.61_{\pm 2.18}$	$80.84_{\pm 2.09}$	$8.62_{\pm 7.84}$	$78.69_{\pm 2.85}$	$78.50_{\pm 2.94}$	$39.77_{\pm 29.06}$
Ada2I	$62.34_{\pm 0.67}$	$62.47_{\pm 0.74}$	$9.55_{\pm 6.20}$	$65.51_{\pm 1.49}$	$65.53_{\pm 1.57}$	$71.70_{\pm 14.28}$	$68.73_{\pm 0.74}$	$68.73_{\pm 0.64}$	$45.48_{\pm 19.01}$

Appendix

Table 12. Performance comparison under different bimodal combinations (A: Audio, V: Visual, L: Language) on non-missing data

Figure 7. Estimated training time (sec/epoch) of the baselines.

Figure 8. MEI evaluated by different task-specific metrics UA/WA/F1 during the training of GCNet on IEMOCAP.

Panel of Figure 9 — Figure 9. Diagnosis of GCNet on a different IEMOCAP train/valid split from Figure 5 .

Panel of Figure 10 — Figure 10. Diagnosis of RedCore on IEMOCAP.

Panel of Figure 11 — Figure 11. Diagnosis of Ada2I on IEMOCAP.

Panel of Figure 12 — Figure 12. Diagnosis of RedCore on CMU-MOSI.

Panel of Figure 13 — Figure 13. Diagnosis of Ada2I on CMU-MOSI.

Panel of Figure 14 — Figure 14. Diagnosis of GCNet on CMU-MOSI.

Explore similar work

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing
Jan 31, 2026Ronghao Lin, Honghao Lu, Ruixing Wu +5Missing-Modality Learning Affective Computing
Sun Yat-Sen University Guangzhou, China · South China Normal University Guangzhou, China · Sun Yat-Sen University Pazhou Laboratory Guangzhou, Guangdong, China
The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods
Sep 10, 2026Ioanna Kaffeza, Efthymios Georgiou, Alexandros PotamianosModality Imbalance Multimodal Optimization
PERSEE Center, Mines Paris-PSL University, Sophia-Antipolis, France · National Technical University of Athens, Athens, Greece · University of Bern, Bern, Switzerland +2
Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities
Apr 7, 2026Rongfei Chen, Tingting Zhang, Xiaoyu Shen +1Multimodal Sentiment Analysis Missing-Modality Learning
School of Computer Science and Technology, Eastern Institute of Technology, Ningbo, China · School of Mechatronic Engineering and Automation, Shanghai University, China

Jan 31, 2026J/K move · Enter open · S save

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

Ronghao Lin, Honghao Lu, Ruixing Wu, Aolin Xiong+4

Sun Yat-Sen University Guangzhou, China · South China Normal University Guangzhou, China · Sun Yat-Sen University Pazhou Laboratory Guangzhou, Guangdong, China

Missing-Modality Learning Affective Computing Multimodal Robustness Multimodal Model Evaluation

Open paper AskPDF arXiv

Current Multimodal Affective Computing (MAC) systems heavily rely on the completeness of multiple modalities to accurately understand human's affective state. However, in real-world scenarios, the availability of modality data is often dynamic and uncertain, leading to substantial performance fluctuations due to the distribution shifts and semantic deficiencies of the incomplete multimodal inputs. Known as the missing modality issue, this challenge poses a critical barrier to the robustness and practical deployment of MAC models. To systematically quantify this issue, we introduce MissMAC-Bench, a comprehensive benchmark designed to establish fair and unified evaluation standards from the perspective of cross-modal synergy. Two guiding principles are proposed, including no missing prior during training, and one single model capable of handling both complete and incomplete modality scenarios, thereby ensuring better generalization. Moreover, to bridge the gap between academic research and real-world applications, our benchmark integrates evaluation protocols with both fixed and random missing patterns at the dataset and instance levels. Extensive experiments conducted on 3 widely-used language models across 4 datasets validate the effectiveness of diverse MAC approaches in tackling the missing modality issue. Our benchmark provides a solid foundation for advancing robust MAC and promotes the development of multimedia data mining. Our code is released in https://github.com/RH-Lin/MissMAC-Bench.

Abstract

Explore similar work

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods

Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities