Multimodal Sentiment Analysis (MSA) seeks to infer human emotions by integrating textual, acoustic, and visual cues. However, existing approaches often rely on all modalities are completeness, whereas real-world applications frequently encounter noise, hardware failures, or privacy restrictions that result in missing modalities. There exists a significant feature misalignment between incomplete and complete modalities, and directly fusing them may even distort the well-learned representations of the intact modalities. To this end, we propose PRLF, a Progressive Representation Learning Framework designed for MSA under uncertain missing-modality conditions. PRLF introduces an Adaptive Modality Reliability Estimator (AMRE), which dynamically quantifies the reliability of each modality using recognition confidence and Fisher information to determine the dominant modality. In addition, the Progressive Interaction (ProgInteract) module iteratively aligns the other modalities with the dominant one, thereby enhancing cross-modal consistency while suppressing noise. Extensive experiments on CMU-MOSI, CMU-MOSEI, and SIMS verify that PRLF outperforms state-of-the-art methods across both inter- and intra-modality missing scenarios, demonstrating its robustness and generalization capability.
Figures & tables
Figure 1: Feature information loss and feature phase shift caused by missing modalities.
Figure 2: The structure of PRLF, which consists of two key components: Adaptive Modality Reliability Estimator (AMRE) and Progressive Interaction module (ProInteract).
Figure 3: Analysis of confidence and Fisher information variations caused by missing key frames in the visual modality.
Dataset
Models
{l}
{a}
{v}
{l,a}
{l,v}
{a,v}
Avg.
{l,a,v}
MOSI
Self-MM [ 45 ]
67.80
40.95
38.52
69.81
74.97
47.12
56.53
84.64
MISA [ 11 ]
81.17
43.51
49.24
80.99
81.49
49.42
64.3
83.36
TETFN [ 31 ]
81.04
39.32
39.32
81.09
80.99
39.32
60.18
82.33
MMIM [ 10 ]
81.17
36.52
31.53
81.78
81.02
37.63
58.28
82.28
UMDF [ 21 ]
82.92
67.80
59.92
85.63
84.09
72.98
75.56
83.36
HRLF [ 20 ]
83.36
69.47
64.59
83.82
83.56
75.62
76.74
84.15
Table 1: Performance comparison on CMU-MOSI and CMU-MOSEI datasets under different missing-modality conditions.
Models
Testing Conditions
{l}
{a}
{v}
{l,a}
{l,v}
{a,v}
Avg.
MISA [ 11 ]
75.15
56.95
56.82
77.07
75.17
56.78
66.32
Self-MM [ 45 ]
67.11
70.36
72.85
73.71
76.76
78.00
73.13
MMIM [ 10 ]
73.18
57.70
57.86
73.43
72.96
57.76
65.48
TETFN [ 31 ]
79.19
56.82
56.82
79.19
79.16
56.82
68
ALMT [ 51 ]
76.17
81.08
81.91
75.75
76.36
81.91
78.86
Table 2: Performance comparison on SIMS under different missing conditions
Figure 4: Comparison results of intra-modality missingness. We report the F1 score.
Iteration
Missing rate p
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Step=1
81.14
79.69
75.44
72.25
71.09
69.58
64.71
60.73
59.91
55.37
Step=2
83.08
80.39
76.87
73.88
72.34
70.65
65.82
62.06
61.15
56.39
Step=3
85.21
81.29
78.06
75.28
73.96
71.97
67.75
64.62
62.23
57.59
Step=4
85.78
81.65
77.99
75.89
74.93
73.05
68.5
65.31
63.09
58.64
Step=5
84.49
81.08
77.29
75.19
73.78
71.24
67.27
64.16
62.33
57.48
Table 3: Iteration Step Ablation results for the intra-modality missingness on MOSI.
Figure 5: Ablation for the intra-modality missingness on MOSI.
Iteration
Testing Conditions
{l}
{a}
{v}
{l,a}
{l,v}
{a,v}
Avg.
{l,a,v}
Step=1
79.62
64.33
59.68
80.47
80.06
69.57
72.29
81.14
Step=2
81.49
66.87
62.34
82.65
82.33
72.64
74.72
83.08
Step=3
82.91
68.73
63.11
84.36
84.27
75.58
76.49
85.21
Step=4
83.82
69.63
64.05
84.98
84.13
76.03
77.11
85.78
Step=5
82.47
68.51
62.77
83.54
82.61
74.88
75.80
84.49
Table 4: Iteration Step Ablation results for the inter-modality missingness on MOSI.
The College of Intelligent Robotics and Advanced Manufacturing (CIRAM), Fudan University · Cornell University · University of Southern California (USC) +1
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Zhongguancun Academy, Beijing 100094, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China +1