Reliability-Aware Multimodal Sentiment Prediction Under Local Modality Missingness
Authors: Minghao Kong, Deng Yao, Xi Zhang, Ying Gao, Jiurun Chen, Zirong Liu, Rongjie Wang
Organizations: School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen, China · Department of Network Intelligence, Pengcheng Laboratory, Shenzhen, China · School of Mathematics, Sun Yat-sen University, Guangzhou, China · School of Physics and Astronomy, Sun Yat-sen University, Zhuhai, China
Multimodal sentiment models are usually evaluated on complete utterances, yet streams often lose a contiguous span of text, audio, or video. We study local missingness on aligned CMU-MOSEI features. Our reliability-aware network (RM-Net) keeps the original temporal positions, represents unobserved positions with a learned token, outputs bounded per-modality intensity contributions gated by observation quality, and is trained with dual-view masking of contiguous segments. We fuse five RM-Net seeds with a LoRA-adapted BERT text expert, using a fusion weight that depends only on the visible fraction of content tokens. Architecture, checkpoint, and fusion choices were made on the validation split. On the held-out split, the fused model reaches a macro-F1 of 0.655 on clean inputs and 0.634 under 40% mixed missingness; its two-view average exceeds the auxiliary-head baseline by 0.024 (95% video-cluster bootstrap interval [0.003, 0.045]). Because this split was inspected during development, these numbers are not a blind test. For explanations, exact Shapley attributions over the eight modality subsets and position-level occlusion show that top-ranked positions reduce the predicted-class probability faster than random positions (AUC gap 0.113 on validation). We also describe an auditable word-level alignment pipeline for raw clips; its outputs do not feed the predictive models.
Figures & tables
Figure 1: Overview of the fused predictor. Each branch is evaluated under the same observation masks, and the fusion weights depend only on how much text remains visible.
Configuration
Val clean
Val mixed
Val avg
Held-out clean
Held-out mixed
RM-Net auxiliary head
0.6144
0.6057
0.6101
0.6237
0.6178
RM-Net, CE weight 1, 15 seeds, bias
0.6260
0.6277
0.6269
0.6187
0.6071
RM-Net + frozen BERT probe (15 seeds), bias
0.6360
0.6367
0.6363
0.6326
0.6284
LoRA text expert only
0.6175
0.6269
0.6222
0.6578
0.6524
RM-Net + LoRA, no bias
0.6351
0.6304
0.6328
0.6600
0.6496
RM-Net + LoRA, with bias (final)
0.6339
0.6394
0.6367
0.6550
0.6341
Table 1: Macro-F1 of the configurations compared during development. The final row was selected on the validation average. Held-out columns were inspected during development and are not a blind test. The frozen-probe row uses a core trained with CE weight 1 and 15 seeds, whereas the final model uses five core seeds.
Clean
Mixed40
Metric
Core
Fused
Core
Fused
Acc
0.6479
0.6809
0.6424
0.6630
Macro-F1
0.6237
0.6550
0.6178
0.6341
MAE
0.6344
0.6099
0.6379
0.6210
r
0.6644
0.6942
0.6677
0.6861
Table 2: Held-out results (727 utterances). Core is RM-Net with its auxiliary classification head and regression output; Fused is RM-Net + LoRA.
Δ Macro-F1 (points)
Δ MAE (reduction)
Subset
20%
40%
60%
20%
40%
60%
T
+0.74
+0.69
− 0.30
0.017
0.012
0.006
A
+3.15
+3.33
+2.87
0.024
0.024
0.023
V
+3.32
+3.41
+3.06
0.024
0.022
0.022
TA
+1.73
+1.31
+0.70
0.019
0.012
0.005
TV
+2.32
+1.85
+0.21
0.018
0.012
0.005
Table 3: Change from the RM-Net auxiliary head to the fused model on the 727-utterance held-out split, for 21 fixed missingness conditions. Positive Macro-F1 values and MAE reductions indicate improvement. Each cell is a single condition, not an independent sample.
Variant
Val MAE
RM-Net (full)
0.5970±0.0046
Clean-only training (no augmentation)
0.6065±0.0071
No consistency term
0.6023±0.0059
Concatenation MLP fusion
0.6026±0.0092
Text only
0.6014±0.0068
Table 4: Core-network ablations on the validation missingness grid (mean ± SD over five training seeds).
Figure 2: Deletion and retention curves of the fused model on the 728 validation utterances. Left panels delete the top-ranked fraction of content positions; right panels keep only that fraction. Top: change in intensity ∣Δy∣ ; bottom: decrease in the predicted-class probability. Solid blue lines use occlusion ranking; dashed orange lines average five random permutations.
Intensity
Class prob.
Delete
Retain
Delete
Retain
Selected
0.2235
0.1796
0.1370
− 0.0645
Random
0.1136
0.1659
0.0244
0.0365
Paired diff.
0.1100
0.0137
0.1126
− 0.1010
95% CI low
0.1010
0.0044
0.1081
− 0.1059
95% CI high
0.1201
0.0238
0.1174
− 0.0963
Table 5: Deletion and retention curves for the fused model on 728 validation utterances. Intervals come from 500 paired bootstrap resamples over utterances, keeping each utterance’s curve intact; they do not cluster by source video.
Figure 3: Error diagnostics of the fused model. (a) Held-out MAE by true-intensity group (727 utterances) for the RM-Net core and the fused model, under clean and 40% mixed-missing inputs. (b) Validation confusion matrix under 40% mixed missingness (rows: true class; columns: predicted class; percentages are row-normalized). (c) Main modality of the validation explanations (728 utterances each): intensity, text 563, vision 105, audio 60; class probability, text 681, vision 40, audio 7.
rank 8, α=16 , query and value projections of all 12 layers; batch 32
Appendix
Table 6: Training and evaluation settings. The fusion search used only the validation split; the held-out split had been inspected during development.
Split
N
Neg
Neu
Pos
Median length
Zero audio / visual frames
Train
3,395
967
758
1,670
22.0
0.082 / 0.132
Valid
728
206
184
338
23.5
0.081 / 0.131
Test
727
207
158
362
23.0
0.079 / 0.135
Appendix
Table 7: Aligned release (50 positions per modality). Neutral corresponds exactly to y=0 and is 22.3% of the training split.
Figure 4: RM-Net core under increasing missingness on the validation split. Each panel is one damaged-modality subset; the blue curve is Macro-F1 (historical thresholds) and the orange curve is MAE. Shading is the standard deviation across five training seeds, after averaging over placements within each seed.
Figure 5: MAE of the RM-Net core by damaged-modality subset (rows) and span position (columns) at missing fractions 0.3 (left) and 0.5 (right). Colour scales differ between the two panels.
Figure 6: RM-Net core under span length and placement. Left: MAE as the number of removed positions grows, by modality subset. Middle: reconstruction R2 on the removed positions for single-modality removal. Right: MAE of a three-position window as its relative start varies.
Case
True polarity / intensity
Polarity change
Intensity change
Text span
V1 ( Vuay0mVKkik_3 )
Neg / −2.0
Neg → Pos
−0.201→0.485
[2,14)
V2 ( fWOIAxzBQFY_7 )
Neu / 0.0
Neu → Pos
0.453→0.805
[12,21)
V3 ( 128059_6 )
Pos / 1.0
Pos → Neg
0.039→−0.729
[15,35)
Appendix
Table 8: Representative flips under mixed missingness on the validation split. Audio and visual spans are [4,16) and [15,27) for V1, [6,15) and [3,12) for V2, and [30,50) and ∅ for V3.
Grouping
Group 1
Group 2
Group 3
Group 4
Span position: head / middle / tail (40%)
0.0214
0.0165
0.0287
–
Span length: 1–4 / 5–8 / 9–12 / ≥ 13 (text)
0.0034
0.0141
0.0192
0.0343
Records per length group
352
658
516
658
Appendix
Table 9: Paired increase in validation MAE caused by text masking (masked minus clean). Position uses the 40% condition; length pools the 20%, 40%, and 60% conditions.
Group ( ∣y∣ )
n
Clean MAE
Mixed40 MAE
0 (Neutral)
184
0.352
0.348
0<∣y∣≤0.5 (weak)
147
0.391
0.399
0.5<∣y∣≤1.5 (moderate)
267
0.568
0.583
∣y∣>1.5 (strong)
130
1.057
1.064
Appendix
Table 10: Validation MAE of the fused model by true-intensity group.
Class
Precision (%)
Recall (%)
F1 (%)
Negative
65.6
72.3
68.8
Neutral
46.3
57.1
51.1
Positive
78.5
63.6
70.3
Appendix
Table 11: Per-class metrics of the fused model on the clean validation split.
Figure 7: Held-out replay of the unlabeled-set A zero spans. Bars show the Macro-F1 gain of the fused model over the RM-Net core for each pattern (sample number 01–30). Panel (b) also masks the [UNK] positions of the set. The dashed line is the mean gain.
Family
Configuration
Val avg
Test avg
Test Δ vs. frozen
Adopted
Frozen RM + LoRA fusion
baseline
0.636652
0.644564
0
Yes
Class-bias shrinkage
bias_shrink_0.5
0.629462
0.646465
+0.001902
No
Quality-adaptive fusion
adaptive_l2_0.1
0.614761
0.650359
+0.005795
No
Quality fusion + restricted review
review_l2_0.01
0.626297
0.657514
+0.012950
No
Appendix
Table 12: Workflow candidates. Averages are over the clean and 40% mixed-missing Macro-F1 values. The test column is shown for transparency; it was inspected earlier and was not used to choose among candidates.
Configuration
Val clean
Val mixed
Val avg
Test clean
Test mixed
Test MAE clean
Test MAE mixed
RM-Net core (auxiliary head)
0.6144
0.6057
0.6101
0.6237
0.6178
0.6344
0.6379
RM-Net core + class bias
0.6169
0.6177
0.6173
0.6185
0.6189
0.6344
0.6379
BERT last-layer probe
0.5976
0.5935
0.5956
0.6280
0.6118
0.6870
0.6969
BERT multi-layer probe
0.6118
0.6007
0.6063
0.6315
0.6371
0.6950
0.7019
BERT + SVD probe
0.6125
0.5930
0.6027
0.6090
0.6161
0.7004
0.7032
BERT scalar-mix probe (5 seeds)
0.6074
0.6165
0.6119
0.6209
0.6261
0.6436
0.6432
Appendix
Table 13: All evaluated candidates. Val and Test columns report Macro-F1 on the clean and 40% mixed-missing inputs; MAE is on the held-out split. Val avg is the mean of the two validation views. The bold row is the final configuration.
Euler angles from the facial transformation matrix (rad)
3
Geometry and dynamics
normalized face-box area, eye aspect ratio, mouth aspect ratio, relative rotation speed between valid frames, fraction of landmarks inside the frame
5
Appendix
Table 15: The 35 visual dimensions.
File
Text
Audio
Vision
L2 aligned (50 positions)
(100,50,768)
(100,50,74)
(100,50,35)
L2 non-aligned (500 blocks)
(100,50,768)
(100,500,74)
(100,500,35)
Appendix
Table 16: Output tensor shapes for the 100 clips.
Check or diagnostic
Result
Clips with all modality files and indices
100 / 100
Clips passing the interface and index checks
100 / 100
Acoustic NaN/Inf replacements
0
Frame-weighted face detection rate
81.97%
Word-level visual availability before the time gate
82.61%
Clips with face detection below 20% (of which none detected)
17 (13)
Appendix
Table 17: Quality checks on the 100 raw clips.
Grouping
Retention R
Mean non-empty groups
Word level
0.2489
20.25
0.25 s windows
0.3120
25.75
0.5 s windows
0.1848
14.03
1.0 s windows
0.1069
7.82
Appendix
Table 18: Acoustic variance retained by each grouping, averaged over the 65 clips with at least one gated word. R is a descriptive variance measure, not a sentiment accuracy.
Word
Time interval (s)
Audio frames
Video grid frames
BERT subwords
F0 (Hz)
Secondly,
[0.5625, 1.0425)
55–102
15–26
1, 2
211.00
another
[1.2625, 1.6225)
125–160
32–40
3
154.51
goal
[1.6825, 1.8825)
167–186
43–47
4
234.28
of
[1.9625, 2.0025)
195–198
50
5
219.63
exhibiting
[2.1025, 2.5825)
209–256
53–64
6
130.28
is
[2.7425, 2.8025)
273–278
69–70
7
196.16
Appendix
Table 19: Word-level mapping for the typical clip. The jaw-opening coefficient of each word, which is also stored with the outputs, is omitted here for space.
Figure 8: Word-level alignment for a typical clip ( -egA8-b7-3M_9 , Positive, 4.9 s). From top: waveform with CTC-estimated word spans; F0 and RMS; three MediaPipe blendshape scores; a projection of the BERT word features onto principal components; and per-word means of selected audio and visual channels. Shaded spans mark word boundaries, and the vertical axis of each panel is in its own units.
Figure 9: Quality summaries over the 100 raw clips: duration of the common timeline, words per clip, detected-face fraction, mean CTC posterior, and retained acoustic variance by grouping (65 clips with at least one gated word). The lower-right panel is an exploratory audio–visual lag histogram; it is not used to shift timelines.
Output target
Signed Spearman
Absolute Spearman
Mean absolute effect difference
Fused intensity
0.6680
0.4748
0.1707
Predicted-class probability
0.2903
0.3522
0.0785
Appendix
Table 20: Agreement between encoded masking and word deletion for the top-ranked word of each validation utterance (728 comparisons).
Case
True polarity / intensity
Predicted polarity
Predicted intensity
Margin
E1 ( f_ZJ7L14oYQ_21 )
Pos / 2.000
Neg
−1.681
0.940
E2 ( EQxkXiWAm08_8 )
Pos / 0.333
Neu
0.100
0.642
E3 ( x2lOwQaAn4g_4 )
Neu / 0.000
Pos
1.675
0.863
Appendix
Table 21: Three misclassified validation utterances, chosen for illustration. The probability margin is the gap between the top two fused probabilities.
Figure 10: Set B, sample 19: local occlusion effects on the classification target (probability of the originally predicted class). The title keeps the release name “Attachment 4 / 19”.
Figure 11: Set B, sample 19: local occlusion effects on the fused intensity. Vision is the main modality for this target. Colours are scaled within each modality, as in Figure 10 .
Set A
Polarity
Intensity
RM core
pneg
pneu
ppos
wcls
wreg
A01
Negative
-0.660
-0.470
0.697
0.185
0.118
0.80
0.60
A02
Neutral
+0.061
-0.052
0.199
0.647
0.154
0.54
0.40
A03
Neutral
+0.152
+0.048
0.187
0.558
0.255
0.57
0.43
A04
Positive
+0.398
+0.322
0.093
0.333
0.573
0.70
0.53
A05
Negative
-1.127
-1.494
0.849
0.091
0.061
0.80
0.60
A06
Positive
+0.710
+0.853
0.047
0.364
0.589
0.62
0.46
Appendix
Table 22: Predictions for unlabeled set A (30 utterances). Polarity and intensity are the fused outputs; RM core is the core intensity; p are the fused class probabilities; wcls and wreg are the classification and intensity text-expert weights. No labels are available.
Figure 12: Unlabeled set A. (a) Fused intensity, coloured by the predicted polarity, with the RM-Net core intensity as open circles. (b) Visible-text fraction f and the two text-expert weights for each utterance.
Set B
Polarity
Intensity
Main (class)
Main (intensity)
Class shares T/A/V (%)
Intensity shares T/A/V (%)
B01
Neutral
+0.369
text
text
65.4 / 19.4 / 15.2
89.6 / 6.2 / 4.2
B02
Positive
+0.250
text
text
66.3 / 20.8 / 12.9
70.6 / 18.1 / 11.3
B03
Negative
-0.350
text
text
78.7 / 8.9 / 12.5
80.1 / 18.6 / 1.3
B04
Negative
-0.358
text
text
75.5 / 9.9 / 14.7
58.6 / 20.7 / 20.7
B05
Neutral
+0.401
text
vision
66.9 / 12.3 / 20.8
19.0 / 19.1 / 61.9
B06
Positive
+0.755
text
text
79.0 / 1.5 / 19.5
68.5 / 13.5 / 17.9
Appendix
Table 23: Predictions and main modalities for unlabeled set B (20 utterances). Shares are the absolute Shapley values of each modality, normalized to 100% within each target.
Set B
Polarity
Main (class)
Top text evidence
Top audio evidence
Top vision evidence
B01
Neutral
text
when (1.98–2.10 s; estimated)
its (7.04–7.14 s; estimated)
ensuring (4.64–5.02 s; estimated)
B02
Positive
text
happiness (1.44–1.80 s; estimated)
want (0.60–0.70 s; estimated)
happiness (1.44–1.80 s; estimated)
B03
Negative
text
just (2.72–2.88 s; estimated)
account, (6.22–6.52 s; estimated)
take (7.04–7.22 s; estimated)
B04
Negative
text
not. (6.02–6.24 s; estimated)
team (1.28–1.44 s; estimated)
chances (2.90–3.24 s; estimated)
B05
Neutral
text
for (4.16–4.26 s; estimated)
for (4.16–4.26 s; estimated)
Wagon (4.64–4.92 s; estimated)
B06
Positive
text
impressive (3.96–4.40 s; estimated)
dance (4.46–4.70 s; estimated)
you’re (0.62–0.76 s; estimated)
Appendix
Table 24: Top-ranked evidence for unlabeled set B: the highest-ranked position for each modality, with its time interval and timing status. Time is in seconds from the common stream start; “candidate low trust” marks times that did not pass the time-quality gate.
Multimodal Sentiment Analysis (MSA) integrates text, audio, and vision to infer human affect, yet real-world multimodal observations are often incomplete. Existing methods for incomplete-observation MSA mainly follow two paradigms. Reconstruction-based methods recover missing information from observed modalities, while joint-representation methods learn directly from incomplete inputs. Although effective, these methods usually treat modality reliability only implicitly within representation learning or fusion design rather than modeling it explicitly. We argue that modality reliability is a central variable in incomplete-observation settings. Failure to model it explicitly gives rise to two related issues. The first is reliability mismatch, in which the affective evidence retained by each modality varies across samples and missing rates. The second is reliability propagation bias, in which messages from degraded modalities may adversely affect cross-modal interaction and predictive performance. To address these issues, we propose MRCF, a Modality Reliability-Calibrated Framework for MSA with incomplete observations. MRCF contains a Reliability-Aware Branch that estimates sample-specific modality reliability from intramodal quality cues and cross-modal semantic consistency, a Reliability-Guided Interaction Branch that uses the estimated scores to modulate cross-modal information flow, and a Reliability-Calibrated Fusion Module that integrates reliability and semantic cues for final prediction. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that MRCF achieves strong performance under standard incomplete-observation protocols. Further analyses provide evidence that explicit reliability modeling helps mitigate reliability mismatch and reliability propagation bias during interaction and fusion.
Chunlei Meng, Jacqueline J. Pang, Pengbin Feng +3
The College of Intelligent Robotics and Advanced Manufacturing (CIRAM), Fudan University · Cornell University · University of Southern California (USC) +1
Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.
Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)
The missing modality problem poses a fundamental challenge in multimodal sentiment analysis, significantly degrading model accuracy and generalization in real world scenarios. Existing approaches primarily improve robustness through prompt learning and pre trained models. However, two limitations remain. First, the necessity of generating missing modalities lacks rigorous evaluation. Second, the structural dependencies among multimodal prompts and their global coherence are insufficiently explored. To address these issues, a Prompt based Missing Modality Adaptation framework is proposed. A Missing Modality Evaluator is introduced at the input stage to dynamically assess the importance of missing modalities using pretrained models and pseudo labels, thereby avoiding low quality data imputation. Building on this, a Modality invariant Prompt Disentanglement module decomposes shared prompts into modality specific private prompts to capture intrinsic local correlations and improve representation quality. In addition, a Dynamic Prompt Weighting module computes mutual information based weights from cross attention outputs to adaptively suppress interference from missing modalities. To enhance global consistency, a Multi level Prompt Dynamic Connection module integrates shared prompts with self attention outputs through residual connections, leveraging global prompt priors to strengthen key guidance features. Extensive experiments on three public benchmarks, including CMU MOSI, CMU MOSEI, and CH SIMS, demonstrate that the proposed framework achieves state of the art performance and stable results under diverse missing modality settings. The implementation is available at https://github.com/rongfei-chen/ProMMA
Rongfei Chen, Tingting Zhang, Xiaoyu Shen +1
School of Computer Science and Technology, Eastern Institute of Technology, Ningbo, China · School of Mechatronic Engineering and Automation, Shanghai University, China