Organizations: Shandong University · The Hong Kong Polytechnic University · National University of Singapore · City University of Hong Kong · Shenzhen Loop Area Institute
Is a face or a voice still heard once the words are known? Incomplete multimodal learning increasingly uses large language models (LLMs) as fusion engines: modality features enter as token blocks and missing modalities become placeholders. We find that LLM-driven fusion can override non-verbal evidence it can read. Contrasting its full-input prediction with counterfactual views, the same model shown one modality or no content, we identify language shortcuts. We build MR-Bench: MR-Bench-A types every test sample of four sentiment benchmarks; MR-Bench-H contains 900 human-validated clips (620 language-shortcut, LS; 280 non-verbal-shortcut, NS) from seven sources, and every model is scored on all of them. Zero-shot MLLMs solve at most 32.7% of the LS clips; adding the transcript to the muted video lowers vision-language accuracy by 46.1-53.1 points. Yet inverting a text-only model's answers solves 94.4%, so LS accuracy alone cannot establish use of non-verbal evidence. Inside the fusion LLM, fused states align increasingly with the text view late in the decoder. Counterfactual-View Reliability learning (CoVeR) trains views as unimodal judges and withholds fused supervision where they disagree. It achieves state-of-the-art results on five benchmarks without extra inference cost in accuracy mode. On MR-Bench-H, it reaches 61.7% LS and 93.6% NS accuracy; robust mode raises LS accuracy to 63.0% while lowering NS accuracy to 84.8%. Among LS clips solved without text, adding text causes errors on 30.1% with CoVeR versus 32.2% with LLM-Fusion. Project page: https://peacelwh.github.io/CoVeR/.
Figures & tables
Figure 2: Constructing MR-Bench. Candidates are clips whose unimodal judgments disagree with the words (here a CH-SIMS clip with human unimodal labels on [−1,1] ); a panel of four text-only LLMs keeps a language-shortcut (LS) candidate only if the majority misreads its transcript, and three annotators keep a clip only if the majority recovers the label from the full clip and the misleading polarity from the shortcut channel. Models are then asked the same binary question from every input they support.
Figure 3: MR-Bench-H results on all 620 LS clips of the released set (seven sources; Tab. S5 ). T: transcript, A: audio, V: video. Fusion models are trained on the training split of each source and averaged over three seeds; they inherit its label prior (dotted: the constant answer “negative”, 61.9%), so they are compared within their family. Every zero-shot MLLM is far below chance; inverting a text-only model’s answers alone reaches 94.4%, so LS accuracy is read together with the 280 NS clips.
Figure 4: Where and how the decision collapses. (a–c) Prior-centred reliance ρjℓ ( Eq. 2 ) of the fused state on the text-, audio- and visual-only views across the decoder layers of LLM-Fusion, on conflict (solid) and consensus (dashed) cases. (d) AUROC for separating conflict from consensus cases with the hidden-state discrepancy 1−cos(hf,ht) that gates activation steering and with the prediction-level disagreement between the audio/visual and the text views. Mean over 3 seeds.
Figure 5: Three ways to fuse. LLM-Fusion supervises the fused prediction on every sample (left); the test-time override lets confidently disagreeing views of the frozen model replace the fused decision (middle, Eqs. 3 and 4 ); CoVeR trains the views as unimodal judges and withholds the fused task loss wherever a judge disagrees with the text judge (right), running the full view alone at test time.
Method
T-free
+Views
CMU-MOSI
CH-SIMSv2
Corr ↑
Cons. ↑
Conf. ↑
SE ↓
Corr ↑
Cons. ↑
Conf. ↑
SE ↓
LLM-Fusion
–
–
50.7
94.1
21.7
78.3
59.2
93.7
58.6
41.4
+ discrepancy-gated steering
✓
✓
50.3
93.8
22.2
77.8
58.9
93.6
58.6
41.4
+ test-time override ( τtest=0.3 )
✓
✓
47.8
92.5
33.9
66.1
55.3
86.8
72.3
27.7
+ learned reliability gate
✗
✓
50.1
92.5
19.4
80.6
58.6
93.1
56.4
43.6
+ conflict re-weighting
✗
✗
50.4
93.6
24.4
75.6
58.1
93.0
52.3
47.7
Table 1: Correcting the collapse on MR-Bench-A. Avg Corr ( ×100 ), Acc2 on consensus (Cons.) and conflict (Conf.) cases and ShortErr (SE), in %. T-free: no training; +Views: extra test-time pass over the views. Mean over 5 seeds; details in the supplementary material.
Method
Venue
T-Miss
CMU-MOSI
CMU-MOSEI
CH-SIMS
CH-SIMSv2
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
MulT [ 42 ]
ACL’19
✓
68.6
65.5
1.160
0.521
73.9
68.9
0.706
0.532
72.6
69.6
0.510
0.414
73.8
71.5
0.393
0.534
MISA [ 19 ]
MM’20
✓
67.0
65.4
1.166
0.479
74.4
72.8
0.702
0.549
72.6
68.7
0.520
0.463
73.6
72.7
0.397
0.550
MMIN [ 60 ]
ACL’21
✗
67.7
67.0
1.205
0.504
76.3
74.8
0.690
0.574
74.6
74.4
0.502
0.508
75.2
74.3
0.388
0.577
TFR-Net [ 55 ]
MM’21
✗
67.6
63.8
1.140
0.501
73.1
67.0
0.696
0.515
73.7
71.3
0.508
0.462
74.6
73.5
0.382
0.565
IMDer [ 45 ]
NeurIPS’23
✗
68.3
65.9
1.151
0.506
75.2
70.7
0.675
0.567
75.3
72.4
0.495
0.478
75.7
75.3
0.371
0.581
Table 2: Comparison with the state of the art under incomplete modalities , averaged over the seven test-time availability patterns (Acc2 and F1 in %). T-Miss ✗: the features of missing modalities serve as reconstruction or distillation targets. Best / second / third highlighted; mean over 3 seeds. Deltas use displayed means and are approximate.
Variant
Avg Corr ↑
Conf. ↑
SE ↓
LLM-Fusion
49.6
38.7
61.3
+ unimodal loss, no gate
52.2
41.3
58.7
+ unimodal loss + gate (CoVeR)
55.1
47.7
52.3
Table 3: Supervision ablation on CH-SIMS. Accuracy mode; Avg Corr ( ×100 ) across seven patterns, conflict Acc2 and ShortErr (SE) on MR-Bench-A (%). Mean over five seeds; the LLM-Fusion row is the three-seed reference of Tabs. 2 and S3 .
Method
Targets
Deviations
MulT [ 42 ]
–
none (zero imputation)
MISA [ 19 ]
–
utterance vectors by mean pooling (no padding)
MMIN [ 60 ]
✓
full-modality teacher (first 25% of epochs) then imagination; official IEMOCAP settings; 80 epochs (22 on CMU-MOSEI) and batch 32–128 to match the official number of updates, since 30 epochs leave the teacher under-trained
TFR-Net [ 55 ]
✓
missing positions from the pattern (whole modality) instead of random frames
IMDer [ 45 ]
✓
score-based recovery per sample from the observed modalities (official 100 sampling steps); trained from scratch instead of from the official complete-data checkpoint
DiCMoR [ 44 ]
✓
flow-based recovery per sample from the observed modalities; trained from scratch instead of from the official complete-data checkpoint
Table S1 : Baseline ports. Complete-modality targets used in training, and deviations from the official code beyond the common protocol.
Table S3 : MR-Bench-A and the conflict patterns of LLM-Fusion. Test-split consensus (Cons.) and conflict (Conf.) counts and, with all modalities observed, Acc2 on both case types, ShortErr (SE) and conflict Acc2 of the same model without text (w/o t ) and of its visual-only view (by Vv ), in %; mean over 3 seeds.
Source
Lang.
Candidates
Reader panel
Validated
LS
NS
Ctrl
LS
NS
Ctrl
LS
NS
Ctrl
CMU-MOSI [ 56 ]
EN
83
135
22
23
130
21
14
119
0
CMU-MOSEI [ 3 ]
EN
139
177
32
53
175
28
23
119
0
CH-SIMS [ 53 ]
ZH
48
26
7
38
22
7
24
16
0
CH-SIMSv2 [ 29 ]
ZH
96
42
14
74
36
14
49
26
0
MUStARD++ [ 38 ]
EN
101
–
10
69
–
10
53
–
0
Table S4 : MR-Bench-H per source : candidates after stage 1, items passing the reader panel (stage 2), and items retained after human validation (stage 3). Validated columns report the final retained counts in the release package. Controls are tracked separately and excluded from the released test set. LS / NS / controls.
Figure S1 : Sample clips of MR-Bench-H (frames at 20/50/80% of each clip; Chinese transcripts with an English gloss). LS: the words suggest the polarity opposite to the label; NS: the face does. Marks show whether a reader answers the label: Qwen2.5-7B on the transcript (T), InternVL3-8B on the muted video (V) and on the video with its transcript (V+T), Qwen2.5-Omni-7B on the full clip (V+A).
Model
Input
LS ↑
95% CI
NS ↑
SE ↓
Zero-shot MLLMs
Qwen2.5-Omni-7B
V+A
28.7
[25.3, 32.4]
94.6
71.3
A
21.8
[18.7, 25.2]
97.9
78.2
V
53.9
[49.9, 57.8]
33.2
46.1
T
3.1
[2.0, 4.7]
99.3
96.8
Qwen2.5-Omni-3B
V+A
25.8
[22.5, 29.4]
92.5
74.2
Table S5 : MR-Bench-H on all 900 clips. Accuracy (%) on the 620 LS and 280 NS clips, 95% Wilson interval of the LS accuracy and LS ShortErr (SE, %). V+A: clip with its audio track, V: muted video, V+T: muted video with the transcript, A: audio, T: transcript. Text inversion flips the valid answers of Qwen2.5-7B; always “negative” is the constant answer.
Model
Input
MOSI (14)
MOSEI (23)
SIMS (24)
SIMSv2 (49)
MUSt. (53)
MELD (176)
IEMO. (281)
Qwen2.5-7B
T
7
4
0
2
6
7
5
InternVL3-8B
V
71
70
58
39
55
72
73
InternVL3-8B
V+T
14
30
25
31
6
23
22
Qwen2.5-Omni-7B
V+A
29
26
38
43
4
27
31
Always “negative”
–
57
48
50
35
100
79
51
LLM-Fusion
t
5
7
8
12
83
35
56
Table S6 : LS accuracy per source (%; number of LS clips in parentheses). Fusion models are trained on the training split of each source; t : their text-only view. Mean over three seeds for the fusion models.
Figure S2 : Adding the transcript on the same 620 LS clips. Accuracy falls by 47.9, 46.1 and 53.1 points for Qwen2.5-VL, InternVL3 and LLaVA-OV, respectively.
Pattern
Method
CMU-MOSI
CMU-MOSEI
CH-SIMS
CH-SIMSv2
Acc2 ↑
MAE ↓
Corr ↑
Acc2 ↑
MAE ↓
Corr ↑
Acc2 ↑
MAE ↓
Corr ↑
Acc2 ↑
MAE ↓
Corr ↑
{t,a,v}
LLM-Fusion
80.2
0.899
0.689
84.2
0.557
0.753
79.1
0.424
0.625
83.9
0.294
0.740
CoVeR
79.4
0.899
0.708
84.8
0.547
0.759
83.8
0.403
0.668
85.4
0.281
0.754
{t,a}
LLM-Fusion
79.6
0.899
0.686
84.2
0.566
0.746
76.8
0.456
0.551
78.5
0.342
0.645
CoVeR
79.9
0.887
0.714
85.2
0.549
0.757
80.2
0.439
0.577
79.4
0.332
0.654
{t,v}
LLM-Fusion
79.1
0.908
0.685
84.3
0.557
0.753
78.8
0.430
0.616
83.6
0.302
0.729
Table S7 : Results per availability pattern (Acc2 in %, MAE, Corr) of the runs in Tab. 2 ; mean over 3 seeds.
Backbone
Params
Corr ↑
Cons. ↑
Conf. ↑
SE ↓
w/o t↑
Qwen2.5-0.5B [ 36 ]
0.5B
49.9
89.4
41.9
58.1
72.6
Qwen3-0.6B [ 37 ]
0.6B
49.6
89.6
38.7
61.3
68.8
Qwen2.5-1.5B [ 36 ]
1.5B
49.4
90.0
35.5
64.5
62.9
Qwen3-1.7B [ 37 ]
1.7B
49.9
88.2
46.8
53.2
67.7
SmolLM2-1.7B [ 2 ]
1.7B
48.9
90.3
41.9
58.1
66.1
Table S8 : Does scaling up solve the problem? LLM-Fusion with five LLM backbones on CH-SIMS (all modalities observed): Avg Corr ( ×100 ) over the seven patterns, Acc2 on MR-Bench-A consensus and conflict cases, ShortErr, and conflict Acc2 of the same model with the text removed (%). Mean over 2 seeds (3 for Qwen3-0.6B). Pink, orange and yellow mark the best, second-best and third-best distinct values, respectively; ties share a color.
τtest
CMU-MOSI
CH-SIMS
CH-SIMSv2
Corr ↑
Cons. ↑
Conf. ↑
Corr ↑
Cons. ↑
Conf. ↑
Corr ↑
Cons. ↑
Conf. ↑
−
50.7
94.1
21.7
48.8
89.2
41.3
59.2
93.7
58.6
0.5
50.3
93.8
23.3
45.9
84.4
51.6
58.1
92.3
63.6
0.4
49.3
93.5
26.7
44.4
81.7
59.4
57.0
90.5
67.4
0.3
47.8
92.5
33.9
42.9
78.5
62.6
55.3
86.8
72.3
0.2
44.2
86.4
48.9
41.5
73.8
67.1
52.9
78.8
79.5
Table S9 : Test-time override trades consensus for conflict accuracy. LLM-Fusion with disagreeing views overriding the fused decision ( Eqs. 3 and 4 ) at test-time margin τtest ( − : no override). Avg Corr ( ×100 ) and MR-Bench-A Acc2 (%) on consensus (Cons.) and conflict (Conf.) cases; all rows from the same evaluation of LLM-Fusion with its views. Mean over 5 seeds. This is the earlier margin-sweep checkpoint series, separate from the updated CH-SIMS ablation.
Method
CMU-MOSI
CMU-MOSEI
CH-SIMS
CH-SIMSv2
Dyn.
Perm.
Dyn.
Perm.
Dyn.
Perm.
Dyn.
Perm.
MulT [ 42 ]
52.1
47.9
53.2
51.2
41.4
44.8
53.4
52.1
MISA [ 19 ]
47.9
44.5
54.9
52.3
46.3
45.8
55.0
54.9
MoMKE [ 50 ]
47.4
42.2
57.4
54.7
44.6
42.2
57.9
54.5
MPLMM [ 16 ]
52.0
46.2
56.7
54.4
50.0
43.2
58.1
56.6
MMIN [ 60 ] †
50.4
46.9
57.4
55.7
50.8
45.7
57.7
57.1
Table S10 : Dynamic masking vs . permanent missingness. Avg Corr ( ×100 ) over the seven test patterns. Dyn.: the main protocol ( Tab. 2 ); Perm.: every training and validation sample keeps one availability pattern for the whole run. † : the losses of these methods target the features of missing modalities, which permanent missingness does not provide; they receive them as an oracle. No complete: only the six incomplete patterns are drawn. Mean over 3 seeds; best permanent result in bold.
Method
Venue
T-Miss
Acc2 ↑
F1 ↑
MAE ↓
Corr ↑
MulT [ 42 ]
ACL’19
✓
77.9
75.8
0.616
0.556
MISA [ 19 ]
MM’20
✓
80.2
77.9
0.599
0.602
MMIN [ 60 ]
ACL’21
✗
80.9
80.6
0.626
0.627
TFR-Net [ 55 ]
MM’21
✗
77.9
76.7
0.612
0.562
IMDer [ 45 ]
NeurIPS’23
✗
79.2
78.9
0.579
0.606
DiCMoR [ 44 ]
ICCV’23
✗
80.1
79.6
0.580
0.615
Table S11 : IEMOCAP (features extracted by us, supplementary material): mean over the seven availability patterns, Acc2 and F1 in %. LLM-Fusion: the backbone of CoVeR without judges and gate. Mean over 3 seeds. Pink, orange and yellow mark the best, second-best and third-best distinct values, respectively; ties share a color.
National University of Defense Technology, changsha, China · 2Chinese University of Hong Kong, Shenzhen, China · 3Intelligent Game and Decision Lab, Beijing, China