Words Speak Louder? Exploring the Language Shortcut of LLM-Driven Incomplete Multimodal Learning
Organizations: Shandong University · The Hong Kong Polytechnic University · National University of Singapore · City University of Hong Kong ·
Abstract
Is a face or a voice still heard once the words are known? Incomplete multimodal learning increasingly uses large language models (LLMs) as fusion engines: modality features enter as token blocks and missing modalities become placeholders. We find that LLM-driven fusion can override non-verbal evidence it can read. Contrasting its full-input prediction with counterfactual views, the same model shown one modality or no content, we identify language shortcuts. We build MR-Bench: MR-Bench-A types every test sample of four sentiment benchmarks; MR-Bench-H contains 900 human-validated clips (620 language-shortcut, LS; 280 non-verbal-shortcut, NS) from seven sources, and every model is scored on all of them. Zero-shot MLLMs solve at most 32.7% of the LS clips; adding the transcript to the muted video lowers vision-language accuracy by 46.1-53.1 points. Yet inverting a text-only model's answers solves 94.4%, so LS accuracy alone cannot establish use of non-verbal evidence. Inside the fusion LLM, fused states align increasingly with the text view late in the decoder. Counterfactual-View Reliability learning (CoVeR) trains views as unimodal judges and withholds fused supervision where they disagree. It achieves state-of-the-art results on five benchmarks without extra inference cost in accuracy mode. On MR-Bench-H, it reaches 61.7% LS and 93.6% NS accuracy; robust mode raises LS accuracy to 63.0% while lowering NS accuracy to 84.8%. Among LS clips solved without text, adding text causes errors on 30.1% with CoVeR versus 32.2% with LLM-Fusion. Project page: https://peacelwh.github.io/CoVeR/.
Figures & tables
| Method | T-free | +Views | CMU-MOSI | CH-SIMSv2 | ||||||
| Corr | Cons. | Conf. | SE | Corr | Cons. | Conf. | SE | |||
| LLM-Fusion | – | – | 50.7 | 94.1 | 21.7 | 78.3 | 59.2 | 93.7 | 58.6 | 41.4 |
| + discrepancy-gated steering | ✓ | ✓ | 50.3 | 93.8 | 22.2 | 77.8 | 58.9 | 93.6 | 58.6 | 41.4 |
| + test-time override ( ) | ✓ | ✓ | 47.8 | 92.5 | 33.9 | 66.1 | 55.3 | 86.8 | 72.3 | 27.7 |
| + learned reliability gate | ✗ | ✓ | 50.1 | 92.5 | 19.4 | 80.6 | 58.6 | 93.1 | 56.4 | 43.6 |
| + conflict re-weighting | ✗ | ✗ | 50.4 | 93.6 | 24.4 | 75.6 | 58.1 | 93.0 | 52.3 | 47.7 |
| Method | Venue | T-Miss | CMU-MOSI | CMU-MOSEI | CH-SIMS | CH-SIMSv2 | ||||||||||||
| Acc2 | F1 | MAE | Corr | Acc2 | F1 | MAE | Corr | Acc2 | F1 | MAE | Corr | Acc2 | F1 | MAE | Corr | |||
| MulT [ 42 ] | ACL’19 | ✓ | 68.6 | 65.5 | 1.160 | 0.521 | 73.9 | 68.9 | 0.706 | 0.532 | 72.6 | 69.6 | 0.510 | 0.414 | 73.8 | 71.5 | 0.393 | 0.534 |
| MISA [ 19 ] | MM’20 | ✓ | 67.0 | 65.4 | 1.166 | 0.479 | 74.4 | 72.8 | 0.702 | 0.549 | 72.6 | 68.7 | 0.520 | 0.463 | 73.6 | 72.7 | 0.397 | 0.550 |
| MMIN [ 60 ] | ACL’21 | ✗ | 67.7 | 67.0 | 1.205 | 0.504 | 76.3 | 74.8 | 0.690 | 0.574 | 74.6 | 74.4 | 0.502 | 0.508 | 75.2 | 74.3 | 0.388 | 0.577 |
| TFR-Net [ 55 ] | MM’21 | ✗ | 67.6 | 63.8 | 1.140 | 0.501 | 73.1 | 67.0 | 0.696 | 0.515 | 73.7 | 71.3 | 0.508 | 0.462 | 74.6 | 73.5 | 0.382 | 0.565 |
| IMDer [ 45 ] | NeurIPS’23 | ✗ | 68.3 | 65.9 | 1.151 | 0.506 | 75.2 | 70.7 | 0.675 | 0.567 | 75.3 | 72.4 | 0.495 | 0.478 | 75.7 | 75.3 | 0.371 | 0.581 |
| Variant | Avg Corr | Conf. | SE |
| LLM-Fusion | 49.6 | 38.7 | 61.3 |
| + unimodal loss, no gate | 52.2 | 41.3 | 58.7 |
| + unimodal loss + gate (CoVeR) | 55.1 | 47.7 | 52.3 |
| Method | Targets | Deviations |
| MulT [ 42 ] | – | none (zero imputation) |
| MISA [ 19 ] | – | utterance vectors by mean pooling (no padding) |
| MMIN [ 60 ] | ✓ | full-modality teacher (first 25% of epochs) then imagination; official IEMOCAP settings; 80 epochs (22 on CMU-MOSEI) and batch 32–128 to match the official number of updates, since 30 epochs leave the teacher under-trained |
| TFR-Net [ 55 ] | ✓ | missing positions from the pattern (whole modality) instead of random frames |
| IMDer [ 45 ] | ✓ | score-based recovery per sample from the observed modalities (official 100 sampling steps); trained from scratch instead of from the official complete-data checkpoint |
| DiCMoR [ 44 ] | ✓ | flow-based recovery per sample from the observed modalities; trained from scratch instead of from the official complete-data checkpoint |
| Dataset | Samples | Consensus | Conflict | Weak |
| CMU-MOSI | 1284/229/686 | 428/92/187 | 55/9/36 | 297/55/186 |
| CMU-MOSEI | 16326/1871/4659 | 6387/675/1669 | 68/4/20 | 675/83/265 |
| CH-SIMS | 1368/456/457 | 621/211/221 | 105/37/31 | 99/38/32 |
| CH-SIMSv2 | 2722/647/1034 | 1423/326/457 | 148/55/73 | 146/42/54 |
| IEMOCAP | 6740/617/2170 | 4660/312/951 | 50/14/87 | 109/34/200 |
| Dataset | Judgments | #Test | LLM-Fusion (%) | |||||
| Cons. | Conf. | Cons. | Conf. | SE | w/o | by | ||
| CMU-MOSI | OOF experts | 187 | 36 | 94.7 | 22.2 | 77.8 | 63.0 | 54.6 |
| CMU-MOSEI | OOF experts | 1669 | 20 | 97.8 | 16.7 | 83.3 | 86.7 | 78.3 |
| CH-SIMS | human | 221 | 31 | 89.6 | 38.7 | 61.3 | 68.8 | 69.9 |
| CH-SIMSv2 | human | 457 | 73 | 93.7 | 59.4 | 40.6 | 83.1 | 84.0 |
| Source | Lang. | Candidates | Reader panel | Validated | ||||||
| LS | NS | Ctrl | LS | NS | Ctrl | LS | NS | Ctrl | ||
| CMU-MOSI [ 56 ] | EN | 83 | 135 | 22 | 23 | 130 | 21 | 14 | 119 | 0 |
| CMU-MOSEI [ 3 ] | EN | 139 | 177 | 32 | 53 | 175 | 28 | 23 | 119 | 0 |
| CH-SIMS [ 53 ] | ZH | 48 | 26 | 7 | 38 | 22 | 7 | 24 | 16 | 0 |
| CH-SIMSv2 [ 29 ] | ZH | 96 | 42 | 14 | 74 | 36 | 14 | 49 | 26 | 0 |
| MUStARD++ [ 38 ] | EN | 101 | – | 10 | 69 | – | 10 | 53 | – | 0 |
| Model | Input | LS | 95% CI | NS | SE |
| Zero-shot MLLMs | |||||
| Qwen2.5-Omni-7B | V+A | 28.7 | [25.3, 32.4] | 94.6 | 71.3 |
| A | 21.8 | [18.7, 25.2] | 97.9 | 78.2 | |
| V | 53.9 | [49.9, 57.8] | 33.2 | 46.1 | |
| T | 3.1 | [2.0, 4.7] | 99.3 | 96.8 | |
| Qwen2.5-Omni-3B | V+A | 25.8 | [22.5, 29.4] | 92.5 | 74.2 |
| Model | Input | MOSI (14) | MOSEI (23) | SIMS (24) | SIMSv2 (49) | MUSt. (53) | MELD (176) | IEMO. (281) |
| Qwen2.5-7B | T | 7 | 4 | 0 | 2 | 6 | 7 | 5 |
| InternVL3-8B | V | 71 | 70 | 58 | 39 | 55 | 72 | 73 |
| InternVL3-8B | V+T | 14 | 30 | 25 | 31 | 6 | 23 | 22 |
| Qwen2.5-Omni-7B | V+A | 29 | 26 | 38 | 43 | 4 | 27 | 31 |
| Always “negative” | – | 57 | 48 | 50 | 35 | 100 | 79 | 51 |
| LLM-Fusion | 5 | 7 | 8 | 12 | 83 | 35 | 56 |
| Pattern | Method | CMU-MOSI | CMU-MOSEI | CH-SIMS | CH-SIMSv2 | ||||||||
| Acc2 | MAE | Corr | Acc2 | MAE | Corr | Acc2 | MAE | Corr | Acc2 | MAE | Corr | ||
| LLM-Fusion | 80.2 | 0.899 | 0.689 | 84.2 | 0.557 | 0.753 | 79.1 | 0.424 | 0.625 | 83.9 | 0.294 | 0.740 | |
| CoVeR | 79.4 | 0.899 | 0.708 | 84.8 | 0.547 | 0.759 | 83.8 | 0.403 | 0.668 | 85.4 | 0.281 | 0.754 | |
| LLM-Fusion | 79.6 | 0.899 | 0.686 | 84.2 | 0.566 | 0.746 | 76.8 | 0.456 | 0.551 | 78.5 | 0.342 | 0.645 | |
| CoVeR | 79.9 | 0.887 | 0.714 | 85.2 | 0.549 | 0.757 | 80.2 | 0.439 | 0.577 | 79.4 | 0.332 | 0.654 | |
| LLM-Fusion | 79.1 | 0.908 | 0.685 | 84.3 | 0.557 | 0.753 | 78.8 | 0.430 | 0.616 | 83.6 | 0.302 | 0.729 | |
| Backbone | Params | Corr | Cons. | Conf. | SE | w/o |
| Qwen2.5-0.5B [ 36 ] | 0.5B | 49.9 | 89.4 | 41.9 | 58.1 | 72.6 |
| Qwen3-0.6B [ 37 ] | 0.6B | 49.6 | 89.6 | 38.7 | 61.3 | 68.8 |
| Qwen2.5-1.5B [ 36 ] | 1.5B | 49.4 | 90.0 | 35.5 | 64.5 | 62.9 |
| Qwen3-1.7B [ 37 ] | 1.7B | 49.9 | 88.2 | 46.8 | 53.2 | 67.7 |
| SmolLM2-1.7B [ 2 ] | 1.7B | 48.9 | 90.3 | 41.9 | 58.1 | 66.1 |
| CMU-MOSI | CH-SIMS | CH-SIMSv2 | |||||||
| Corr | Cons. | Conf. | Corr | Cons. | Conf. | Corr | Cons. | Conf. | |
| 50.7 | 94.1 | 21.7 | 48.8 | 89.2 | 41.3 | 59.2 | 93.7 | 58.6 | |
| 0.5 | 50.3 | 93.8 | 23.3 | 45.9 | 84.4 | 51.6 | 58.1 | 92.3 | 63.6 |
| 0.4 | 49.3 | 93.5 | 26.7 | 44.4 | 81.7 | 59.4 | 57.0 | 90.5 | 67.4 |
| 0.3 | 47.8 | 92.5 | 33.9 | 42.9 | 78.5 | 62.6 | 55.3 | 86.8 | 72.3 |
| 0.2 | 44.2 | 86.4 | 48.9 | 41.5 | 73.8 | 67.1 | 52.9 | 78.8 | 79.5 |
| Method | CMU-MOSI | CMU-MOSEI | CH-SIMS | CH-SIMSv2 | ||||
| Dyn. | Perm. | Dyn. | Perm. | Dyn. | Perm. | Dyn. | Perm. | |
| MulT [ 42 ] | 52.1 | 47.9 | 53.2 | 51.2 | 41.4 | 44.8 | 53.4 | 52.1 |
| MISA [ 19 ] | 47.9 | 44.5 | 54.9 | 52.3 | 46.3 | 45.8 | 55.0 | 54.9 |
| MoMKE [ 50 ] | 47.4 | 42.2 | 57.4 | 54.7 | 44.6 | 42.2 | 57.9 | 54.5 |
| MPLMM [ 16 ] | 52.0 | 46.2 | 56.7 | 54.4 | 50.0 | 43.2 | 58.1 | 56.6 |
| MMIN [ 60 ] † | 50.4 | 46.9 | 57.4 | 55.7 | 50.8 | 45.7 | 57.7 | 57.1 |
| Method | Venue | T-Miss | Acc2 | F1 | MAE | Corr |
| MulT [ 42 ] | ACL’19 | ✓ | 77.9 | 75.8 | 0.616 | 0.556 |
| MISA [ 19 ] | MM’20 | ✓ | 80.2 | 77.9 | 0.599 | 0.602 |
| MMIN [ 60 ] | ACL’21 | ✗ | 80.9 | 80.6 | 0.626 | 0.627 |
| TFR-Net [ 55 ] | MM’21 | ✗ | 77.9 | 76.7 | 0.612 | 0.562 |
| IMDer [ 45 ] | NeurIPS’23 | ✗ | 79.2 | 78.9 | 0.579 | 0.606 |
| DiCMoR [ 44 ] | ICCV’23 | ✗ | 80.1 | 79.6 | 0.580 | 0.615 |
Multimodal Learning
Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.
See all 359 papers