cs.CVNov 16, 2026

Words Speak Louder? Exploring the Language Shortcut of LLM-Driven Incomplete Multimodal Learning

Authors: Qiangchang Wang, Chong Xiao, Xiaokun Bai, Rundong He, Zhuang Qi, Xusheng Liang, Zhibin Wu, Wenhao Li

Organizations: Shandong University · The Hong Kong Polytechnic University · National University of Singapore · City University of Hong Kong · 

Abstract

Is a face or a voice still heard once the words are known? Incomplete multimodal learning increasingly uses large language models (LLMs) as fusion engines: modality features enter as token blocks and missing modalities become placeholders. We find that LLM-driven fusion can override non-verbal evidence it can read. Contrasting its full-input prediction with counterfactual views, the same model shown one modality or no content, we identify language shortcuts. We build MR-Bench: MR-Bench-A types every test sample of four sentiment benchmarks; MR-Bench-H contains 900 human-validated clips (620 language-shortcut, LS; 280 non-verbal-shortcut, NS) from seven sources, and every model is scored on all of them. Zero-shot MLLMs solve at most 32.7% of the LS clips; adding the transcript to the muted video lowers vision-language accuracy by 46.1-53.1 points. Yet inverting a text-only model's answers solves 94.4%, so LS accuracy alone cannot establish use of non-verbal evidence. Inside the fusion LLM, fused states align increasingly with the text view late in the decoder. Counterfactual-View Reliability learning (CoVeR) trains views as unimodal judges and withholds fused supervision where they disagree. It achieves state-of-the-art results on five benchmarks without extra inference cost in accuracy mode. On MR-Bench-H, it reaches 61.7% LS and 93.6% NS accuracy; robust mode raises LS accuracy to 63.0% while lowering NS accuracy to 84.8%. Among LS clips solved without text, adding text causes errors on 30.1% with CoVeR versus 32.2% with LLM-Fusion. Project page: https://peacelwh.github.io/CoVeR/.

Figures & tables

Explore similar work

CardsList
  1. Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs

    Sep 29, 2026Yueran Ma, Ronghao LinAudio QAMultimodal Large Language Models

  2. MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias

    Jun 16, 2026Xingming Li, Ao Cheng, Qiyao Sun +4VLM RobustnessVision-Language Models

  3. Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

    Jul 28, 2026Jiaang Li, Chengzu Li, Zhaochong An +4VLM EvaluationMultimodal Large Language Models