cs.CLNov 9, 2026

When Audio and Video Mislead: Evaluating Multimodal AI for Conversational Emotion Recognition

Authors: Youngseok Kim, Minjae Kim, Eunsoo Lee, Jihie Kim

Organizations: Dongguk University, Seoul, Republic of Korea

Abstract

Multimodal Emotion Recognition in Conversation (MERC) combines lexical, acoustic, visual and conversational cues, but accepting audio and video does not establish their effective or robust use. We audit off-the-shelf Omni models while holding the target transcript and prior textual context fixed and adding, removing or replacing target audio and video. In the primary evaluation of Qwen2.5-Omni-7B, Baichuan-Omni-1.5 and MiniCPM-o 2.6 on MELD and the full five-session six-class IEMOCAP set, T+A improves Accuracy in all six model-dataset pairs, with four gains significant after false discovery rate (FDR) correction. Across three source mappings, unrelated A/V (RandA/V) lowers Accuracy relative to Full in all 18 comparisons; all 18 remain significant under dialogue-cluster inference and BH-FDR correction, both on common-valid pairs and when invalid/OOM outcomes are treated as incorrect. Comparisons with modality-omission baselines further distinguish replacement penalties from mismatch-specific harm. A Qwen/MELD case study finds significantly greater gold-vs-best-candidate margin recovery in Late than Early or Middle under decision-query audio knockout, relative to an edge-count-matched Video-Sham control, using Holm-corrected tests. These results distinguish genuine-modality benefit from sensitivity and vulnerability to misleading input.

Figures & tables

Explore similar work

CardsList
  1. EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving

    Nov 9, 2026Fenghai Li, Bingxuan Li, Jiawen Zhang +4Multimodal Emotion RecognitionMultimodal Large Language Models

  2. Your Multimodal Speech Model Says I Have a Face for Radio

    May 28, 2026Maya K. Nachesa, Vlad Niculae, Vagrant GautamASR EvaluationDemographic Bias in ASR

  3. Mind the Divergence: Divergence-Guided Reliability-Aware Fusion and Adaptive LLM Prompting for Conversational Emotion Recognition

    Nov 6, 2026Jingwen Wang, Jiao Li, Yuhang Qian +2Emotion Recognition in ConversationsReliability-Aware Multimodal Fusion