SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations
Authors: Yiqiang Cai, Chengyan Wu, Bolei Ma, Bo Chen, Yun Xue, Julia Hirschberg, Ziwei Gong
Organizations: Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials · School of Electronic Science and Engineering (School of Microelectronics), South China Normal University · LMU Munich & Munich Center for Machine Learning · Shenzhen University · Columbia University
Multimodal emotion recognition in conversations (MERC) requires integrating multimodal signals while being robust to noise and modeling contextual reasoning. Existing approaches often emphasize fusion but overlook uncertainty in noisy features and fine-grained reasoning. We propose SURE (Synergistic Uncertainty-aware REasoning) for MERC, a framework that improves robustness and contextual modeling. SURE consists of three components: an Uncertainty-Aware Mixture-of-Experts module to handle modality-specific noise, an Iterative Reasoning module for multi-turn reasoning over context, and a Transformer Gate module to capture intra- and inter-modal interactions. Experiments on benchmark MERC datasets show that SURE consistently outperforms state-of-the-art methods, demonstrating its effectiveness in robust multimodal reasoning. These results highlight the importance of uncertainty modeling and iterative reasoning in advancing emotion recognition in conversational settings.
Figures & tables
Figure 1: The SURE Framework. The top shows the overall framework of multimodal inputs and classifying process. The bottom consists of three main components: Uncertainty-Aware MoE, Iterative Reasoning, and Transformer Gate, corresponding to the three major intermediate modules in the SURE framework.
IEMOCAP (6-Ways)
MELD (7-Ways)
Datasets
#Dialogue
#Utterance
#Dialogue
#Utterance
Train+Val
120
5810
1153
11098
Test
31
1623
280
2610
Table 1: Datasets statistics.
Models
IEMOCAP
MELD
Acc
F1
Acc
F1
Graph-based Methods
MMGCN
66.36
66.26
60.42
58.31
MM-DFN
68.21
68.18
62.49
59.46
Joyful
70.55
71.03
62.53
61.77
MMPCGN
68.90
68.00
60.70
59.30
Table 2: Comparison with baselines on IEMOCAP and MELD.
IEMOCAP
MELD
Set-ups
Acc
F1
Acc
F1
Methods
SURE
75.31
74.80
67.97
67.36
w/o MoE
74.99
74.23
67.65
67.02
w/o Reasoning
75.02
74.42
67.32
66.92
Modality
Table 3: Model variations and fusion network ablation results.
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong