Multimodal emotion recognition in conversations (MERC) integrates textual, acoustic, and visual cues to identify utterance-level emotions. However, existing approaches generally do not explicitly adapt multimodal processing to the degree of inter-modal predictive disagreement. In real conversations, semantic incongruity, concealed emotion, noisy observations, and ambiguity can affect modality-specific emotional evidence differently, while the reliability of individual modalities varies across samples. Rather than identifying each source of inconsistency separately, we model their common consequence for emotion recognition as predictive disagreement among unimodal emotion distributions over the same emotion-label set. Such disagreement makes indiscriminate fusion prone to propagating unreliable or competing evidence. Inspired by the Brunswik Lens Model, we propose MiD-ERC, a divergence-centric framework that quantifies inter-modal predictive disagreement using a normalized divergence score and estimates sample-specific modality reliability. MiD-ERC uses these signals to coordinate divergence-guided cross-modal gating, divergence-modulated reliability weighting, and divergence-conditioned adaptive LLM prompting. The framework is trained end-to-end using only standard emotion annotations, without requiring additional conflict labels, auxiliary data construction, or external retrieval modules. Experiments on IEMOCAP and MELD demonstrate that MiD-ERC achieves state-of-the-art performance, with particularly pronounced improvements on high-divergence samples.
Figures & tables
Figure 1: Orson ( Akass and McCabe 2019 ) displays an angry facial expression while speaking in a relatively calm tone. MiD-ERC adapts multimodal fusion according to the resulting disagreement.
Figure 2: Schematic diagram of the MiD-ERC framework.
Dataset
Conversations
Utterances
Classes
train
val
test
train
val
test
IEMOCAP
120
31
5810
1623
6
MELD
1039
114
280
9989
1109
2610
7
Table 1: Statistics of the two datasets.
Methods
IEMOCAP
MELD
Happy
Sad
Neutral
Angry
Excited
Frustrated
Acc
W-F1
Neutral
Surprise
Fear
Sadness
Joy
Disgust
Anger
Acc
W-F1
★ DialogueGCN ♭ ( Ghosal et al. 2019 )
51.57
80.48
57.69
53.95
72.81
57.33
63.22
62.89
75.97
46.05
-
19.60
51.20
-
40.83
58.62
56.36
★ CFN-ESA ♯ ( Li et al. 2024 )
53.67
80.60
71.65
70.32
74.82
68.06
70.78
71.04
80.05
58.78
21.62
41.82
66.50
26.92
54.18
67.85
66.70
★ LSDGNN+ICL ♯ ( Li et al. 2025b )
47.20
81.53
71.00
69.74
73.19
68.58
70.35
70.24
77.58
57.90
18.18
37.94
61.70
29.70
51.88
64.67
64.07
★ GraphSmile ‡ ( Li et al. 2025a )
60.14
81.38
68.48
70.45
77.60
66.02
70.98
71.00
62.99
51.96
32.14
28.85
39.24
56.62
79.95
66.67
65.46
★ ConxGNN ♯ ( Van et al. 2025 )
-
-
-
-
-
-
68.52
68.64
-
-
-
-
-
-
-
66.28
65.69
Table 2: We compare the overall performance of various methods using Acc, W-F1, and per-class F1 scores. ♯ , ♭ , and ‡ denote results from original papers, MMDFN ( Hu et al. 2022a ) , and our reproduction, respectively. ♣ signifies external components or re-annotated datasets, while ★ indicates open-source code availability.
Methods
IEMOCAP
MELD
Single
Total
Single
Total
MiD-ERC
0.18
292.14
0.15
391.50
Causal-ERC
0.34
551.82
0.28
730.80
Table 3: Inference time comparison.
Methods
IEMOCAP
MELD
Acc
W-F1
Acc
W-F1
Ours
73.44
73.38
71.79
70.60
w/ T only
71.47
71.35
70.46
69.15
w/ TA only
72.58
72.51
70.88
69.55
w/o Speaker-Aware GCN
72.21
72.34
70.91
69.58
w/o Luni
71.04
71.05
71.11
69.72
Table 4: Ablation study on IEMOCAP and MELD.
Figure 3: Performance across divergence levels on IEMOCAP and MELD.
Figure 4: Learned gate gi versus its three inputs.
Figure 5: Fusion weight wiξ versus relative reliability.
Figure 6: Weight–Reliability Alignment and Performance.
Figure 7: Sensitivity to the divergence threshold θ .
Figure 8: Case study of MiD-ERC on a conflict sample.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Relation prompt templates used for dual-prompt conflict-aware LLM reason- ing mechanism.
Methods
IEMOCAP
MELD
Acc
W-F1
Acc
W-F1
Qwen2.5-7B-Instruct
73.44
73.38
71.79
70.60
Llama-3.1-8B-Instruct
73.72
73.64
71.95
70.81
Qwen3-8B
73.01
72.97
71.83
70.76
Appendix
Table 5: Ablation study on IEMOCAP and MELD.
Dataset
Variant
Params.
Acc
W-F1
IEMOCAP
MiD-ERC
147.826M
73.44
73.38
Shared Context Encoders
145.056M
70.61
70.30
Relative Change
↓1.87%
↓3.85%
↓4.20%
MELD
MiD-ERC
136.733M
71.79
70.60
Shared Context Encoders
135.768M
70.11
68.42
Relative Change
↓0.71%
↓2.33%
↓3.09%
Appendix
Table 6: Effect of parameter sharing in unimodal context encoders. Relative changes are calculated with respect to MiD-ERC.
Emotion Recognition in Conversations studies how dialogue context, speaker dynamics, and language, voice, and visual cues reveal changing emotional states.
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong