Organizations: LEAP Lab, Department of Electrical Engineering, IISc Bangalore, Bangalore, 560012, Karnataka, India · Microsoft, Bangalore, Karnataka, India
Emotion Recognition in Conversations (ERC) presents unique challenges, requiring models to capture the temporal flow of multi-turn dialogues and to effectively integrate cues from multiple modalities. We propose Mixture of Speech-Text Experts for Recognition of Emotions (MiSTER-E), a modular Mixture-of-Experts (MoE) framework designed to decouple two core challenges in ERC: modality-specific context modeling and multimodal information fusion. MiSTER-E leverages large language models (LLMs) fine-tuned for both speech and text to provide rich utterance-level embeddings, which are then enhanced through a convolutional-recurrent context modeling layer. The system integrates predictions from three experts-speech-only, text-only, and cross-modal-using a learned gating mechanism that dynamically weighs their outputs. To further encourage consistency and alignment across modalities, we introduce a supervised contrastive loss between paired speech-text representations and a KL-divergence-based regulariza-tion across expert predictions. Importantly, MiSTER-E does not rely on speaker identity at any stage. Experiments on three benchmark datasets-IEMOCAP, MELD, and MOSI-show that our proposal achieves 70.9%, 69.5%, and 87.9% weighted F1-scores respectively, outperforming several baseline speech-text ERC systems. We also provide various ablations to highlight the contributions made in the proposed approach.
Figures & tables
Figure 1: (a) Training of the unimodal feature extraction module (Sec. 3.2 ) (b) The entire pipeline of MiSTER-E. The speech and text embedding modules are frozen during training of the rest of the pipeline. Two context addition networks (Sec. 3.3.1 ) are trained for the two modalities along with a multimodal network (Sec. 3.3.2 ). Finally, a mixture of experts gating network (Sec. 3.4 ) is trained to predict the emotion category for each utterance.
Symbol
Description
D
Emotion Recognition in Conversations (ERC) dataset
P
Number of conversations in the dataset
C={c1,c2,…,cP}
Set of all conversations in D
ci
The i -th conversation
Ui=(ui1,ui2,…,uiNi)
Sequence of utterances in conversation ci
Ni
Number of utterances in conversation ci
Table 1: Notations used in the problem formulation and model description.
Figure 2: (a) The context addition network (for the speech modality) and (b) the multimodal network used in MiSTER-E. The inputs to both the blocks are derived from the uni-modal feature extractor modules. TIN stands for Temporal Inception Network, MHA stands for multi-head attention.
Method
IEMOCAP ( 6 -class)
MELD ( 7 -class)
MOSI ( 2 -class)
SMIN [ 32 ] ##
70.5%
63.7%
79.7%
MultiEmo [ 52 ] #
66.9%±2.0
65.3%±0.5
83.7%±0.3
HCAM [ 11 ]
70.5%
65.8%
85.8%
DF-ERC [ 53 ]
69.5%
64.5%
–
DER-GCN [ 54 ]
64.7%
62.6%
–
ELR-GNN [ 55 ]
64.4%
63.2%
–
Table 2: Comparison of different methods on IEMOCAP, MELD, and MOSI datasets using weighted-F1 scores. All reported results are for speech-text systems. ## We report the numbers when no external emotional data is used for training SMIN. # We ran the public implementations provided by the authors under our experimental settings. Superscripted results denote the mean and standard deviation over 3 random initializations, where applicable.
Figure 3: Performance of MiSTER-E with different values of the focal loss hyperparameter.
Method
# Params
Text Feats.
Speech Feats.
IEMOCAP
MELD
MultiEmo [ 52 ]
≈450 M
RoBERTa
OpenSMILE
66.9%
65.3%
+ LLM/SLLM
≈14 B
LLaMA
SALMONN
69.5%
68.2%
HCAM [ 11 ]
≈750 M
RoBERTa
wav2vec
70.5%
65.8%
+ LLM/SLLM
≈14 B
LLaMA
SALMONN
70.6%
68.4%
MMGAT-EMO [ 36 ]
≈450 M
EmoBERTa
OpenSMILE
65.5%
66.1%
+ LLM/SLLM
≈14 B
LLaMA
SALMONN
68.6%
67.7%
Table 3: Comparison of some of the baseline methods when re-designed with and without LLM features. The results show that the proposal is applicable even for non-LLM representations like OPENSMILE and wav2vec.
Figure 4: Comparison of model performance on IEMOCAP and MELD. (a) Effect of different MoE gating strategies in MiSTER-E. (b) Performance of unimodal SLLMs and LLMs across the two datasets.
Figure 5: (a) Distribution of weights for the experts for the different datasets and (b) The performance of the experts and MiSTER-E for the two datasets
Figure 6: A case study from MELD where there are 16 utterances in the conversation. The weights assigned to the different experts are shown. In most cases, text is assigned the highest weight (except utterance 2 and 9 ). S: Speech Expert, T: Text Expert, M: Multimodal Expert. The predictions of MiSTER-E are also shown. The model correctly predicts all the utterances in the conversation, inspite of the emotion transitions.
Figure 7: (a) Confusion matrix for IEMOCAP. (b) Class-wise F1-scores on IEMOCAP and MELD, showing variation across different emotions.
Figure 8: (a) Results of the text only experiment and, (b) Performance of MiSTER-E with varying values of λ (Eq. 18 ) for the two datasets. The validation set performance mirrors the trends of the test set. For IEMOCAP, validation accuracy is 63% for λ=0 and 63.6% for λ=2 . For MELD, λ=1 yields the highest validation performance ( 65.6% ) compared to 65.4% for λ=0
Figure 9: A case study from MELD where there are emotion shifts throughout the conversation. Out of the 7 utterances in the conversation, MiSTER-E correctly predicts 6 .
Multimodal emotion recognition in conversation (ERC) requires adapting to the instance-dependent reliability of different evidence sources. Lexical content may be decisive, vocal expression may provide complementary cues, or accurate recognition may require cross-modal interaction; fixed fusion does not explicitly account for this variation. We propose ReH-FUSE, a reliability-aware framework with dialogue-aware text, audio, and cross-modal experts. Its decision-level router first models the relative preference between text and audio and then balances the resulting unimodal mixture against the cross-modal expert. This factorization separates unimodal competition from cross-modal selection. Across three independent runs on IEMOCAP, ReH-FUSE achieves 74.34% weighted F1 and 73.11% macro F1; on MELD, it achieves 68.03% weighted F1. Controlled ablations show that learned routing outperforms uniform expert averaging and benefits from cross-modal interaction.
Guan-Hua Wen, Hou-Chiang Tseng, Kuan-Yu Chen
National Taiwan University of Science and Technology, Taipei, Taiwan
Emotion Recognition in Conversation (ERC) is essential for effective human-machine interaction, aiming to identify speakers' emotional states in multi-turn dialogues. Early text-based methods struggle with complex scenarios like sarcasm because they inherently neglect vital non-verbal information. While recent Vision-Language Models (VLMs) address this by analyzing video directly, they are not inherently tailored for ERC and often focus on emotionally irrelevant background regions or passive listeners rather than the active speaker. Furthermore, fine-tuning these large models incurs prohibitive computational costs. Additionally, isolated visual signals are frequently ambiguous or technically compromised without the context of linguistic content and vocal prosody. To address these challenges, we propose VISAFF, a speaker-centered VISual AFFective feature learning framework for ERC. VISAFF consists of two stages: Speaker-Centered Affective Grounding and Reliability-Guided Affective Complementation. VISAFF utilizes a tuning-free approach to unlock the reasoning capabilities of frozen VLMs, efficiently steering them to focus on the active speaker's emotional visual cues without heavy training overheads. In the second stage, we introduce a reliability-guided affective complementation mechanism that dynamically leverages textual and acoustic modalities to compensate for visual uncertainty. Experiments on two real-world datasets demonstrate that VISAFF achieves highly competitive performance compared to state-of-the-art methods in a tuning-free setting, significantly enhancing computational efficiency by eliminating the need for expensive fine-tuning of large VLMs. The source code is available at https://anonymous.4open.science/r/speaker-2365/.
Linan ZHU, Zihao Zhai, Xiao Han +4
1Zhejiang University of Technology · 2ETH Zurich, Switzerland
Multimodal emotion recognition in conversations (MERC) requires integrating multimodal signals while being robust to noise and modeling contextual reasoning. Existing approaches often emphasize fusion but overlook uncertainty in noisy features and fine-grained reasoning. We propose SURE (Synergistic Uncertainty-aware REasoning) for MERC, a framework that improves robustness and contextual modeling. SURE consists of three components: an Uncertainty-Aware Mixture-of-Experts module to handle modality-specific noise, an Iterative Reasoning module for multi-turn reasoning over context, and a Transformer Gate module to capture intra- and inter-modal interactions. Experiments on benchmark MERC datasets show that SURE consistently outperforms state-of-the-art methods, demonstrating its effectiveness in robust multimodal reasoning. These results highlight the importance of uncertainty modeling and iterative reasoning in advancing emotion recognition in conversational settings.
Yiqiang Cai, Chengyan Wu, Bolei Ma +4
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials · School of Electronic Science and Engineering (School of Microelectronics), South China Normal University · LMU Munich & Munich Center for Machine Learning +2