Organizations: LEAP Lab, Department of Electrical Engineering, IISc Bangalore, Bangalore, 560012, Karnataka, India · Microsoft, Bangalore, Karnataka, India
Emotion Recognition in Conversations (ERC) presents unique challenges, requiring models to capture the temporal flow of multi-turn dialogues and to effectively integrate cues from multiple modalities. We propose Mixture of Speech-Text Experts for Recognition of Emotions (MiSTER-E), a modular Mixture-of-Experts (MoE) framework designed to decouple two core challenges in ERC: modality-specific context modeling and multimodal information fusion. MiSTER-E leverages large language models (LLMs) fine-tuned for both speech and text to provide rich utterance-level embeddings, which are then enhanced through a convolutional-recurrent context modeling layer. The system integrates predictions from three experts-speech-only, text-only, and cross-modal-using a learned gating mechanism that dynamically weighs their outputs. To further encourage consistency and alignment across modalities, we introduce a supervised contrastive loss between paired speech-text representations and a KL-divergence-based regulariza-tion across expert predictions. Importantly, MiSTER-E does not rely on speaker identity at any stage. Experiments on three benchmark datasets-IEMOCAP, MELD, and MOSI-show that our proposal achieves 70.9%, 69.5%, and 87.9% weighted F1-scores respectively, outperforming several baseline speech-text ERC systems. We also provide various ablations to highlight the contributions made in the proposed approach.
Figures & tables
Figure 1: (a) Training of the unimodal feature extraction module (Sec. 3.2 ) (b) The entire pipeline of MiSTER-E. The speech and text embedding modules are frozen during training of the rest of the pipeline. Two context addition networks (Sec. 3.3.1 ) are trained for the two modalities along with a multimodal network (Sec. 3.3.2 ). Finally, a mixture of experts gating network (Sec. 3.4 ) is trained to predict the emotion category for each utterance.
Symbol
Description
D
Emotion Recognition in Conversations (ERC) dataset
P
Number of conversations in the dataset
C={c1,c2,…,cP}
Set of all conversations in D
ci
The i -th conversation
Ui=(ui1,ui2,…,uiNi)
Sequence of utterances in conversation ci
Ni
Number of utterances in conversation ci
Table 1: Notations used in the problem formulation and model description.
Figure 2: (a) The context addition network (for the speech modality) and (b) the multimodal network used in MiSTER-E. The inputs to both the blocks are derived from the uni-modal feature extractor modules. TIN stands for Temporal Inception Network, MHA stands for multi-head attention.
Method
IEMOCAP ( 6 -class)
MELD ( 7 -class)
MOSI ( 2 -class)
SMIN [ 32 ] ##
70.5%
63.7%
79.7%
MultiEmo [ 52 ] #
66.9%±2.0
65.3%±0.5
83.7%±0.3
HCAM [ 11 ]
70.5%
65.8%
85.8%
DF-ERC [ 53 ]
69.5%
64.5%
–
DER-GCN [ 54 ]
64.7%
62.6%
–
ELR-GNN [ 55 ]
64.4%
63.2%
–
Table 2: Comparison of different methods on IEMOCAP, MELD, and MOSI datasets using weighted-F1 scores. All reported results are for speech-text systems. ## We report the numbers when no external emotional data is used for training SMIN. # We ran the public implementations provided by the authors under our experimental settings. Superscripted results denote the mean and standard deviation over 3 random initializations, where applicable.
Figure 3: Performance of MiSTER-E with different values of the focal loss hyperparameter.
Method
# Params
Text Feats.
Speech Feats.
IEMOCAP
MELD
MultiEmo [ 52 ]
≈450 M
RoBERTa
OpenSMILE
66.9%
65.3%
+ LLM/SLLM
≈14 B
LLaMA
SALMONN
69.5%
68.2%
HCAM [ 11 ]
≈750 M
RoBERTa
wav2vec
70.5%
65.8%
+ LLM/SLLM
≈14 B
LLaMA
SALMONN
70.6%
68.4%
MMGAT-EMO [ 36 ]
≈450 M
EmoBERTa
OpenSMILE
65.5%
66.1%
+ LLM/SLLM
≈14 B
LLaMA
SALMONN
68.6%
67.7%
Table 3: Comparison of some of the baseline methods when re-designed with and without LLM features. The results show that the proposal is applicable even for non-LLM representations like OPENSMILE and wav2vec.
Figure 4: Comparison of model performance on IEMOCAP and MELD. (a) Effect of different MoE gating strategies in MiSTER-E. (b) Performance of unimodal SLLMs and LLMs across the two datasets.
Figure 5: (a) Distribution of weights for the experts for the different datasets and (b) The performance of the experts and MiSTER-E for the two datasets
Figure 6: A case study from MELD where there are 16 utterances in the conversation. The weights assigned to the different experts are shown. In most cases, text is assigned the highest weight (except utterance 2 and 9 ). S: Speech Expert, T: Text Expert, M: Multimodal Expert. The predictions of MiSTER-E are also shown. The model correctly predicts all the utterances in the conversation, inspite of the emotion transitions.
Figure 7: (a) Confusion matrix for IEMOCAP. (b) Class-wise F1-scores on IEMOCAP and MELD, showing variation across different emotions.
Figure 8: (a) Results of the text only experiment and, (b) Performance of MiSTER-E with varying values of λ (Eq. 18 ) for the two datasets. The validation set performance mirrors the trends of the test set. For IEMOCAP, validation accuracy is 63% for λ=0 and 63.6% for λ=2 . For MELD, λ=1 yields the highest validation performance ( 65.6% ) compared to 65.4% for λ=0
Figure 9: A case study from MELD where there are emotion shifts throughout the conversation. Out of the 7 utterances in the conversation, MiSTER-E correctly predicts 6 .
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials · School of Electronic Science and Engineering (School of Microelectronics), South China Normal University · LMU Munich & Munich Center for Machine Learning +2