Multimodal Emotion Recognition

Multimodal emotion recognition studies how to infer emotional categories, valence and arousal, or changing affective states from signals such as speech, language, facial and body movement, and physiological sensors. Common approaches include attention-based fusion, cross-modal transformers and graphs, modality-specific experts, and temporal models that use conversational context; newer systems also adapt pretrained multimodal models or add structured emotion reasoning. Combining channels can improve recognition when one cue is weak, but performance depends on handling missing, noisy, or conflicting evidence and differences across speakers, settings, and cultures. A central challenge is whether benchmark gains generalize to spontaneous real-world interactions, where emotion labels are subjective and the most informative modality can vary by person and situation.