Multimodal Emotion Recognition

Ask about this topic

Overview

Multimodal emotion recognition studies how to infer emotional categories, valence and arousal, or changing affective states from signals such as speech, language, facial and body movement, and physiological sensors. Common approaches include attention-based fusion, cross-modal transformers and graphs, modality-specific experts, and temporal models that use conversational context; newer systems also adapt pretrained multimodal models or add structured emotion reasoning. Combining channels can improve recognition when one cue is weak, but performance depends on handling missing, noisy, or conflicting evidence and differences across speakers, settings, and cultures. A central challenge is whether benchmark gains generalize to spontaneous real-world interactions, where emotion labels are subjective and the most informative modality can vary by person and situation.

Latest papers

CardsList

152 papers15 papers in the last four weeks, down 29% on the four weeks before. 0.1% of all new papers.

  1. Beyond Multimodal Emotion Conflict: Temporal Context-Aware Hidden Emotion Detection with Multimodal Large Language Models

    Nov 1, 2026Yonghao Wang, Jiaxin ZhouEmotion RecognitionMultimodal Emotion Recognition

    The Chinese University of Hong Kong, China · Lingnan University, China

  2. Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models

    Oct 29, 2026Fengji Jing, Guy LabanMultimodal Emotion RecognitionAudio-Language Model Evaluation

    Faculty of Science, The University of Hong Kong, Hong Kong, China · Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, Beer Sheva, Israel · School of Brain Sciences and Cognition, Ben-Gurion University of the Negev, Beer Sheva, Israel +1

  3. Beyond Vision-Language Models: Visual-Semantic Affect Graphs for Emotion Recognition in Artworks

    Oct 26, 2026Ngoc-Duy Mai, Ha-Thanh Nguyen, Hideaki TakedaMultimodal Emotion RecognitionValence-Arousal Modeling

    National Institute of Informatics, Tokyo, Japan · The Graduate University for Advanced Studies, SOKENDAI, Kanagawa, Japan · Research and Development Center for Large Language Models, NII, Tokyo, Japan +1

  4. Traffic-Context-Aware Driver Emotion Recognition with Expectation-Deviation-Driven Inference

    Oct 24, 2026Longhao Yan, Jinhao Liang, Kaidi YangMultimodal Emotion RecognitionDriver Monitoring

    Department of Civil and Environmental Engineering, National University of Singapore, Singapore 117576.

  5. CASE: Confusion-Aware Skill Evolution for Frozen Multimodal Emotion Models

    Oct 23, 2026Siyu Wu, Zeyu Liang, Linghao Wang +4Multimodal Emotion RecognitionMultimodal Large Language Models

    Jilin University · National Chiao Tung University · National Tsing Hua University

  6. AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning

    Oct 22, 2026Chuan Han Li, Yun Chien ChengMultimodal Emotion RecognitionMultimodal Fusion

    Department of Mechanical Engineering, College of Engineering, National Yang Ming Chiao Tung University, Hsin-Chu, Taiwan · Department of Electrical Engineering, College of Engineering, National Taiwan University, Taipei, Taiwan

  7. MERP: A Latent-Bottleneck Framework for Robust Multi-Modal Multi-Label Emotion Recognition

    Oct 21, 2026Chuangxin Zhao, Meimei Zhang, Canran Xiao +6Multimodal Emotion RecognitionMultimodal Robustness

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Trine University, Angola, IN, USA · Sun Yat-sen University, Shenzhen Campus, Shenzhen, China +8

  8. PI-MLLM: Psychophysics-Inspired Multimodal Large Language Model for Oddly Satisfying Video Emotion Analysis

    Oct 20, 2026Yuheng Zhang, Jie Zhang, Yuecong Min +4Multimodal Emotion RecognitionMultimodal Large Language Models

    Institute of Computing Technology, Chinese Academy of Sciences · CNCERT/CC · Nanyang Technological University +1

  9. Multiscale Temporal-Spectral-Spatial EEG Learning With Modality-Specific and Modality-Invariant Representations for Emotion Recognition

    Oct 20, 2026Jian Zhang, Manrong Liu, Yilei Huang +3EEG Emotion RecognitionMultimodal Emotion Recognition

    Henan Engineering Research Center for Artificial Intelligence Theory and Algorithms, School of Mathematics and Statistics, Henan University, Kaifeng, 475004, China. · The Second Clinical College, Henan University, Kaifeng, 475004, China.

  10. Context-Aware Topological Distillation Graph Network for Asymmetric EEG-Eye Movement Emotion Recognition

    Oct 19, 2026Peijun Shi, Xiujin Zhu, Zhenguo Lao +4Multimodal Emotion RecognitionEEG Emotion Recognition

    Department of Electrical Engineering, Faculty of Engineering, Universiti Malaya, 50603 Lembah Pantai, Kuala Lumpur, Malaysia. · Department of Neurosurgery, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, Hubei 430022, China · Hubei Provincial Clinical Research Center for Brain Computer Interface, Huazhong University of Science and Technology, Wuhan, Hubei 430022, China.

  11. GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition

    Oct 13, 2026Ting Zhu, Huizhi Liang, Boguslaw Obara +2Multimodal Emotion RecognitionMultimodal Fusion

    Newcastle University, Newcastle upon Tyne, United Kingdom · Taiyuan University of Technology, Shanxi, China

  12. EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

    Oct 13, 2026Zixuan Xing, Lancheng Gao, Shengyan Li +4T2I Generation EvaluationMultimodal Emotion Recognition

    Shanghai Jiao Tong University

  13. Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition

    Oct 9, 2026Muhammad Usama, Yeoreum JungMultimodal Emotion RecognitionUncertainty-Aware Multimodal Fusion

    Braindeck Inc., Seoul, Republic of Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea

  14. From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding

    Oct 8, 2026Jia Li, Yichao He, Yangchen Yu +6Multimodal Emotion RecognitionEmotion Recognition

    Hefei University of Technology · Nanyang Technological University

  15. Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes

    Oct 8, 2026Hao Li, Jinye Zhang, Bobo Li +5Affective ComputingMultimodal Emotion Recognition

    School of Computer Science, National Engineering Research Center for Multimedia Software, Wuhan University, Wuhan 430072, China

  16. From Expression to Reaction: Role-aware Visual Transfer and Stimulus-guided Reasoning for Interlocutor Emotion Recognition

    Oct 2, 2026Wei Wang, Zhaowu Li, Jianjie Luo +3Emotion Recognition in ConversationsMultimodal Emotion Recognition

    Guangdong University of Technology Guangzhou, China · Hong Kong Metropolitan University Hong Kong, China

  17. Modeling Shared and Individual Structure for Cross-Subject Continuous Affect Regression from EEG-fNIRS

    Oct 2, 2026Xuan Wang, Bing Wang, Shuai Chang +3Valence-Arousal ModelingAffective Computing

  18. LongEmo: Towards Emotion Understanding and Reasoning in Long Videos

    Sep 30, 2026Shuo Zhang, Yifan Zhou, Han Wang +15Memory-Augmented VLMsMultimodal Emotion Recognition

    BUPT · SJTU · THU +6

  19. Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue

    Sep 30, 2026Yutong Hu, Jinho ChoiEmotion Recognition in ConversationsMultimodal Emotion Recognition

    Emory University Atlanta, GA

  20. VISTA: Value-Informed Event Appraisal for Multimodal Emotion Conflict

    Sep 29, 2026Jiale Dai, Liuxian Ma, Xiaoke Niu +5Multimodal Emotion RecognitionMultimodal Fusion

    State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · College of Artificial Intelligence, Tsinghua University · China United Network Communications Group Co., Ltd.

  21. Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

    Sep 29, 2026Cheng Ye, Weidong Chen, Zhaobo Qi +2Multimodal Large Language ModelsMultimodal Emotion Recognition

    University of Science and Technology of China, Hefei · Harbin Institute of Technology, Weihai

  22. Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention

    Sep 29, 2026Cheng Ye, Weidong Chen, Peipei Song +1Video CaptioningMultimodal Emotion Recognition

    University of Science and Technology of China, Hefei

  23. AffectReveal: Event-Grounded Emotion Recognition Beyond Visual Appearances

    Sep 29, 2026Yihao Qian, Runhao Zeng, Sicheng Zhao +3Evidence-Grounded ReasoningMultimodal Emotion Recognition

    South China University of Technology, Guangzhou, China · Shenzhen MSU-BIT University, Shenzhen, China · Tsinghua University, Beijing, China

  24. T-SNN: Temporal Simplicial Neural Network for EEG Decoding

    Sep 27, 2026Nikita Malik, Shubhajit Roy, Mohit Kataria +4Cross-Subject EEG DecodingEEG Decoding

    Bharti School of Telecommunication Technology and Management Indian Institute of Technology, Delhi · Department of Computer Science & Engineering Indian Institute of Technology, Gandhinagar · Yardi School of Artificial Intelligence Indian Institute of Technology, Delhi +4

  25. Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition

    Sep 25, 2026Philip H. Lee, Shreeram Suresh Chandra, John H. L. HansenEEG Artifact RemovalDifferential Attention

    Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, Baltimore, MD, USA · Center for Robust Speech Systems (CRSS), University of Texas at Dallas, Richardson, TX, USA

  26. Oracle Complementarity Is Not Realizable Complementarity in Frozen-Encoder Audio-Visual Emotion Recognition

    Sep 24, 2026Benjamin HurtMultimodal FusionMultimodal Emotion Recognition

    University of California, Santa Barbara

  27. BiCFlow-MER: Orchestrating Discriminative and Generative Multimodal Emotion Recognition via Conditional Transport

    Sep 23, 2026Yanbing Wang, Shenyue Wang, Chunyang YuEmotion Recognition in ConversationsMultimodal Emotion Recognition

    OPPO Research Institute

  28. From Stress to Affect: Multimodal Deep Learning for Physiological Emotion Recognition Across Wearable Sensor Modalities

    Sep 17, 2026Desta Haileselassie Hagos, Saurav Keshari Aryal, Legand L. BurgeEmotion RecognitionMultimodal Emotion Recognition

    Department of Electrical Engineering and Computer Science, Howard University, Washington DC, USA

  29. Affective Shared Autonomy: Temporal Affect Dynamics and Subjective Evaluation in Bimanual Teleoperation Tasks

    Sep 17, 2026Zhengji Liang, Guiyin Tian, Sijin Qu +2Robot TeleoperationMultimodal Emotion Recognition

    Department of Data and Systems Engineering, The University of Hong Kong, Hong Kong SAR, China. · Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.

  30. Emotion Experience, Expression, and Perception: Emotion Analysis on Multimodal Social Media Posts

    Sep 16, 2026Christopher Bagdon, Carina Silberer, Roman KlingerSocial Media AnalysisMultimodal Emotion Recognition

    Fundamentals of Natural Language Processing, University of Bamberg, Germany · Institut für Maschinelle Sprachverarbeitung, University of Stuttgart, Germany