Multimodal Emotion Recognition

Follow this topic Ask about this topic

Latest papers 141

19 papers in the last four weeks, down 10% on the four weeks before. 0.1% of all new papers.

All topics

Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG
Sep 3, 2026Birgit Nierula, Karam Tomotaki-Dawoud, Mert Akguel +5Multimodal Emotion Recognition Multimodal Fusion
Fraunhofer HHI, Berlin, Germany · Humboldt University of Berlin, Berlin, Germany
Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition
Sep 2, 2026Jiaqi Qiao, Yifan Lyu, Xiujuan XuMultimodal Emotion Recognition Emotion Recognition in Conversations
Dalian University of Technology
Video2Reaction: Training Foundation Video Models to Predict Audience Reaction
Sep 1, 2026Sidong Zhang, Trang Nguyen, Shiv Shankar +4VLM Adaptation Video-Language Models
UMass Amherst 130 Governors Drive, Amherst · Dolby Laboratories 1275 Market Street, San Francisco
Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Aug 31, 2026Jiaqi Zhang, Zheng Pang, Mengting Li +9Missing-Modality Learning Memory-Augmented Neural Networks
Jiangsu University · Griffith University · National University of Singapore +8
HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence
Aug 27, 2026Fei Ma, Zebang Cheng, Minghui Li +11Multimodal Model Evaluation Multimodal Emotion Recognition
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China. · Shenzhen University, Shenzhen, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, China. +6
LG-GER: Language-Guided Group Emotion Recognition via Multimodal Evidence Distillation
Aug 24, 2026Ahmed Shehab Khan, Zhiyuan Li, Yan TongEmotion Recognition Multimodal Emotion Recognition
University of South Carolina Columbia, South Carolina, USA
AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes
Aug 24, 2026Yibo Wang, Rui Yang, Jisheng Dang +7VLM Reasoning Affective Computing
Lanzhou University, Lanzhou, China · Hainan University, Haikou, China · School of Computing, National University of Singapore, Singapore
EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialogue
Aug 21, 2026Yi Zheng, Yifan Xu, Yan Zhou +8Affective Computing Emotion Recognition in Conversations
Neuro-Geospatial Modelling of EEG Affective States Using Literature-Informed Environmental Context
Aug 21, 2026Utsav Poudel, Jagannath Aryal, Subramaniyaswamy VairavasundaramEEG Emotion Recognition Multimodal Emotion Recognition
School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu 632014, India · Earth Observation and AI Research Group, Department of Infrastructure Engineering, The University of Melbourne, Melbourne, VIC 3010, Australia
TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs
Aug 18, 2026Muhammad Haseeb Aslam, Alessandro Koerich, Marco Pedersoli +2Test-Time Adaptation of VLMs Missing-Modality Learning
LIVIA, ETS Montreal, Canada · Aiim Lab, Queen’s University, Canada.
Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Models
Aug 17, 2026Xiutian Zhao, Luqi Sun, Björn Schuller +1Multimodal Foundation Models Multimodal Emotion Recognition
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Group on Language, Audio & Music (GLAM), Imperial College London, UK
MUPA $^{2}$ E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment
Aug 17, 2026Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz +1Multimodal Emotion Recognition Unified Multimodal Models
Honda Research Institute Japan, Wako City, Japan
E $^3$ mo-Bench: A Scalable Benchmark for Multimodal Evoked and Expressed Emotion Understanding via Bayesian Pairwise Alignment
Aug 11, 2026Lancheng Gao, Ziheng Jia, Shengyan Li +4Multimodal Large Language Models Multimodal Emotion Recognition
Shanghai Jiao Tong University
Rationale-Guided Learning for Multimodal Emotion Recognition
Aug 11, 2026Sujung Oh, Jung Uk Kim, Sangmin LeeMultimodal Reasoning Emotion Recognition in Conversations
Pixel Lab, Sungkyunkwan University, South Korea · Visual AI Lab, Kyung Hee University, South Korea · Pixel Lab, Korea University, South Korea
CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition
Aug 8, 2026Bojing Hou, Ruohao Li, Yitong Zhu +2Graph Neural Networks Multimodal Robustness
The Hong Kong University of Science and Technology (Guangzhou)
BioKD: Selective Physiology-to-Video Knowledge Distillation via Reliability Gate for Emotion Recognition
Aug 6, 2026Bojing Hou, Ruohao Li, Yitong Zhu +3Cross-Modal Knowledge Distillation Selective Knowledge Distillation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · New York University, New York, NY, USA
HyPASE: Hyperbolic Geometry for Parameter-Efficient Speech Emotion Fine-Tuning Framework for Large Audio-Language Models
Aug 5, 2026Tian Jin, Ruikang Zhang, Zefeng Zhao +2Audio-Language Models Multimodal Emotion Recognition
Tongji University Shanghai, China · Peking University Beijing, China · The Chinese University of Hong Kong, Shenzhen Shenzhen, China
ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models
Aug 4, 2026Xiaolin Chen, Xuemeng Song, Wenhao Shi +3Multimodal Large Language Models Multimodal Emotion Recognition
National University of Singapore · Southern University of Science and Technology · Nanyang Technological University
Context-Aware Mixture of Domain Experts for Bodily Expression of Emotion in the Wild
Aug 3, 2026Mohammad Mahdi Dehshibi, David MasipMultimodal Emotion Recognition Mixture-of-Experts Models
Department of Computer Science, University of the West of England, Bristol, U.K. · Department of Computer Science, Universitat Oberta de Catalunya, Barcelona, Catalonia, Spain.
CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition
Jul 31, 2026Wenzhuo Sun, Mingjian Liang, Richard Attfield +3Ensemble Learning Multimodal Emotion Recognition
Monash University, Melbourne, Australia · Monash Suzhou Research Institute, Suzhou, China
Knowledge-Guided Multimodal Reasoning over Interacting Streams for Video-Level Ambivalence and Hesitancy Recognition
Jul 28, 2026Podakanti Satyajith Chary, Barath Parthiban, Pranesh Velmurugan +2Multimodal Reasoning Multimodal Emotion Recognition
Department of Engineering Science, IIT Hyderabad, India · Department of Biomedical Engineering, IIT Hyderabad, India
Looking for Affect in Spontaneous Finnish Speech through Linguistic Interpretability
Jul 27, 2026Kalle Lahtinen, Liisa Mustanoja, Okko RäsänenValence-Arousal Modeling Multimodal Emotion Recognition
Signal Processing Research Centre, Tampere University, Finland · Research Centre Plural, Tampere University, Finland
Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions
Jul 24, 2026Chen-Yang Xu, Lan Zhang, Fei-Yi Fan +2Valence-Arousal Modeling Multimodal Emotion Recognition
School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China
MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement
Jul 23, 2026Daiqing Wu, Dongbao Yang, Jiashu Yao +4VLM Evaluation Multimodal Emotion Recognition
Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing 100084, China · Nankai University, Tianjin 300071, China · Beijing Institute of Technology, Beijing 100081, China +1
EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization
Jul 23, 2026Lihuang Fang, Yuchen Zou, kebing Jin +1RL for Language Models Agentic RL
Guangdong University of Technology Guangzhou, China · Southern University of Science and Technology Shenzhen, China · Xi’an Jiaotong University Xi’an, China +1
Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation
Jul 20, 2026Jiabing Yang, Yixiang Chen, Yuan Xu +6Human Preference Evaluation Pairwise Preference Evaluation
University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences Beijing, China
SpEmoC: A Balanced Speaker-Segment Multimodal Emotion Benchmark
Jul 20, 2026Sania Bano, Shahzad Ahmad, Santosh Kumar Vipparthi +2Emotion Recognition in Conversations Multimodal Emotion Recognition
CVPR Lab, Indian Institute of Technology Ropar, India · Østfold University College, Norway · CVPR Lab, SCSS, Trinity College Dublin, Ireland
EII-SCL: Harnessing Emotional Inertia for Multimodal Emotion Recognition in Conversation
Jul 19, 2026Zilong Huang, Kong Aik Lee, Chong-Xin Gan +3Emotion Recognition in Conversations Multimodal Emotion Recognition
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Jul 19, 2026Zilong Huang, Kong Aik Lee, Junjie Li +2Emotion Recognition in Conversations Multimodal Emotion Recognition
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University · Speech, Language, and Cognition Laboratory, The University of Hong Kong
InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring
Jul 16, 2026Hao Yang, Yanyan Zhao, Kewei Zhao +11Multimodal Robustness Multimodal Emotion Recognition
Harbin Institute of Technology · SERES

Sep 3, 2026J/K move · Enter open · S save

Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG

Birgit Nierula, Karam Tomotaki-Dawoud, Mert Akguel, Mustafa Tevfik Lafci+4

Fraunhofer HHI, Berlin, Germany · Humboldt University of Berlin, Berlin, Germany

Multimodal Emotion Recognition Multimodal Fusion Emotion Recognition

Open paper AskPDF arXiv

Head-mounted displays (HMDs) fundamentally limit emotion recognition in virtual reality (VR): by occluding the upper face, they render conventional image-based facial expression analysis incomplete, particularly for applications requiring real-time affective assessment. We address this challenge by fusing lower-face video with facial electromyography (EMG) from the occluded upper face to classify seven emotional categories (six basic emotions plus neutral). We introduce a synchronized multimodal dataset from 20 participants, pairing lower-face video with seven-channel upper-face EMG elicited by validated emotion stimuli. Under subject-independent test, our proposed late-fusion architecture merging convolutional visual embeddings with RBF-kernel EMG representations achieves 51% macro-F1, outperforming both image-only (41%) and EMG-only (43%) baselines. These results demonstrate that upper-face EMG provides robust complementary information under HMD-induced visual occlusion and establish a foundation for multimodal emotion recognition in naturalistic VR environments. This approach facilitates affect-adaptive applications, including communication training and therapeutic interventions. The dataset will be shared upon request under an ethical-use agreement.