Multimodal Emotion Recognition

Latest papers 141

19 papers in the last four weeks, down 10% on the four weeks before. 0.1% of all new papers.

Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?
Jul 14, 2026Kaiwen Zheng, Junchen Fu, Wenhao Deng +3Efficient Multimodal Inference Multimodal Large Language Models
University of Glasgow United Kingdom · Institute of Computing Technology, Chinese Academy of Sciences China · School of Artificial Intelligence, Shandong University China +1
HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition
Jul 14, 2026Aleksei Bakin, Andrey V. SavchenkoMulti-Task Learning Multimodal Emotion Recognition
Central University, Moscow, Russia · Sber AI Lab, Moscow, Russia · HSE University, Laboratory of Algorithms and Technologies for Network Analysis, Nizhny Novgorod, Russia
Simple Features and Honest Calibration for Ambivalence and Hesitancy Recognition in Video
Jul 13, 2026Vikas Kumar, Aditya Mishra, Haroon R. LoneMultimodal Emotion Recognition Ambivalence and Hesitancy Recognition
Indian Institute of Science Education and Research Bhopal, India
SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits
Jul 9, 2026Adis Alihodzic, Selma Skopljakovic HubljarSHAP Feature Attribution Multimodal Emotion Recognition
Department of Mathematical and Computer Sciences, Faculty of Science, University of Sarajevo, Sarajevo, Bosnia and Herzegovina.
Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild
Jul 8, 2026Trang Nguyen, Sidong Zhang, Shiv Shankar +4Multimodal Emotion Recognition Affective Computing
University of Massachusetts Amherst, 130 Governors Drive, Amherst, MA, USA · Dolby Laboratories, 1275 Market Street, San Francisco, CA, USA
Rethinking the Role of Feature Engineering and Learning Strategies in Few-Shot Hidden Emotion Recognition
Jun 30, 2026Xiaochuan Guo, Jihao Gu, Haixu Liu +6Few-Shot Learning Multimodal Emotion Recognition
Hefei University of Technology, China · University College London, United Kingdom · The University of Sydney, Australia +5
MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy
Jun 26, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +8RL for Language Model Reasoning Multimodal Large Language Models
University of Science and Technology of China · 2SenseTime Research · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
Jun 24, 2026Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu +2Audio-Language Model Evaluation Voice Agent Evaluation
New York University · NVIDIA · Carnegie Mellon University +1
SAC $^2$ -Net: Semantic Anchoring and Complementary-Consensus Fusion for Multimodal Micro-Expression Recognition
Jun 24, 2026Xuepeng Zheng, Tong Chen, Chaoping Gui +3Cross-Modal Alignment Multimodal Emotion Recognition
College of Electronic and Information Engineering, Southwest University, Chongqing, 400715, China · Faculty of psychology, Southwest University, Chongqing, 400715, China · Chongqing Key Laboratory of Generic Technology and System of Service Robots, Southwest University, Chongqing, 400715, China +1
Omni-Perception Policy Optimization for Multimodal Emotion Reasoning
Jun 24, 2026Zhiyuan Han, Beier Zhu, Wenwen Tong +6Multimodal Reasoning Multimodal Hallucination
University of Science and Technology of China · SenseTime · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center +1
CDER-SME: A Cross-Device Event-RGB Micro-Expression Dataset under Multi-Level Stress Induction
Jun 16, 2026Jingting Li, Hui Sha, Su-Jing WangCross-Modal Alignment Multimodal Emotion Recognition
State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, 100101, China · Department of Psychology, University of the Chinese Academy of Sciences, Beijing, 100049, China · School of Computer Science, Jiangsu University of Science and Technology, Jiangsu, 212028, China
Deep Temporal Modeling and Ensemble Fusion for Multimodal Emotion Recognition from Physiological Signals
Jun 12, 2026Desta Haileselassie Hagos, Saurav Keshari Aryal, Patrick Ymele-Leki +2Temporal Convolutional Networks Deep Ensembles
Howard University Washington, DC, USA
Explicit Representation Alignment for Multimodal Sentiment Analysis
Jun 8, 2026Baode Wang, Ziming Wang, Huacan Wang +2Cross-Modal Alignment Sentiment Analysis
AgentAlpha
UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment
May 29, 2026Zheng Wang, Shuo Wang, Junhong WangUnsupervised Domain Adaptation Multimodal Emotion Recognition
Institute of Advanced Technology, University of Science and Technology of China, Hefei, 230026, China · Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230026, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230026, China
Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection
May 28, 2026Martha Teiko Teye, Yaw Marfo Missah, Emmanuel Ahene +2Sentiment Analysis Emotion Recognition in Conversations
Cluster of Excellence, University of Stuttgart, 70174 Stuttgart, German · Department of Computer Science, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana · Institute for Ethics in Artificial Intelligence, Technical University of Munich, 80333 Münich, Germany
Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach
May 27, 2026Anderson AugusmaMultimodal Emotion Recognition Privacy-Preserving ML
Thèse de Doctorat de l’Université Grenoble Alpes
From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition
May 24, 2026Dimitrios Kollias, Panagiotis Tzirakis, Alan Cowen +10Facial Action Unit Detection Valence-Arousal Modeling
Queen Mary University of London (QMUL), UK · Hume AI, USA · Google Deepmind, USA +7
Beyond Acoustic Emotion Recognition: Multimodal Pathos Analysis in Political Speech Using LLM-Based and Acoustic Emotion Models
May 21, 2026Juergen DietrichAudio-Language Model Evaluation Political Discourse Analysis
Democracy Intelligence gGmbH, Germany
Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow
May 21, 2026Chengsheng Zhang, Chenghao Sun, Zhining Xie +1VLM Hallucination VLM Interpretability
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · AIPD, Tencent
Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction
May 21, 2026Dinithi Dissanayake, Shaveen Silva, Ovindu Atukorala +2Multimodal Emotion Recognition Multimodal Fusion
Augmented Human Lab, National University of Singapore, Singapore · University of Moratuwa, Sri Lanka
Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition
May 20, 2026Junghyun Lee, Hyunseo Kim, Hanna Jang +1Multimodal Emotion Recognition Multimodal Fusion
Department of Artificial Intelligence and Software, Ewha Womans University, Seoul, Republic of Korea
Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognition
May 20, 2026Phuong-Anh Nguyen, The-Son Le, Duc-Trong Le +1Modality Imbalance Emotion Recognition in Conversations
Faculty of Information Technology, VNU University of Engineering and Technology, 100000, Hanoi, Vietnam.
AffectVerse: Emotional World Models for Multimodal Affective Computing
May 19, 2026Bo Zhao, Fanghua Ye, Yixin Ji +3Cross-Modal Learning Multimodal Large Language Models
1Great Bay University · 2Tencent · 3Tsinghua University +1
VISAFF: Speaker-Centered Visual Affective Feature Learning for Emotion Recognition in Conversation
May 18, 2026Linan ZHU, Zihao Zhai, Xiao Han +4VLM Adaptation Emotion Recognition in Conversations
1Zhejiang University of Technology · 2ETH Zurich, Switzerland
Navigating the Emotion Tree: Hierarchical Hyperbolic RAG for Multimodal Emotion Recognition
May 16, 2026Zeheng Wang, Bo Zhao, Yijie Zhu +6Multimodal RAG Hierarchical Classification
Great Bay University Guangdong, Dongguan, China · Tencent Youtu Lab Shanghai, China · CUHK, shenzhen Guangdong, Shenzhen, China
MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
May 14, 2026Tianwei Chen, Takuya Furusawa, Yuki Hirakawa +3Multi-Label Classification Multimodal Large Language Models
ZOZO NEXT Inc.
Multimodal Hidden Markov Models for Persistent Emotional State Tracking
May 13, 2026Anamika Ragu, Aneesh JonelagaddaEmotion Recognition in Conversations Valence-Arousal Modeling
Kaliber AI, San Mateo, California, USA.
EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
May 9, 2026Pengze Guo, Jingxi Liang, Zhiwen Xie +2Multimodal Large Language Models Multimodal Emotion Recognition
NLP2CT Lab, Department of Computer and Information Science, University of Macau · School of Computer Science, Central China Normal University · University of Macau
Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)
May 7, 2026Ankit SanjyalClass-Imbalanced Learning Multimodal Emotion Recognition
Department of Computer Science – Data Science Fordham University New York, USA
Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
May 7, 2026Xiwen Luo, Jia Li, Rencheng Song +2Multimodal Emotion Recognition Invariant Representation Learning
Department of Biomedical Engineering, and Anhui Province Key Laboratory of Measuring Theory and Precision Instrument, Hefei University of Technology, Hefei, 230009 China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601 China

Jul 14, 2026J/K move · Enter open · S save

Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

Kaiwen Zheng, Junchen Fu, Wenhao Deng, Hu Han+2

University of Glasgow United Kingdom · Institute of Computing Technology, Chinese Academy of Sciences China · School of Artificial Intelligence, Shandong University China+1

Efficient Multimodal Inference Multimodal Large Language Models Multimodal Emotion Recognition Knowledge Distillation

Open paper AskPDF arXiv

Recent advances in multimodal large language models (MLLMs) have significantly improved the performance of multimodal emotion recognition (MER) and enabled interpretable description generation by jointly modeling video, audio, and language, etc. However, these performance improvements are often accompanied by an increase in model parameter size (e.g, at least 7B), which simultaneously incurs high computational costs and reduces inference efficiency, thereby hindering real-time deployment on resource-constrained platforms such as robots and mobile devices. This raises a fundamental question: do we really need the multimodal MER model larger than 1B parameters for high-quality MER? In this paper, we challenge the assumption that larger models are inherently necessary and proposes a lightweight MER framework (called Light-MER), which achieves better and faster multimodal sentiment understanding and recognition through knowledge distillation. It can transfer knowledge from a strong, large-scale teacher model to a lightweight sub-billion-parameter student model, aiming to preserve rich multimodal emotion reasoning and recognition while substantially improving deployment efficiency. Specifically, we introduce two new optimization strategies to enhance knowledge transfer: (1) a new optimal transport loss that combines Sliced Wasserstein Distance with hidden-state alignment, and (2) a new multi-reward optimization strategy based on GRPO that balances MER performance and efficiency, aimed at further enhancing the learning capabilities of student models. Extensive experiments on nine benchmark datasets demonstrate that Light-MER achieves state-of-the-art performance while significantly improving inference efficiency. This highlights the strong potential of small multimodal emotion language models for future research. Code is available at https://github.com/GAIR-Lab/Light-MER.