Multimodal Sentiment Analysis

Follow this topic Ask about this topic

Multimodal sentiment analysis studies how words, vocal signals, and facial or body cues can be combined to understand sentiment and emotion in speech, video, and social media. Common approaches align and fuse modality representations with cross-attention, temporal models, or mixture-of-experts, while newer work also uses large language models, prompting, and structured reasoning. Benchmarks such as CMU-MOSI, CMU-MOSEI, and CH-SIMS support evaluation of sentiment prediction, including when inputs are noisy or modalities are missing. A central challenge is making fusion robust without letting one modality dominate; it remains unsettled when to repair missing inputs, how to measure each modality’s value, and whether more complex reasoning improves on efficient discriminative models.

Latest papers

54 papers7 papers in the last four weeks, down 22% on the four weeks before. 0.0% of all new papers.

Explicit Representation Alignment for Multimodal Sentiment Analysis
Jun 8, 2026Baode Wang, Ziming Wang, Huacan Wang +2Cross-Modal Alignment Sentiment Analysis
AgentAlpha
Beyond Generative Decoding: Discriminative Hidden-State Readout from a Native Omni-Modal LLM for Multimodal Sentiment Analysis
Jun 4, 2026Bin Wen, Tien-Ping TanEfficient Multimodal Inference Sentiment Analysis
School of Computer Sciences, Universiti Sains Malaysia, Penang, Malaysia.
A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis
May 27, 2026Jianheng Dai, Jiazhang Liang, Sijie MaiModality Imbalance Gradient Interference
School of Computer Science, South China Normal University, Guangzhou, Guangdong, China
Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
May 16, 2026Chenglizhao Chen, Yuchen Cao, Xinyu Liu +3Missing-Modality Learning Multimodal Robustness
Qingdao Institute of Software, College of Computer Science and Technology, China University of Petroleum (East China) · Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software · The Hong Kong University of Science and Technology (Guangzhou)
Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration
May 1, 2026Chunlei Meng, Pengbin Feng, Rong Fu +7Cross-Modal Learning Multi-Agent Collaboration
The College of Intelligent Robotics and Advanced Manufacturing, Fudan University · University of Southern California · University of Macau +5
Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective
Apr 20, 2026Sijie Mai, Shiqin HanDisentangled Representation Learning Causal Representation Learning
School of Computer Science, South China Normal University
Sentiment Analysis of German Sign Language Fairy Tales
Apr 17, 2026Fabrizio Nunnari, Siddhant Jain, Patrick GebhardSentiment Analysis Multimodal Sentiment Analysis
German Research Center for Artificial Intelligence (DFKI) Saarland Informatics Campus D3 2, 66123 Saarbrücken, Germany
Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis
Apr 14, 2026Kang He, Yuzhe Ding, Xinrong Wang +3Multimodal Sentiment Analysis Modality Imbalance
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute
MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes
Apr 13, 2026Samir Wagle, Reewaj Khanal, Abiral AdhikariHate Speech Detection Multimodal Sentiment Analysis
Department of Computer Science and Engineering, Kathmandu University, Nepal
QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis
Apr 7, 2026Yitong Zhu, Yuxuan Jiang, Guanxuan Jiang +4Multimodal Sentiment Analysis Multimodal Robustness
The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Aarhus University
Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities
Apr 7, 2026Rongfei Chen, Tingting Zhang, Xiaoyu Shen +1Multimodal Sentiment Analysis Missing-Modality Learning
School of Computer Science and Technology, Eastern Institute of Technology, Ningbo, China · School of Mechatronic Engineering and Automation, Shanghai University, China
CAGMamba: Context-Aware Gated Cross-Modal Mamba Network for Multimodal Sentiment Analysis
Apr 4, 2026Minghai Jiao, Jing Xiao, Peng Xiao +6Multimodal Sentiment Analysis Multimodal Fusion
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Xingning Power Supply Bureau, Guangdong Power Grid Co., Ltd., Meizhou, China
AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba
Mar 19, 2026Yan Li, Yifei Xing, Xiangyuan Lan +3Multimodal Sentiment Analysis Multimodal Fusion
Pengcheng Laboratory, Shenzhen, China · Shaanxi University of Science & Technology, Xi’an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, China
Face-Guided Sentiment Boundary Enhancement for Weakly-Supervised Temporal Sentiment Localization
Mar 16, 2026Cailing Han, Zhangbin Li, Jinxing Zhou +5Weakly Supervised Learning Multimodal Sentiment Analysis
Hefei University of Technology · MBZUAI · NUS +2
MissBench: Benchmarking Multimodal Affective Analysis under Imbalanced Missing Modalities
Mar 10, 2026Tien Anh Pham, Phuong-Anh Nguyen, Duc-Trong Le +1Multimodal Model Evaluation Missing-Modality Learning
VNU University of Engineering and Technology, Hanoi, Vietnam
C2F-Thinker: Coarse-to-Fine Reasoning with Hint-Guided Reinforcement Learning for Multimodal Sentiment Analysis
Mar 10, 2026Miaosen Luo, Zhenhao Yang, Jieshen Long +3Sentiment Analysis Multimodal Sentiment Analysis
School of Computer Science, South China Normal University Guangzhou, Guangdong, China · School of Electronic and Information Engineering, South China University of Technology Guangzhou, Guangdong, China
Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities
Mar 10, 2026Jindi Bao, Jianjun Qian, Mengkai Yan +1Multimodal Sentiment Analysis Missing-Modality Learning
Nanjing University of Science and Technology, Nanjing 210094, China · Hohai University, Nanjing 211100, China
FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis
Feb 28, 2026Xianxun Zhu, Zezhong Sun, Imad Rida +4Multimodal Federated Learning Multimodal Sentiment Analysis
Shanghai University, China · Nanyang Technological University, Singapore · Macquarie University, Australia +3
Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis
Feb 23, 2026Chunlei Meng, Jiabin Luo, Zhenglin Yan +4Multimodal Sentiment Analysis Multimodal Disentangled Representation Learning
Fudan University · Peking University · University of Macau
Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection
Feb 18, 2026Rong Fu, Ziming Wang, Shuo Yin +3Multimodal Emotion Recognition Multimodal Sentiment Analysis
Independent Researcher
Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysis
Feb 18, 2026Rong Fu, Ziming Wang, Chunlei Meng +4Machine Unlearning Multimodal Sentiment Analysis
Independent Researcher
Uncertainty-Guided Multimodal Consensus for Multimodal Sentiment Analysis
Oct 23, 2025Guowei Zhong, Junjie Li, Huaiyu Zhu +2Multimodal Sentiment Analysis Uncertainty-Aware Multimodal Fusion
College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027, China · Zhejiang University Jinhua Research Institute, Jinhua, 321000, China · College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, 310023, China
Resource-Limited Joint Multimodal Sentiment Reasoning and Classification via Chain-of-Thought Enhancement and Distillation
Aug 7, 2025Haonan Shangguan, Xiaocui Yang, Shi Feng +3Efficient Multimodal Inference Sentiment Analysis
School of Computer Science and Engineering, Northeastern University, Shenyang, China
Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis
Aug 7, 2025Menghua Jiang, Yuxia Lin, Baoliang Chen +3Multimodal Robustness Sentiment Analysis
School of Computer Science, South China Normal University, Guangzhou, China, 510631 · School of Electronics and Information Technology, Sun Yat-sen University, Guangzhou, China, 510275

Jun 8, 2026J/K move · Enter open · S save

Explicit Representation Alignment for Multimodal Sentiment Analysis

Baode Wang, Ziming Wang, Huacan Wang, Ronghao Chen+1

AgentAlpha

Cross-Modal Alignment Sentiment Analysis Multimodal Emotion Recognition Multimodal Sentiment Analysis Multimodal Fusion Multimodal Understanding

Open paper AskPDF arXiv

Multimodal affective analysis aims to understand human sentiment and emotion by jointly modeling heterogeneous modalities such as text and images. However, multimodal models often fail to consistently outperform strong text-only baselines, with performance varying significantly across fusion strategies. In this work, we identify representation misalignment between independently pretrained modality encoders as a key bottleneck for effective multimodal learning, and show through controlled experiments that alignment prior to fusion is often more important than fusion complexity. To address this issue, we propose a unified multimodal affective analysis framework that leverages vision-language models (VLMs) to convert visual content into structured textual descriptions, projecting heterogeneous modalities into a shared linguistic space and enabling interpretable text-centric reasoning. To further improve robustness, we introduce a hybrid learning strategy that combines semantic token selection with a batch-level uniformity regularization objective, encouraging a more dispersed and stable global feature space while mitigating noise introduced by VLM-generated descriptions. Experiments on multiple multimodal sentiment and emotion benchmarks show that our method consistently outperforms strong unimodal and multimodal baselines, achieving state-of-the-art performance. Our analysis further highlights the critical role of representation alignment in multimodal affective learning.