Multimodal Sentiment Analysis

Follow this topic Ask about this topic

Multimodal sentiment analysis studies how words, vocal signals, and facial or body cues can be combined to understand sentiment and emotion in speech, video, and social media. Common approaches align and fuse modality representations with cross-attention, temporal models, or mixture-of-experts, while newer work also uses large language models, prompting, and structured reasoning. Benchmarks such as CMU-MOSI, CMU-MOSEI, and CH-SIMS support evaluation of sentiment prediction, including when inputs are noisy or modalities are missing. A central challenge is making fusion robust without letting one modality dominate; it remains unsettled when to repair missing inputs, how to measure each modality’s value, and whether more complex reasoning improves on efficient discriminative models.

Latest papers

53 papers7 papers in the last four weeks, down 22% on the four weeks before. 0.0% of all new papers.

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis
Oct 22, 2026Pan Wang, Lipeng Ke, Huajun Ying +5Multimodal Sentiment Analysis Multimodal Learning
Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA
I $^2$ -KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis
Oct 22, 2026Xiaoting Xie, Feng Qiao, Xianwei Zhang +5Multimodal Sentiment Analysis Multimodal Fusion
School of Information Science and Engineering, Shandong University, Qingdao, Shandong, China · Department of Computer Science, University of Exeter, Exeter, UK · School of Computing Science, University of Glasgow, Glasgow, UK +1
Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis
Oct 20, 2026Xiaohua Huang, Bingrui Lei, Xiuyan Shao +4Multimodal Sentiment Analysis Modality Reliance in Multimodal Models
the School of Communication and Information Engineering, Nanjing Institute of Technology, Nanjing 210096, China. · the School of Economics and Management, Southeast University, Nanjing 211196, China. · the School of Computer Science and Engineering, Macau University of Science and Technology, Macau SAR, 999078, China. +2
Beyond Invariance: Factor-wise Selective Treatment of Shifts for Multimodal Learning
Oct 15, 2026Guimin Hu, Qiwei Peng, Boyan Xu +1Distribution Shift Robustness Multimodal Robustness
Guangdong University of Technology · University of Copenhagen
Reliability-Aware Multimodal Sentiment Prediction Under Local Modality Missingness
Oct 14, 2026Minghao Kong, Deng Yao, Xi Zhang +4Multimodal Sentiment Analysis Missing-Modality Learning
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen, China · Department of Network Intelligence, Pengcheng Laboratory, Shenzhen, China · School of Mathematics, Sun Yat-sen University, Guangzhou, China +1
InstMoE: Adaptive Multimodal Routing with Specialized Experts
Oct 4, 2026Guimin Hu, Xiang He, Yingjian Li +3Cross-Modal Alignment Multimodal Sentiment Analysis
Guangdong University of Technology · University of Copenhagen · Pengcheng Laboratory +1
Prediction-Layer Branch Calibration for Multimodal Sentiment Analysis
Sep 29, 2026Yulin Sun, Kele Xu, Yong DouMultimodal Sentiment Analysis Multimodal Fusion
National University of Defense Technology
Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis
Sep 24, 2026Menghua Jiang, Haokai Gao, Xiangui Kang +2Multimodal Sentiment Analysis Multimodal Robustness
School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University
SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data
Sep 24, 2026Wenhao Li, Zhibin Wu, Chong Xiao +1Missing-Modality Learning Multimodal Robustness
Software School, Shandong University · Shenzhen Loop Area Institute
Divide and Conquer: Mixture-of-Bottleneck Experts in Informative Ordinal Space for Video-based Multimodal Sentiment Analysis
Sep 16, 2026Ronghao Lin, Qiaolin He, Zefeng Lu +5Mixture of Experts Multimodal Sentiment Analysis
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China · Nanyang Technological University, Singapore, 639798 · School of Electronics and Information Technology, Sun Yat-sen University, Guangzhou 510006, China +4
The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods
Sep 10, 2026Ioanna Kaffeza, Efthymios Georgiou, Alexandros PotamianosModality Imbalance Multimodal Optimization
PERSEE Center, Mines Paris-PSL University, Sophia-Antipolis, France · National Technical University of Athens, Athens, Greece · University of Bern, Bern, Switzerland +2
Robust Multimodal Sentiment Analysis with Incomplete Modalities via Semantic-aware Completeness based Reconstruction
Sep 10, 2026Han-Jun Choi, Byunggill Joe, Saim Shin +1Missing-Modality Learning Multimodal Robustness
Korea Electronics Technology Institute (KETI) Seongnam, South Korea
Fine-Grained Visual Preprocessing and Dual-Stream Temporal Modeling for Multimodal Sentiment Analysis on Social Media
Sep 7, 2026Su Li, Yigong Zhang, Lei Xiong +1Sentiment Analysis Multimodal Sentiment Analysis
School of Information Engineering, Kunming University, No. 2 Puxin Road, Guandu District, Yangpu Campus, Kunming, 650214, Yunnan, China
Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis
Sep 6, 2026Xiaoran Kou, Jingyi Wu, Peng Sun +2Multimodal Sentiment Analysis Prompt Tuning
College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China
MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
Sep 5, 2026Pengfei Shao, Jisheng Dang, Jiawen Fang +7Multimodal Sentiment Analysis Multimodal Emotion Recognition
School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China +2
Multimodal Adaptive Expert Selection with Text Routing and Ordinal Prototype Optimization for Sentiment Analysis
Aug 31, 2026Xiaode Chen, Jiakang Yu, Hongtao Deng +3Representation Learning Mixture of Experts
Jianghan University Wuhan, China
Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combination
Aug 20, 2026Kaixin Xu, NaiJin Liu, Yulin Kang +8Multimodal Sentiment Analysis Multimodal Learning
School of Software Technology, Zhejiang University · Beijing WuZi University · Ant Group +3
Robust Incomplete Multimodal Sentiment Analysis via Iterative Proxy Correction
Aug 20, 2026Zhifa Geng, Subin Huang, Hao Guo +3Multimodal Sentiment Analysis Missing-Modality Learning
School of Computer and Information Anhui Polytechnic University Wuhu, China · Bigdata and Responsible Artificial Intelligence for National Governance, Renmin University of China, Beijing 100872, China
Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis
Aug 17, 2026Shanshan Lin, Yuesheng Wu, Chao Chen +4Sentiment Analysis Multimodal Sentiment Analysis
Fuzhou University, Fuzhou, China · Harbin Institute of Technology (Shenzhen), Shenzhen, China · Jiangxia University, Fuzhou, China
MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis
Aug 6, 2026Yuhua Wen, Yingying Zhou, Qifei Li +4Missing-Modality Learning Multimodal Disentangled Representation Learning
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Zhongguancun Academy, Beijing 100094, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China +1
Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations
Aug 4, 2026Chunlei Meng, Jacqueline J. Pang, Pengbin Feng +3Missing-Modality Learning Multimodal Sentiment Analysis
The College of Intelligent Robotics and Advanced Manufacturing (CIRAM), Fudan University · Cornell University · University of Southern California (USC) +1
Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis
Jul 30, 2026Wei Chen, Junkai Li, Tongguan Wang +4Multimodal Large Language Models Multimodal Classification
Huazhong Agricultural University Wuhan, China · Hubei University Wuhan, China · Engineering Research Center of Intelligent Technology for Agriculture, Hubei Engineering Technology Research Center of Agricultural Big Data, Key Laboratory of Smart Farming for Agricultural Animals.
Should Missing Modalities Always Be Necessary to Repair for Multi-modal Sentiment Analysis?
Jul 19, 2026Yubo Gao, Haotian Wu, Xiaoyu Xu +9Missing-Modality Learning Multimodal Sentiment Analysis
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Nanyang Technological University +2
Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis
Jul 14, 2026Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2Cross-Modal Representation Learning Multimodal Large Language Models
National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2
CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection
Jul 13, 2026Qiyang Sun, Yi Chang, Yupei Li +3Language Model Calibration Multimodal Sentiment Analysis
GLAM – the Group on Language, Audio, & Music, Imperial College London, UK · College of Telecommunications and Information Engineering, Nanjing University of Posts and Telecommunications, China · College of Computer Science and Electronic Engineering, Hunan University, China +2
MRUF: Multi-granularity Routing with Uncertainty-Aware Fusion for Robust Multimodal Sentiment Analysis
Jul 12, 2026Haoran Ma, Yinfeng Yu, Liejun WangMultimodal Sentiment Analysis Multimodal Robustness
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China. · Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing. · Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center. +2
SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits
Jul 9, 2026Adis Alihodzic, Selma Skopljakovic HubljarMixture-of-Experts Models Multimodal Emotion Recognition
Department of Mathematical and Computer Sciences, Faculty of Science, University of Sarajevo, Sarajevo, Bosnia and Herzegovina.
Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts
Jul 7, 2026Andrei-George Durdun, Victor Constantinescu, Radu Tudor IonescuCross-Modal Learning Sentiment Analysis
Department of Computer Science, University of Bucharest, 15G Iuliu Maniu, Bucharest 061075, Romania · Department of Data Science, PPC Romania, 30 Mircea Voda, Bucharest 030667, Romania
GRACE: Boosting Video MLLMs with Grounded Action-Centric Evidence for Viewer Sentiment Prediction
Jun 15, 2026Ruoxuan Yang, Tieyuan Chen, Xiaofeng Huang +6VLM Reasoning Multimodal Sentiment Analysis
Shanghai Jiao Tong University, China · Hangzhou Dianzi University, China · The 52nd Research Institute of China Electronics Technology Group Corporation,2026 China +4
MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs
Jun 14, 2026Hangling XieMultimodal ICL Multimodal Large Language Models
Nanjing University of Posts and Telecommunications, Wenyuan Road No.9, Xixia District, Nanjing City, Jiangsu Province, China

Oct 22, 2026J/K move · Enter open · S save

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

Pan Wang, Lipeng Ke, Huajun Ying, Pritish Mohapatra+4

Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA

Multimodal Sentiment Analysis Multimodal Learning Multimodal Robustness

Open paper AskPDF arXiv

Multimodal sentiment analysis (MSA) aims to predict human sentiments by integrating signals from different modalities such as text, video, and audio. However, raw multimodal sequences often suffer from semantic inconsistency--exhibiting redundancy or conflicts within and across modalities--which hinders robust understanding and increases computational cost. To this end, we introduce ConsMSA, which explicitly formalizes semantic distribution consistency across both

\textit{intra}

- and

\textit{inter}

-modality, providing a consistency-aware mechanism for robust and efficient multimodal sentiment prediction. Specifically, ConsMSA projects multimodal token features into a shared sentiment space to compute an Intra- and Inter-modality Consistency Score (

I^2CS

). By coupling this score with predictive relevance, we formulate consistency-aware importance signals that are utilized: (i) as a consistency regularizer to align latent distributions during training, (ii) to derive semantic-aware weights for adaptive multimodal token reweighting, and (iii) as a practical criterion to prune redundant or conflicting tokens. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that ConsMSA achieves state-of-the-art performance while remaining robust under aggressive token compression--retaining only 10% of tokens yields comparable accuracy. These results establish semantic distribution consistency as a promising foundation for synergizing predictive robustness with computational efficiency.