Multimodal Robustness

Follow this topic Ask about this topic

Multimodal robustness asks whether systems that combine images, text, audio, medical data, or sensors can still make reliable predictions when some inputs are missing, corrupted, or in conflict. Common approaches include modality dropout, reliability-aware gating and expert routing, uncertainty estimation, and methods that separate shared information from modality-specific or spurious features. Researchers evaluate these methods on tasks such as medical segmentation, sentiment analysis, autonomous driving, and multimodal language-model reasoning, using corruption, missing-input, domain-shift, and adversarial benchmarks. A central tension is that adding modalities can improve performance when evidence is complementary but can also introduce interference or overreliance, and gains on controlled benchmarks do not always carry over to real deployment conditions.

Latest papers

279 papers26 papers in the last four weeks, down 10% on the four weeks before. 0.2% of all new papers.

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis
Oct 22, 2026Pan Wang, Lipeng Ke, Huajun Ying +5Multimodal Sentiment Analysis Multimodal Learning
Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA
Reliability-Aware Probabilistic Fusion for Multimodal Learning with Incomplete Evidence
Oct 21, 2026Patitapaban Palo, Abhisek Tiwari, Mattia Carletti +3Reliability-Aware Multimodal Fusion Uncertainty-Aware Multimodal Fusion
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, UK
MERP: A Latent-Bottleneck Framework for Robust Multi-Modal Multi-Label Emotion Recognition
Oct 21, 2026Chuangxin Zhao, Meimei Zhang, Canran Xiao +6Multimodal Emotion Recognition Multimodal Robustness
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Trine University, Angola, IN, USA · Sun Yat-sen University, Shenzhen Campus, Shenzhen, China +8
CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings
Oct 19, 2026Muntaqim Ahmed Raju, Priyanka Siddappa, Md Shifat Haider Al Amin +2Multimodal Fusion Chest X-Ray Classification
University of Massachusetts Lowell, USA · NYC Health & Hospitals/Jacobi-NCB, USA
Beyond Invariance: Factor-wise Selective Treatment of Shifts for Multimodal Learning
Oct 15, 2026Guimin Hu, Qiwei Peng, Boyan Xu +1Distribution Shift Robustness Multimodal Robustness
Guangdong University of Technology · University of Copenhagen
Reliability-Aware Multimodal Sentiment Prediction Under Local Modality Missingness
Oct 14, 2026Minghao Kong, Deng Yao, Xi Zhang +4Multimodal Sentiment Analysis Missing-Modality Learning
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen, China · Department of Network Intelligence, Pengcheng Laboratory, Shenzhen, China · School of Mathematics, Sun Yat-sen University, Guangzhou, China +1
On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs
Oct 14, 2026Mikołaj Piórczyński, Wojciech Masarczyk, Damian Sójka +2Test-Time Adaptation Test-Time Adaptation of VLMs
IDEAS Research Institute · Warsaw University of Technology · Pozna´n University of Technology +4
DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision
Oct 13, 2026Yifan Lan, Zhongye Liu, Yurui Chang +4Adversarial Robustness Multimodal Robustness
The Pennsylvania State University
SYNPRED: A Synergistic Approach to Multimodal Learning for Clinical Prediction
Oct 13, 2026Ivana Janíčková, Yen Y. Tan, Thomas H. Helbich +6Clinical Prediction Multimodal Disentangled Representation Learning
Computational Imaging Research Lab, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Comprehensive Center for Artificial Intelligence in Medicine, Medical University of Vienna, Austria · Christian Doppler Laboratory for Machine Learning Driven Precision Imaging, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria +3
SpikeOcc: Towards Resilient Multimodal 3D Occupancy Perception via Spiking Experts
Oct 11, 2026Yiding Sun, Zijie Xu, Hongqiang Lin +73D Occupancy Prediction Multimodal Robustness
Peking University · Xi’an Jiaotong University · Zhejiang University +1
ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail
Oct 11, 2026Anurag Sharma, Aditya Sharma, Sapna BaalwanMultimodal Model Evaluation Medical Image Benchmarks
Orthopaedic Surgeon · FutrBridge · Independent Researcher
Evidential Multimodal Transformers with Dempster-Shafer Fusion for Emotion Recognition
Oct 9, 2026Muhammad Usama, Yeoreum JungMultimodal Emotion Recognition Uncertainty-Aware Multimodal Fusion
Braindeck Inc., Seoul, Republic of Korea · School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea
SenseWeather: Multimodal Sensing for Robust Robotic Perception under Extreme Conditions
Oct 8, 2026Fucai Ke, Sukai Huang, Junyi Hao +4Robotic Perception Multimodal Robustness
Faculty of Information Technology, Monash University, Clayton, VIC 3800, Australia
Targeted Modality Dropout for Real-Robot Manipulation Robust to Intermittent Vision Loss
Oct 7, 2026Genki Shikada, Kazuki Osamura, Masaru Ide +2Missing-Modality Learning Multimodal Robustness
Fujitsu Limited, Kanagawa 211-8588, Japan. · Faculty of Science and Engineering, Waseda University, Tokyo 169-8050, Japan. · National Institute of Advanced Industrial Science and Technology, Tokyo 100-8921, Japan.
Beyond Perturbation Magnitude: Direction-Dependent Responses in Multimodal Geometric Representations
Oct 6, 2026Yongsheng Luo, Wengan He, Yu Li +2Neural Representation Geometry Multimodal Robustness
Zhuhai College of Science and Technology, Zhuhai, China · Zhuhai UNO Technology Co., Ltd., Zhuhai, China
MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
Oct 6, 2026Souptik Sen, Zahra AhmadiMissing-Modality Learning Audio-Visual Understanding
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed), Hannover, Germany
Dynamic Alignment and Calibration for Multimodal Learning
Oct 6, 2026Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu +1Multimodal Robustness Cross-Modal Alignment
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu,China · Tianyijiaotong Technology Ltd.,China · School of Computer Science and Technology, Hainan University, Haikou, China
Source-Learned Reliance for Selective Test-Time Adaptation of Multimodal Time Series
Oct 5, 2026Payal Mohapatra, Yueyuan Sui, Haodong Yang +3Missing-Modality Learning Test-Time Adaptation
Arm Inc., USA · Northwestern University, USA
SAFE-MR: Evidence Sufficiency Learning for Selective Multimodal Rumor Detection
Oct 5, 2026Shiwen NiMultimodal Robustness Selective Prediction
Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology
Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness
Oct 3, 2026Songyuan Sui, Zhen Tan, Mohan Zhang +3Missing-Modality Learning Multimodal Robustness
Rice University · Stevens Institute of Technology · University of North Carolina at Chapel Hill
OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning
Sep 30, 2026Xingming Shui, Dapeng Chen, Bowei Liu +5Audio-Visual Understanding Multimodal Robustness
Tsinghua Shenzhen International Graduate School, Tsinghua University · Independent Researcher
TSMD: Temporal-Stream Modality Dropout for Robust Video Highlight Detection
Sep 30, 2026Bo-Yuan Cheng, Kuan-Yu Chen, Po-Han Huang +2Missing-Modality Learning Multimodal Robustness
AI Research Center, Inventec Corporation, Taiwan · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan
ChartDensity-Bench: Benchmarking MLLMs for Numerical Data Reconstruction under Visual Density
Sep 30, 2026Xinhe Wu, Yadong JinMultimodal Robustness Chart Data Extraction
iSoftStone AI Research Lab (ISSAIR) iSoftStone Beijing, China
End-to-End Self-Supervised RGB-T Tracking without Modality Misleading
Sep 29, 2026Shenglan Li, Rui Yao, Kunyang Sun +4Multimodal Robustness Visual Object Tracking
School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, China. · Mine Digitization Engineering Research Center of the Ministry of Education, China. · University of Auckland, Auckland, New Zealand. +1
An Effective, Reliable, and Robust Framework for Human Activity Recognition Using Wearable Sensors
Sep 29, 2026Nafees Ahmad, Ho-fung Leung, Muhammad Adil Abid +1Human Activity Recognition Wearable Sensing
The Chinese University of Hong Kong, China · Independent Researcher, China · Malmö University, Sweden +1
Repair Before You Fuse: Frozen-Host Adaptation for Corrupted-but-Present Sensors
Sep 28, 2026Gia-Huy Thai, Quang-Thinh Ly, Anh-Minh Phan +1Lidar-Based 3D Object Detection Multimodal Robustness
University of Science, VNU-HCM, Ho Chi Minh City 700000, Vietnam · Michigan State University, East Lansing, MI 48824, USA · Center for Environmental Intelligence, VinUniversity, Hanoi 100000, Vietnam +1
Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity
Sep 26, 2026Yishu Zhang, Yun Li, Daiwei ZhangMultimodal Robustness Multimodal Large Language Models
University of North Carolina at Chapel Hill
Refinement Symmetry in Multimodal Transformers
Sep 26, 2026Yuhao Du, Shunian ChenAttention Mechanisms Multimodal Robustness
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
Representation Editing for Multimodal Test-Time Adaptation
Sep 26, 2026Longfei Huang, Xiangyu Wu, Yang YangTest-Time Adaptation Multimodal Robustness
Nanjing University of Science and Technology · Alibaba Group
Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis
Sep 24, 2026Menghua Jiang, Haokai Gao, Xiangui Kang +2Multimodal Sentiment Analysis Multimodal Robustness
School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Science, South China Normal University · School of Electronics and Information Technology, Sun Yat-sen University

Oct 22, 2026J/K move · Enter open · S save

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

Pan Wang, Lipeng Ke, Huajun Ying, Pritish Mohapatra+4

Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA

Multimodal Sentiment Analysis Multimodal Learning Multimodal Robustness

Open paper AskPDF arXiv

Multimodal sentiment analysis (MSA) aims to predict human sentiments by integrating signals from different modalities such as text, video, and audio. However, raw multimodal sequences often suffer from semantic inconsistency--exhibiting redundancy or conflicts within and across modalities--which hinders robust understanding and increases computational cost. To this end, we introduce ConsMSA, which explicitly formalizes semantic distribution consistency across both

\textit{intra}

- and

\textit{inter}

-modality, providing a consistency-aware mechanism for robust and efficient multimodal sentiment prediction. Specifically, ConsMSA projects multimodal token features into a shared sentiment space to compute an Intra- and Inter-modality Consistency Score (

I^2CS

). By coupling this score with predictive relevance, we formulate consistency-aware importance signals that are utilized: (i) as a consistency regularizer to align latent distributions during training, (ii) to derive semantic-aware weights for adaptive multimodal token reweighting, and (iii) as a practical criterion to prune redundant or conflicting tokens. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that ConsMSA achieves state-of-the-art performance while remaining robust under aggressive token compression--retaining only 10% of tokens yields comparable accuracy. These results establish semantic distribution consistency as a promising foundation for synergizing predictive robustness with computational efficiency.