Multimodal Learning

Follow this topic Ask about this topic

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

335 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis
Oct 22, 2026Pan Wang, Lipeng Ke, Huajun Ying +5Multimodal Sentiment Analysis Multimodal Learning
Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA
AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning
Oct 22, 2026Chuan Han Li, Yun Chien ChengMultimodal Emotion Recognition Multimodal Fusion
Department of Mechanical Engineering, College of Engineering, National Yang Ming Chiao Tung University, Hsin-Chu, Taiwan · Department of Electrical Engineering, College of Engineering, National Taiwan University, Taipei, Taiwan
Cross-Dimension and Cross-Modality Relational Tensor Representation Learning for Multimodal Health Data Modeling
Oct 17, 2026Jiawei Guo, Meng ZhaoMultimodal Learning Cross-Modal Representation Learning
Department of Industrial and Systems Engineering, Lehigh University
CIIR - Class-Indexed Interface Representation Learning
Oct 16, 2026Doo Yon Kim, Christian WallravenCross-Modal Representation Learning Cross-Modal Representation Transfer
TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning
Oct 15, 2026Liang Nathan Zheng, Jiyuan Liu, Wenhao Liang +3Missing-Modality Learning Multimodal Learning
School of Computer Science and Information Technology, Adelaide University, Adelaide, SA 5000, Australia
FlexUMM: Flexible Representation Learning for Unified Multimodal Models
Oct 15, 2026Yinyi Luo, Haoyue Bai, Yi Ge +3Task Interference Unified Multimodal Models
Carnegie Mellon University · William & Mary · University of Maryland
FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models
Oct 15, 2026Jiaming Zhang, Zherui Li, Siqi Guo +3VLM Robustness Adversarial Attacks on VLMs
College of Computing and Data Science Nanyang Technological University, Singapore
TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data
Oct 15, 2026Max Rosenblattl, Milana Gurbanova, Karl Aurel Deck +16Time Series Language Modeling Wearable Health Monitoring
Stanford University, USA · Aionic Labs, Switzerland · Agentic Systems Lab, ETH Zurich, Switzerland +7
MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data
Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal Learning Multimodal Foundation Models
University of California, Riverside · Amazon
Task-Centric Multimodal Learning for Cognitive Impairment Detection
Oct 14, 2026Sankalp Vaish, Rishank Singh, Sumukh Veeramalla +3Multimodal Learning Speech-Based Dementia Detection
Department of Computer Science, University of Massachusetts Boston, MA, USA · School of Medicine, University of North Carolina at Chapel hill, NC, USA
CogDrift-4D: Hierarchical Multimodal Modeling of Temporal Behavioral Decay for MCI Detection
Oct 14, 2026Sankalp Vaish, Sumukh Veeramalla, Rishank Singh +3Dementia Care Speech-Based Dementia Detection
Department of Computer Science, University of Massachusetts Boston, MA, USA · School of Medicine, University of North Carolina at Chapel hill, NC, USA
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
Oct 11, 2026Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu +4Multimodal Learning Multimodal Sensor Fusion
Center for Machine Vision and Signal Analysis, University of Oulu, Finland · Candour Ltd, Oulu, Finland
Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning
Oct 10, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive Learning Multimodal Learning
Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
Oct 8, 2026Qun Dai, Liangjian Wen, Jiang Duan +7Multimodal SSL Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5
PXtal: Learning to Align Powder X-Ray Diffraction and Crystal Structures under Information Asymmetry across Modalities
Oct 7, 2026Zhuoran Yang, Christopher M. Collins, Bei Peng +3Cross-Modal Alignment Multimodal Learning
School of Computer Science and Informatics, University of Liverpool, Liverpool, UK · Leverhulme Research Centre for Functional Materials Design, Materials Innovation Factory, University of Liverpool, Liverpool, UK · School of Computer Science, University of Sheffield, Sheffield, UK +1
Temporal Visuo-Tactile Learning for Dexterous Grasp Stability
Oct 7, 2026Ken Nakahara, Aleksei Buvailik, Prokhor Kotov +1Robotic Grasping Tactile Sensing for Robotic Manipulation
LASR Lab, Faculty of Computer Science, TU Dresden, Germany
Unpaired Canonical Correlation Analysis
Oct 7, 2026Nir Ben-Ari, Ronen Talmon, Uri ShahamCanonical Correlation Analysis Multi-View Learning
Department of Computer Science and AI Bar-Ilan University Ramat-Gan, Israel · Electrical and Computer Engineering Technion Haifa, Israel
Mu-DisCoCat: A Variational Pipeline for Compositional Generalization on Quantum Processors
Oct 6, 2026Mina Abbaszadeh, Matilda Karabina Moore, Raem Haq +2Quantum Machine Learning Variational Quantum Circuits
Department of Computer Science University College London London, United Kingdom · Institute for Logic, Language and Computation University of Amsterdam Amsterdam, Netherlands
On the Geometry of Multimodal Saturation: Riemannian VICReg
Oct 5, 2026Nessim Ben Abbes, Duc Han Le, Sabri Mtibaa +1Self-Supervised Learning Geometric Representation Learning
Protectline · Télécom Paris
Multimodal Dual-Encoder Retrieval for Automated ICD Coding
Oct 3, 2026Abhinav Bohra, Anuj BohraAutomated Medical Coding Multimodal Learning
Amazon Seattle, USA · Rutgers University New Jersey, USA
Transcriptome-informed multi-modal AI for predicting neoadjuvant therapy response from breast cancer biopsies
Oct 2, 2026Jungkyu Park, Dhruva Biswas, Joseph Cappadona +22Clinical Outcome Prediction Computational Pathology
Ataraxis AI, New York, NY, USA. · Breast Oncology Program, Dana-Farber Cancer Institute, Boston, MA, USA. · Harvard Medical School, Boston, MA, USA. +13
ZeroMAG: Zero-Shot Multimodal Adapter Generation for Plug-and-Play EEG Foundation Models
Oct 2, 2026Yubo Wang, Jingying Ma, Xinliang Zhou +8Multimodal Learning EEG Foundation Models
Brown University · Stanford University · National University of Singapore +4
FiberGeoText: A Vision-Language Model for Population- Level Organization of Superficial White Matter
Oct 2, 2026Yuqian Chen, R. Jarrett Rushmore, Guikun Chen +5Neuroimaging Medical Image Representation Learning
Department of Radiology, Mass General Brigham, Boston MA, USA · Harvard Medical School, Boston MA, USA · Department of Anatomy and Neurobiology, Boston University Chobanian & Avedisian School of Medicine, Boston MA, USA +2
Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
Oct 1, 2026Damiano Marsili, Raphi Kang, Aditya Mehta +2Mixture of Experts Mixture-of-Experts Models
California Institute of Technology
Surface-volume self-supervised representation learning of brain MRI for genetic discovery
Oct 1, 2026Tian Xia, Nuo Chen, Zihao Zhu +4Self-Supervised Learning Medical Image Representation Learning
D. Bradley McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, US. · Department of Biomedical Informatics and Data Science, Yale School of Medicine, New Haven, US. · Department of Electrical and Computer Engineering, Texas A&M University, College Station, US. +1
SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts
Oct 1, 2026Xiaoli Liu, Yujie Liang, Jialin Li +2Spiking Neural Networks Expert Routing
School of Computer Science and Engineering, University of Electronic Science and Technology of China
Can We Anticipate Violence? Multimodal Learning from Pre-Incident Behavioral Cues
Sep 30, 2026Sindhuja Penchala, Mohammed Yusuf Mujawar, Noorbakhsh Amiri Golilarz +2Human Behavior Prediction Multimodal Learning
Department of Computer Science, The University of Alabama, Tuscaloosa, AL 35487, USA
Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback
Sep 30, 2026Ho-min Park, Byungkon KangMultimodal Grounding Multimodal Embedding
Data Science Center, Texas Children’s Hospital Baylor College of Medicine Houston, Texas 77030, USA · Department of Computer Science SUNY Korea Incheon, Republic of Korea
PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
Sep 29, 2026Cong Wei, Xuanchi Ren, Bryan Chu +8Unified Multimodal Models Multimodal Generation
NVIDIA
What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory
Sep 29, 2026Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene +5Robot Task Progress Estimation Contact-Rich Robotic Manipulation
Computer Science Department, University of Maryland, College Park, MD, USA

Oct 22, 2026J/K move · Enter open · S save

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

Pan Wang, Lipeng Ke, Huajun Ying, Pritish Mohapatra+4

Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA · Amazon, Sunnyvale, USA

Multimodal Sentiment Analysis Multimodal Learning Multimodal Robustness

Open paper AskPDF arXiv

Multimodal sentiment analysis (MSA) aims to predict human sentiments by integrating signals from different modalities such as text, video, and audio. However, raw multimodal sequences often suffer from semantic inconsistency--exhibiting redundancy or conflicts within and across modalities--which hinders robust understanding and increases computational cost. To this end, we introduce ConsMSA, which explicitly formalizes semantic distribution consistency across both

\textit{intra}

- and

\textit{inter}

-modality, providing a consistency-aware mechanism for robust and efficient multimodal sentiment prediction. Specifically, ConsMSA projects multimodal token features into a shared sentiment space to compute an Intra- and Inter-modality Consistency Score (

I^2CS

). By coupling this score with predictive relevance, we formulate consistency-aware importance signals that are utilized: (i) as a consistency regularizer to align latent distributions during training, (ii) to derive semantic-aware weights for adaptive multimodal token reweighting, and (iii) as a practical criterion to prune redundant or conflicting tokens. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that ConsMSA achieves state-of-the-art performance while remaining robust under aggressive token compression--retaining only 10% of tokens yields comparable accuracy. These results establish semantic distribution consistency as a promising foundation for synergizing predictive robustness with computational efficiency.