Overview

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

CardsList

358 papers21 papers in the last four weeks, down 53% on the four weeks before. 0.1% of all new papers.

  1. Unbundling Multimodal Vision Encoding

    Nov 12, 2026Ellis Brown, Chris Hoang, Shengyi Qian +9Multimodal Foundation ModelsMultimodal Learning

  2. Relaxing Cross-Modal Correspondence for Multimodal Remote Sensing Pretraining

    Nov 10, 2026Xianping Ma, Yu Zhang, Mingzhe Jiang +5Multimodal PretrainingMultimodal Learning

  3. Structured Kernel Surrogate Models for Multimodal Attribution

    Nov 9, 2026Ziniu Zhang, Zhenshuo Zhang, Jianglin Lu +3Multimodal Model InterpretabilitySurrogate Modeling

  4. Operation-aware Multimodal Clinical Decision Support System for ICU Discharge Risk Assessment

    Nov 9, 2026Guoqing Zhang, Rafik Hadfi, Takayuki ItoClinical Decision SupportClinical Outcome Prediction

  5. Layer Roles Matter: Mitigating Modality Imbalance in Multimodal Emotion Reasoning via Adaptive Layer Profiling

    Nov 9, 2026Pengyu Huang, Peipei Song, Beier Zhu +3Modality ImbalanceMultimodal Reasoning

  6. UniMore: Fine-Grained Supervision for Unified Understanding and Generation

    Nov 6, 2026Haoyue Bai, Yinyi Luo, Yicheng He +3Unified Multimodal ModelsMultimodal Learning

  7. Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning

    Nov 5, 2026Liangjian Wen, Jiangtao Zheng, Jiang Duan +8Multimodal SSLMultimodal Learning

  8. InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

    Nov 5, 2026Liangjian Wen, Qun Dai, Jiang Duan +7Multimodal LearningMultimodal Fusion

  9. Multimodal Prompt Learning with Dual-Granularity Contrastive Alignment for Preoperative Prediction of Lung Adenocarcinoma Invasiveness Using 3D CT

    Nov 2, 2026Qing Huang, Xiaotong Tan, Lintong Yao +6Prompt-Based Multimodal LearningMedical Image Classification

  10. Hybrid Federated Learning for Multimodal Fusion under Missing Modalities and Site Heterogeneity

    Nov 2, 2026Xiaofei Wu, Rongmei Liang, Jian Qing ShiMultimodal Federated LearningFederated Learning

  11. HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning

    Nov 2, 2026Alice Driessen, Emil Malmsten, Steffen Wolf +11Multimodal FusionMultimodal Learning

  12. Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?

    Oct 30, 2026Ziyuan Wang, Genliang Zhu, Xiaobo ChenMultimodal Model EvaluationMultimodal Learning

  13. Zero-Shot Multimodal Event Spotting in Soccer Broadcasts

    Oct 27, 2026Sadia Kamal, Mahmudul Hasan, Tony BraskichSports Video AnalysisSoccer Analytics

  14. DocXEval: A Systematic Evaluation of Explainability in Multimodal Document Understanding

    Oct 27, 2026Saifullah Saifullah, David Dembinsky, Stefan Agne +2Multimodal Model InterpretabilityExplainability Evaluation

  15. MM-Net: A Multimodal Multi-Task Network for Sleep Staging and Sleep Apnea Assessment After Ischemic Stroke

    Oct 27, 2026Md Wahiduzzaman Suva, Esm E Moula Chowdhury Abha, Md Imtiaj Alam Sajin +2Sleep Stage ClassificationMultimodal Learning

  16. ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

    Oct 22, 2026Pan Wang, Lipeng Ke, Huajun Ying +5Multimodal Sentiment AnalysisMultimodal Learning

  17. AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning

    Oct 22, 2026Chuan Han Li, Yun Chien ChengMultimodal Emotion RecognitionMultimodal Fusion

  18. TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning

    Oct 15, 2026Liang Nathan Zheng, Jiyuan Liu, Wenhao Liang +3Missing-Modality LearningMultimodal Learning

  19. FlexUMM: Flexible Representation Learning for Unified Multimodal Models

    Oct 15, 2026Yinyi Luo, Haoyue Bai, Yi Ge +3Task InterferenceUnified Multimodal Models

  20. FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

    Oct 15, 2026Jiaming Zhang, Zherui Li, Siqi Guo +3VLM RobustnessAdversarial Attacks on VLMs

  21. TimeCap: A Scalable Open-Source Pipeline for Captioning Wearable Health Time-Series Data

    Oct 15, 2026Max Rosenblattl, Milana Gurbanova, Karl Aurel Deck +16Time Series Language ModelingWearable Health Monitoring

  22. MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data

    Oct 14, 2026Robert Moseley, Md Kaykobad Reza, Ameya Patil +2Multimodal LearningMultimodal Foundation Models

  23. Task-Centric Multimodal Learning for Cognitive Impairment Detection

    Oct 14, 2026Sankalp Vaish, Rishank Singh, Sumukh Veeramalla +3Multimodal LearningSpeech-Based Dementia Detection

  24. CogDrift-4D: Hierarchical Multimodal Modeling of Temporal Behavioral Decay for MCI Detection

    Oct 14, 2026Sankalp Vaish, Sumukh Veeramalla, Rishank Singh +3Dementia CareSpeech-Based Dementia Detection

  25. Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition

    Oct 11, 2026Yunxiang Luo, Constantino Álvarez Casado, Xiaoting Wu +4Multimodal LearningMultimodal Sensor Fusion

  26. Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

    Oct 10, 2026Tillmann Rheude, Roland Eils, Benjamin WildMultimodal Contrastive LearningMultimodal Learning