Multimodal Learning

Ask about this topic

Overview

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

CardsList

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

  1. Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

    Apr 27, 2026Zhiheng Liu, Weiming Ren, Xiaoke Huang +12Unified Multimodal ModelsMultimodal Pretraining

    Meta AI · University of Waterloo · The University of Hong Kong

  2. All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation

    Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model EvaluationModality Reliance in Multimodal Models

    National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)

  3. VitaminP: cross-modal learning enables whole-cell segmentation from routine histology

    Apr 26, 2026Yasin Shokrollahi, Karina B. Pinao Gonzales, Elizve N. Barrientos Toro +5Cross-Modal LearningComputational Pathology

    Department of Translational Molecular Pathology, Division of Pathology and Laboratory Medicine, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. · Institute for Data Science in Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA. · The University of Texas MD Anderson Cancer Center, Houston, TX, USA.

  4. Emotion-Conditioned Short-Horizon Human Pose Forecasting with a Lightweight Predictive World Model

    Apr 26, 2026Jingni Huang, Peter BloodsworthGated Multimodal FusionHuman Motion Generation

  5. Inter-Stance: A Dyadic Multimodal Corpus for Conversational Stance Analysis

    Apr 24, 2026Xiang Zhang, Xiaotian Li, Taoyue Wang +9Multimodal Learning

    State University of New York at Binghamton · Choate Rosemary Hall · Rensselaer Polytechnic Institute +2

  6. Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models

    Apr 23, 2026Muhammad Shafique, Abdul Basit, Muhammad Abdullah Hanif +3Model CompressionEfficient Multimodal Inference

    eBRAIN Lab, New York University (NYU) Abu Dhabi, Abu Dhabi, United Arab Emirates

  7. AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce

    Apr 22, 2026Biao Zhang, Lixin Chen, Bin Zhang +3Multimodal Large Language ModelsCross-Modal Retrieval

    Taobao & Tmall Group of Alibaba

  8. From Skeletons to Pixels: Few-Shot Precise Event Spotting via Representation and Prediction Distillation

    Apr 21, 2026Zhong Han Ervin Yeoh, Jiang KanCross-Modal Knowledge DistillationFew-Shot Learning

    Department of Business Analytics School of Computing National University of Singapore · Department of Computer Science School of Computing National University of Singapore

  9. A multimodal and temporal foundation model for virtual patient representations at healthcare system scale

    Apr 20, 2026Andrew Zhang, Tong Ding, Sophia J. Wagner +8Multimodal IRRepresentation Learning

    Department of Pathology, Mass General Brigham, Harvard Medical School, Boston, MA · Cancer Program, Broad Institute of Harvard and MIT, Cambridge, MA · Data Science Program, Dana-Farber Cancer Institute, Boston, MA +5

  10. PestVL-Net: Enabling Multimodal Pest Learning via Fine-grained Vision-Language Interaction

    Apr 19, 2026Xueheng Li, Tao Hu, Ke Cao +5Agricultural Image AnalysisVision-Language Models

    Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences · University of Science and Technology of China · 3Zhongke Hefei Institute of Technology Innovation Engineering

  11. IMA-MoE: An Interpretable Modality-Aware Mixture-of-Experts Framework for Characterizing the Neurobiological Signatures of Binge Eating Disorder

    Apr 18, 2026Lin Zhao, Qiaohui Gao, Elizabeth Martin +5Multimodal ClassificationMixture-of-Experts Models

    Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA · College of Engineering, Northeastern University, Boston, 02115, MA, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, 30602, NY, 10029 +1

  12. DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models

    Apr 18, 2026Biao Wu, Yiwu Zhong, Meng Fang +1Multimodal Large Language ModelsData Selection

    Australian Artificial Intelligence Institute · Peking University · University of Liverpool

  13. Cross-Modal Bayesian Low-Rank Adaptation for Uncertainty-Aware Multimodal Learning

    Apr 17, 2026Habibeh Naderi, Behrouz Haji Soleimani, Stan MatwinLow-Rank AdaptationUncertainty-Aware Multimodal Fusion

    Dalhousie University, Halifax NS, Canada

  14. Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimation

    Apr 17, 2026Habibeh Naderi, Behrouz Haji Soleimani, Stan MatwinLow-Rank AdaptationBayesian LoRA

    Dalhousie University, Halifax NS, Canada

  15. The Acoustic Camouflage Phenomenon: Re-evaluating Speech Features for Financial Risk Prediction

    Apr 16, 2026Dhruvin Dungrani, Disha DungraniSpeech ProcessingMultimodal Learning

    Department of Information Systems, Independent Researchers

  16. EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts

    Apr 14, 2026Runhe Zhou, Shanglin Li, Guanxiang Huang +5Multimodal LearningMultimodal Fusion

    Nanyang Technological University, Singapore · BIFOLD, Berlin Institute for the Foundations of Learning and Data, Berlin, Germany · ATR, Kyoto, Japan +2

  17. Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

    Apr 14, 2026Hao Wang, Jiqing Zhang, Xin Yang +4Camouflaged Object DetectionMultimodal Learning

    Information Science and Technology College, Dalian Maritime University, Liaoning, 116026, China · Dalian University of Technology, Liaoning, 116024, China · Beijing University of Technology, Beijing, 100124, China

  18. TriFit: Trimodal Fusion with Protein Dynamics for Mutation Fitness Prediction

    Apr 13, 2026Seungik ChoProtein Fitness PredictionMolecular Property Prediction

    Department of Physics and Astronomy, Rice University, Texas, USA.

  19. Backbone-Conditional Behavior of Modality Gating in Multi-Modal Prostate MRI Segmentation: A 5-Fold Cross-Validation and Gate Mechanism Analysis

    Apr 12, 2026Yongbo Shu, Wenzhao Xie, Shanhu Yao +4Missing-Modality LearningCross-Modality Medical Image Segmentation

    The Second Xiangya Hospital of Central South University, Changsha, 410011, Hunan, China · School of Life Sciences, Central South University, Changsha, 410013, Hunan, China · Hunan Provincial Key Laboratory of Medical Information Research (Central South University), Changsha, 410011, Hunan, China +2

  20. Towards Multi-Source Domain Generalization for Sleep Staging with Noisy Labels

    Apr 11, 2026Kening Wang, Di Wen, Yufan Chen +6Sleep Stage ClassificationDomain Generalization

    Karlsruhe Institute of Technology · ETH Zurich · Hunan University +1

  21. Exploring Temporal Representation in Neural Processes for Multimodal Action Prediction

    Apr 9, 2026Marco Gabriele Fedozzi, Yukie Nagai, Francesco Rea +1Temporal Representation LearningRobotics

    DIBRIS Department, University of Genoa, Via All’Opera Pia 13 16145 Genoa, Italy · CONTACT Unit, Italian Institute of Technology, Via Enrico Melen 83, 16152 Genoa, Italy · International Research Center for Neurointelligence, The University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan

  22. FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

    Apr 8, 2026Xiangru Jian, Hao Xu, Wei Pang +13Multimodal Model EvaluationMultimodal Learning

    University of Waterloo, Canada · University of Sydney, Australia · Singapore Management University, Singapore +7

  23. PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization

    Apr 7, 2026Shicai Wei, Chunbo Luo, Qiang Zhu +1Multimodal LearningMultimodal Optimization

    Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China. · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Sichuan, China. · Peng Cheng Laboratory, Shenzhen, China.

  24. SVC 2026: the Second Multimodal Deception Detection Challenge and the First Domain Generalized Remote Physiological Measurement Challenge

    Apr 7, 2026Dongliang Zhu, Zhiyi Niu, Bo Zhao +14Multimodal Deception DetectionRemote Photoplethysmography

    Wuhan University · Hefei University of Technology · Great Bay University +4

  25. Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities

    Apr 7, 2026Rongfei Chen, Tingting Zhang, Xiaoyu Shen +1Multimodal Sentiment AnalysisMissing-Modality Learning

    School of Computer Science and Technology, Eastern Institute of Technology, Ningbo, China · School of Mechatronic Engineering and Automation, Shanghai University, China

  26. StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developing

    Apr 6, 2026StarVLA CommunityVision-Language-Action ModelsMultimodal Learning

    Von Neumann Institute, HKUST

  27. Multimodal Structure Learning: Disentangling Shared and Specific Topology via Cross-Modal Graphical Lasso

    Apr 5, 2026Fei Wang, Yutong Zhang, Xiong WangMultimodal LearningMultimodal Graph Learning

    Department of Applied Mathematics and Statistics, Stony Brook University · School of Mathematics, Sichuan University · School of Computer Science and Technology, USTC

  28. EffiMiniVLM: A Compact Dual-Encoder Regression Framework

    Apr 3, 2026Yin-Loon Khor, Yi-Jie Wong, Yan Chai HumMultimodal RegressionEfficient VLM Training

    Universiti Malaya, Kuala Lumpur 50603, Malaysia · Universiti Tunku Abdul Rahman, Selangor 43000, Malaysia

  29. CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation

    Apr 3, 2026Zelin Zhang, Kedi Li, Huiqi Liang +3Multimodal FusionSemantic Segmentation

    The University of Sydney · University of Technology Sydney

  30. AViS-Mamba: Adaptive Visual Steering of Audio State-Space Dynamics for Violence Detection

    Apr 2, 2026Damith Chamalke Senadeera, Dimitrios Kollias, Gregory SlabaughAudio-Visual UnderstandingAudio-Visual Representation Learning

    Queen Mary University of London, UK · Digital Environment Research Institute (DERI), London, UK