Multimodal Learning

Ask about this topic

Overview

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

CardsList

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

  1. 3D Classification of Paramagnetic Rim Lesions in Multiple Sclerosis via Asymmetric QSM-FLAIR Modeling

    Jun 15, 2026Veronica Pignedoli, Giacomo Boffa, Nicoletta Noceti +3Medical Image ClassificationBrain MRI

    MaLGa, DIBRIS, University of Genova, Genova, Italy · DINOGMI, University of Genova, Genova, Italy · IRCCS Azienda Ospedaliera Metropolitana, Genova, Italy

  2. SPICE: Synergy and Partial Information Based Curriculum Evolution

    Jun 15, 2026Ankush Pratap Singh, Houwei Cao, Yong LiuCurriculum LearningMultimodal Learning

    New York Institute of Technology · New York, NY, USA · New York University

  3. MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

    Jun 15, 2026Kyeongmin Yeo, Yunhong Min, Minhyuk SungLatent Diffusion ModelsMultimodal Generation

    KAIST

  4. DynFS-MoE: Dynamic Functional-Structural Mixture-of-Experts for Post-Traumatic Epilepsy Diagnosis

    Jun 15, 2026Jun-En Ding, Spencer Chen, Henry Noren +6Medical DiagnosisNeuroimaging

    Department of Industrial and Systems Engineering, Rutgers University, Piscataway, NJ, USA · Department of Systems Engineering, Stevens Institute of Technology, Hoboken, NJ, USA · Department of Neurosurgery, Robert Wood Johnson Medical School, Rutgers University, New Brunswick, NJ, USA

  5. Unsupervised Learning for Missing Modalities in Multimodal Learning

    Jun 14, 2026Hassan Ismkhan, Hamid BouchahciaMissing-Modality LearningUnsupervised Learning

    Bournemouth University Bournemouth, UK

  6. When the Same Musical Knowledge Forgets Differently: A Clean Probe of Pathway-Dependent Forgetting

    Jun 13, 2026Yu Liu, Zhiwei Yang, Wenxiao Zhang +8Music UnderstandingAudio-Language Models

    Institute of Information Engineering, CAS · School of Cyber Security, UCAS · The University of Western Australia +1

  7. Multimodal Ordinal Modeling of Alzheimer's Disease Severity Using Structural MRI and Clinical Data

    Jun 10, 2026Boris-Stephan Rauchmann, Jonathan Laib, Buse Ercik +2Alzheimer's DiseaseMedical Image Classification

    Department of Neuroradiology, LMU University Hospital, Ludwig Maximilian University of Munich, Munich, Germany · Department of Psychiatry and Psychotherapy, LMU University Hospital, Ludwig Maximilian University of Munich, Munich, Germany

  8. Parameter-Efficient Adapter Tuning for Tabular-Image Multimodal Learning

    Jun 10, 2026Jiaqi LuoFine-TuningAdapter Tuning

    School of Mathematical Sciences, Soochow University, No.1 Shizi Street, Suzhou, 215006, Jiangsu Province, China

  9. Information-Theoretic Decomposition for Multimodal Interaction Learning

    Jun 10, 2026Zequn Yang, Yake Wei, Haotian Ni +2Multimodal FusionMultimodal Learning

    Gaoling School of Artificial Intelligence, Renmin University of China, Beijing · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2

  10. RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

    Jun 8, 2026Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang +2HealthcareAudio-Language Models

    The Ohio State University · University of Southern California · University of Chicago

  11. Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning

    Jun 6, 2026Cong Wan, Ying He, Zhongzhan Huang +1Test-Time ScalingMultimodal Reasoning

    Sun Yat-sen University

  12. LongMoE: Longitudinal Multimodal Learning via Trajectory-Aware Mixture-of-Experts

    Jun 6, 2026Maxx Richard Rahman, Prakhar Kumar, Wolfgang MaassMissing-Modality LearningMultimodal Robustness

    German Research Centre for Artificial Intelligence (DFKI) Saarbrücken, Germany

  13. Federated Foundation Models over Vehicular Networks

    Jun 5, 2026Kasra Borazjani, Fardis Nadimi, Payam Abdisarabshali +5Multimodal Federated LearningIntelligent Transportation Systems

  14. GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention

    Jun 4, 2026Giordano Cicchetti, Eleonora Grassucci, Danilo ComminielloCross-AttentionMultimodal Transformer

    Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy

  15. Continual Visual and Verbal Learning Through a Child's Egocentric Input

    Jun 3, 2026Xiaoyang Jiang, Yanlai Yang, Kenneth A. Norman +2Egocentric Video UnderstandingReplay-Based Continual Learning

    Agentic Learning AI Lab, New York University · Department of Psychology, Princeton University

  16. M3imic: Learning a Versatile Whole-Body Controller for Multimodal Motion Mimicking

    Jun 3, 2026Zuxing Lu, Ziang Zheng, Yao Lyu +7RL for RoboticsHumanoid Robot Control

    School of Automation, Southeast University, Nanjing 210096, China · School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China · Department of Robotics, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi 7909, UAE

  17. Differentiable Efficient Operator Search

    Jun 3, 2026Xiaohuan Pei, Jiyuan Zhang, Yuanfan Guo +4Efficient Multimodal InferenceMultimodal Token Compression

    1The University of Sydney · 2ByteDance · 3Shanghai Jiao Tong University +1

  18. GridVQA-X: A Framework for Evaluating Multimodal Explainability Methods

    Jun 2, 2026Sujay Belsare, Sudarshan Nikhil, Sushant Kumar +2Multimodal ReasoningMultimodal Model Interpretability

    1IIIT Hyderabad, India · University of Virginia, USA

  19. A Benchmark for Semi-supervised Multi-modal Crowd Counting

    Jun 2, 2026Haoliang Meng, Xiaopeng Hong, Yabin Wang +1Crowd CountingSemi-Supervised Learning

    Harbin Institute of Technology · Pengcheng Laboratory

  20. Multi-Modal Machine Learning for Breast Cancer Recurrence Prediction

    Jun 1, 2026Jiahao Shao, Xudong Wang, Anam Nawaz Khan +3Breast CancerClinical Outcome Prediction

    Department of Industrial & Systems Engineering, The University of Tennessee, Knoxville, TN 37996, USA · The University of Tennessee Medical Center, Knoxville, TN 37920, USA

  21. Towards Automated Discovery: A Review of Generative Models, Multimodal Learning and Closed-Loop Workflows in Inverse Materials Design

    Jun 1, 2026Anand Babu, Rogério Almeida Gouvêa, Gian-Marco RignaneseCrystal Structure GenerationAutonomous Scientific Discovery

    Institute of Condensed Matter and Nanosciences, Université Catholique de Louvain, Louvain-la-Neuve, Belgium. · WEL Research Institute, Avenue Pasteur 6, Wavre, Belgium.

  22. When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models

    Jun 1, 2026Sarmistha Das, Vaibhav Vishal, Shreyas Guha +3Mixture-of-Experts Language ModelsLow-Resource Language Processing

    Department of Computer Science and Engineering, Indian Institute of Technology Patna, India · School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Thailand

  23. PaCX-MAE: Physiology-Augmented Chest X-Ray Masked Autoencoder

    Jun 1, 2026Yancheng Liu, Kenichi Maeda, Manan PancholyCross-Modal Knowledge DistillationChest X-Ray Classification

    Department of Computer Science, Brown University, Providence, RI, USA.

  24. Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining

    May 31, 2026Guanghao Zhu, Zeyu Liu, Zhitian Hou +10Vision-Language ModelsMultimodal Pretraining

    The Hong Kong Polytechnic University · Sun Yat-sen University · PolyU-Daya Bay Technology and Innovation Research Institute

  25. Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation

    May 31, 2026Zijia Chen, Yuenan Hou, Xinhua Jiang +3Robot Policy LearningMixture of Experts

    College of Electronic Science and Technology, National University of Defense Technology Changsha, 410073, China · Shanghai AI Laboratory Shanghai, 200000, China

  26. Response-Aware Multimodal Learning for Post-Treatment Visual Acuity Forecasting

    May 30, 2026Phuoc-Nguyen Bui, Van-Vi Vo, Duc-Tai Le +5Long-Term Time Series ForecastingClinical Outcome Prediction

    Research Convergence Institute, Sungkyunkwan University, Korea · Dept. of AI Systems Engineering, Sungkyunkwan University, Korea · College of Computing and Informatics, Sungkyunkwan University, Korea +2

  27. V-LynX: Token Interface Alignment for Video+X LLMs

    May 30, 2026Jungin Park, Jiyoung Lee, Kwanghoon SohnCross-Modal AlignmentMultimodal QA

    Yonsei University, Seoul, South Korea · Ewha Womans University, Seoul, South Korea

  28. Your Multimodal Speech Model Says I Have a Face for Radio

    May 28, 2026Maya K. Nachesa, Vlad Niculae, Vagrant GautamASR EvaluationDemographic Bias in ASR

    Language Technology Lab University of Amsterdam · Heidelberg Institute for Theoretical Studies

  29. OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment

    May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Cross-Modal LearningCross-Modal Alignment

    Hong Kong University of Science and Technology, Hong Kong · Vrije Universiteit Amsterdam, Netherlands · UiT – The Arctic University of Norway, Norway +3

  30. MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

    May 28, 2026Abdulkadir Gokce, Badr AlKhamissi, Martin SchrimpfNeuroimagingNeural Encoding

    NeuroAI Lab, EPFL