Multimodal Learning

Ask about this topic

Overview

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

CardsList

335 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

  1. Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning

    May 26, 2025Sanghyuk Chun, Olga RussakovskyCross-Modal AlignmentMultimodal Model Evaluation

    1Princeton University

  2. Explainability in mulimodal deep transformation models for stroke outcome prediction

    Apr 7, 2025Lisa Herzog, Jonas Brändli, Maurice Schneeberger +7Clinical Outcome PredictionExplainable Medical Image Analysis

    University Hospital Zurich, Department of Neurology, Frauenklinikstrasse 26, 8091 Zurich, Switzerland · Zurich University of Applied Sciences, Institute of Data Analysis and Process Design (IDP), Technikumstrasse 9, 8401 Winterthur, Switzerland · University of Zurich, Epidemiology, Biostatistics and Prevention Institute (EBPI), Hirschengraben 84, 8001 Zurich, Switzerland +1

  3. Multimodal AI predicts clinical outcomes of drug combinations from preclinical data

    Mar 4, 2025Yepeng Huang, Xiaorui Su, Varun Ullanat +10Clinical Outcome PredictionMultimodal Learning

    Department of Biomedical Informatics, Harvard Medical School, Boston, MA · Program in Biological and Biomedical Sciences, Harvard Medical School, Boston, MA · Harvard College, Cambridge, MA +8

  4. Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions

    Feb 18, 2025Aditya K Surikuchi, Raquel Fernández, Sandro PezzelleVisual StorytellingMultimodal Learning

    Institute for Logic, Language and Computation (ILLC), University of Amsterdam, Netherlands

  5. Enhancing brain age estimation with structural MRI and synthesized cerebral blood volume maps

    Dec 1, 2024Jordan Jomsky, Zongyu Li, Kay C. Igwe +8Alzheimer's DiseaseBrain Age Prediction

    Department of Biomedical Engineering, Columbia University, New York, NY 10027-7041, USA · Zuckerman Institute, Columbia University, New York, NY 10027-7041, USA · Taub Institute for Research on Alzheimer’s Disease and the Aging Brain, Vagelos College of Physicians and Surgeons, Columbia University, New York, NY, USA +5