Multimodal Learning

Ask about this topic

Overview

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

CardsList

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

  1. Impact of Multimodal and Conversational AI on Learning Outcomes and Experience

    Apr 2, 2026Karan Taneja, Anjali Singh, Ashok K. GoelEducational Dialogue SystemsAI in Education

    Georgia Institute of Technology, Atlanta GA 30332, USA · University of Texas at Austin, Austin TX 78712, USA

  2. BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

    Apr 2, 2026Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef +2Language ModelingMultimodal Learning

    Diabolocom · MICS, CentraleSupélec, Université Paris-Saclay · Artefact Research Center +1

  3. Contrastive Multi-Modal Hypergraph Reasoning for 3D Crowd Mesh Recovery

    Apr 1, 2026Minghao Sun, Chongyang Xu, Yitao Xie +23D Reconstruction3D Human Reconstruction

    Tianjin University · Nanyang Technological University · Sichuan University

  4. MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding

    Apr 1, 2026Junxian Wu, Chenghan Fu, Zhanheng Nie +6Multimodal LearningMultimodal Large Language Models

    Alibaba Group, Hangzhou, China

  5. MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures

    Mar 30, 2026Tim Strohmeyer, Lucas Morin, Gerhard Ingmar Meijer +3Document Image UnderstandingMultimodal Understanding

    IBM Research · ETH Zurich

  6. LongCat-Next: Lexicalizing Modalities as Discrete Tokens

    Mar 29, 2026Meituan LongCat Team, Bin Xiao, Chao Wang +86Unified Multimodal ModelsMultimodal Learning

  7. Generative Score Inference for Multimodal Data

    Mar 27, 2026Xinyu Tian, Xiaotong ShenLLM Uncertainty EstimationMultimodal Learning

    School of Statistics, University of Minnesota

  8. Knowledge is Power: Advancing Few-shot Action Recognition with Multimodal Semantics from MLLMs

    Mar 27, 2026Jiazheng Xing, Chao Xu, Hangjie Yuan +4Few-Shot LearningMultimodal Learning

    College of Control Science and Engineering, Zhejiang University, Hangzhou, 310027, Zhejiang, China. · College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, 310023, Zhejiang, China. · College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, 310027, Zhejiang, China. +1

  9. Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning

    Mar 26, 2026Diyar Altinses, Andreas SchwungMultimodal LearningMultimodal Anomaly Detection

    Department of Automation Technology and Learning Systems, South Westphalia University of Applied Sciences, Soest, Germany

  10. Sparse-by-Design Cross-Modality Prediction: L0-Gated Representations for Reliable and Efficient Learning

    Mar 26, 2026Filippo CenacchiActivation SparsityNeural Network Pruning

    School of Computing, Macquarie University, Sydney, Australia

  11. Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

    Mar 25, 2026Ciem Cornelissen, Sam Leroux, Pieter SimoensMultimodal SSLMultimodal Learning

    IDLab, Department of Information Technology, Ghent University - imec, Belgium

  12. GoCoMA: Hyperbolic Multimodal Representation Fusion for Large Language Model-Generated Code Attribution

    Mar 24, 2026Nitin Choudhury, Bikrant Bikram Pratap Maurya, Bhavinkumar Vinodbhai Kuwar +1Authorship AttributionLanguage Model Fingerprinting

  13. Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deployment

    Mar 23, 2026Zigui Wang, Minghui Sun, Jiang Shu +3Multimodal LearningMultimodal Classification

    Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC, USA · Duke Center for Autism and Brain Development, Duke University School of Medicine, Durham, NC, USA · Department of Psychiatry and Behavioral Sciences, Durham, NC, USA +1

  14. CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models

    Mar 22, 2026Nan Zhou, Huiqun Wang, Yaoyan Zheng +1VLM AdaptationMultimodal Large Language Models

    State Key Laboratory of Complex and Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University

  15. Detached Skip-Links and RR-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR

    Mar 20, 2026Ziye Yuan, Ruchang Yao, Chengxin Zheng +3Multimodal Large Language ModelsMultimodal Learning

    State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China · Tsinghua University, Beijing, China · Baidu Inc, Beijing, China

  16. BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates

    Mar 20, 2026Phuong-Anh Nguyen, Tien Anh Pham, Duc-Trong Le +1Modality ImbalanceMultimodal Learning

    VNU University of Engineering and Technology, Hanoi, Vietnam

  17. Toward High-Fidelity Visual Reconstruction: From EEG-Based Conditioned Generation to Joint-Modal Guided Rebuilding

    Mar 20, 2026Zhijian Gong, Tianren Yao, Wenjia Dong +1Image ReconstructionEEG-Based Visual Decoding

    Beijing University of Technology, Beijing · Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing

  18. Semantic Audio-Visual Navigation in Continuous Environments

    Mar 20, 2026Yichen Zeng, Hebaixu Wang, Meng Liu +4Audio-Visual NavigationMultimodal Transformer

    Wuhan University · Zhongguancun Academy · Shandong Jianzhu University +4

  19. Multimodal Model for Computational Pathology:Representation Learning and Image Compression

    Mar 19, 2026Peihang Wu, Zehong Chen, Lijian XuComputational PathologyWhole-Slide Image Analysis

    Shenzhen University of Advanced Technology

  20. Foundations and Architectures of Artificial Intelligence for Motor Insurance

    Mar 19, 2026Teerapong PanboonyuenMLOpsFinancial Services

  21. On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings

    Mar 18, 2026David Restrepo, Miguel L Martins, Chenwei Wu +5Modality Gap in VLMsMultimodal Learning

    CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA +3

  22. Caption First, VQA Second: Knowledge Density, Not Task Format, Drives Multimodal Scaling

    Mar 17, 2026Hongjian Zou, Yue Ge, Qi Ding +2Multimodal Large Language ModelsMultimodal Learning

    vivo AI Lab, Shenzhen, China · Wuhan University, Wuhan, China

  23. Learning from Limited and Incomplete Data: A Multimodal Framework for Predicting Pathological Response in NSCLC

    Mar 16, 2026Alice Natalina Caragliano, Giulia Farina, Fatih Aksu +10Clinical Outcome PredictionClinical Prediction

    Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy · Fondazione Policlinico Universitario Campus Bio-Medico, Rome, Italy · Department of Medicine and Surgery, Università Campus Bio-Medico di Roma, Rome, Italy +1

  24. Multimodal Deep Learning for Early Prediction of Patient Deterioration in the ICU: Integrating Time-Series EHR Data with Clinical Notes

    Mar 16, 2026Binesh SadanandanClinical Risk PredictionLongitudinal EHR Modeling

  25. Opportunistic Cardiac Health Assessment: Estimating Phenotypes from Localizer MRI through Multi-Modal Representations

    Mar 13, 2026Busra Nur Zeybek, Özgün Turgut, Yundi Zhang +5Left Ventricular Ejection Fraction EstimationCardiac MRI

    Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Germany · Munich Center for Machine Learning (MCML), Germany · Department of Diagnostic and Interventional Neuroradiology, School of Medicine, TUM University Hospital, Germany. +1

  26. GLEAM: A Multimodal Imaging Dataset and HAMM for Glaucoma Classification

    Mar 13, 2026Jiao Wang, Chi Liu, Yiying Zhang +8Glaucoma DetectionOphthalmic Image Analysis

    College of the Information Science and Engineering, Northeastern University, Shenyang 110819, China · Department of Ophthalmology, Shenyang Fourth People’s Hospital, Shenyang 110000, China

  27. TerraFlow: Multimodal, Multitemporal Representation Learning for Earth Observation

    Mar 13, 2026Nazar Puriy, Johannes Jakubik, Benedikt Blumenstiel +1Representation LearningGeospatial Foundation Models

    IBM Research Europe, Zurich, Switzerland · ETH Zurich, Zurich, Switzerland

  28. Learning Pore-scale Multiphase Flow from 4D Velocimetry

    Mar 12, 2026Chunyang Wang, Linqi Zhu, Yuxuan Gu +9Porous Media FlowComputational Fluid Dynamics

    Department of Earth Science and Engineering, Imperial College London. · Department of Geology, Ghent University. · Department of Energy Science and Engineering, Stanford University. +2

  29. FBCIR: Balancing Cross-Modal Focuses in Composed Image Retrieval

    Mar 12, 2026Chenchen Zhao, Jianhuan Zhuo, Muxi Chen +6Modality Reliance in Multimodal ModelsComposed Image Retrieval

    Department of Computer Science and Engineering, The Chinese University of Hong Kong · Tencent AI Data Department · School of Mathematical Sciences, Dalian University of Technology +1

  30. Echo2ECG: Enhancing ECG Representations with Cardiac Morphology from Multi-View Echos

    Mar 9, 2026Michelle Espranita Liman, Özgün Turgut, Alexander Müller +3Electrocardiogram Representation LearningMultimodal Learning

    Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM) and TUM University Hospital, Germany · Department of Cardiology, TUM University Hospital, Germany · Department of Computing, Imperial College London, UK +1