Multimodal Learning

Ask about this topic

Overview

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

CardsList

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

  1. CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery

    Aug 24, 2026Mahir Shahriar Tamim, Sharjil Khan, Md. Samiul Alim +3Multimodal UnderstandingCross-Modal Learning

    Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh

  2. Multimodal Prompt Learning with Irregular EHRs for Robust Monitoring of Critical Care Patients

    Aug 22, 2026Yixin Yang, Yueyang Sun, Weichen Liu +2Missing-Modality LearningMultimodal Learning

    Faculty of Engineering, Shenzhen MSU-BIT University · International School, Beijing University of Posts and Telecommunications · School of Computer Science and Engineering, Southeast University +1

  3. Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combination

    Aug 20, 2026Kaixin Xu, NaiJin Liu, Yulin Kang +8Multimodal Sentiment AnalysisMultimodal Learning

    School of Software Technology, Zhejiang University · Beijing WuZi University · Ant Group +3

  4. MUST-PET: MUltimodal Self-supervised learning across Tracers for whole-body PET/CT-based lesion segmentation

    Aug 20, 2026Bashirul Azam Biswas, Amartya Bhattacharya, Biratal Raj Wagle +3Multimodal SSLPET/CT Tumor Segmentation

    Department of Biomedical Data Science Geisel School of Medicine at Dartmouth Hanover, NH 03755, USA · Radiology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA · Radiation Oncology Dartmouth Hitchcock Medical Center Lebanon, NH 03766, USA

  5. Neoadjuvant chemotherapy response prediction using pretreatment diffusion and contrast-enhanced magnetic resonance imaging with clinical variables

    Aug 18, 2026Pablo García Marcos, Paula Puerta González, Guillermo Lorenzo +6Cancer Drug Response PredictionClinical Outcome Prediction

    Computer Science Department, University of Oviedo, Asturias, Spain · Biomedical Engineering Center (BME), University of Oviedo, Asturias, Spain · Electrical Engineering Department, University of Oviedo, Asturias, Spain +7

  6. CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications

    Aug 16, 2026Yubo Zhang, Yiyao LiuCross-Modal Representation LearningSelf-Supervised Pre-Training

  7. Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

    Aug 15, 2026Yihang Du, Juhao Liang, Zhengzhao Lai +2Multimodal Large Language ModelsVision-Language Alignment

    The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · National Health Data Institute (Shenzhen)

  8. PathFinder: Joint Decompositions of Linked Multimodal Datasets

    Aug 15, 2026Ying-Qiu Zheng, Alex Fung, Stephen M Smith +2Low-Rank Matrix DecompositionMatrix Factorization

    Oxford Centre for Integrative Neuroimaging University of Oxford Oxford, UK

  9. MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneity

    Aug 14, 2026Adiba Orzikulova, Dong Min Kim, Jaehong Yoon +1Multimodal Federated LearningFederated Learning

    KAIST · NTU Singapore

  10. Hyperbolic Multimodal Continual Learning

    Aug 10, 2026Jiahong Liu, Ming Shen, Xiaohao Liu +4Cross-Modal Representation LearningInvariant Representation Learning

    Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · School of Computing, National University of Singapore, Singapore · Department of Computer Science, Yale University, New Haven, CT, United States +1

  11. Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation

    Aug 10, 2026Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham +1Multimodal EmbeddingAdapter Tuning

    Eximius Labs · Wabash College · Skop Intelligence Co.

  12. Conformal Calibration for Multi-Modal Regression with Missing Modalities

    Aug 7, 2026Ilia AziziGroup-Conditional Conformal PredictionMultimodal Regression

    Department of Operations, HEC Lausanne, University of Lausanne, Switzerland · BegooAI, Switzerland

  13. DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding

    Aug 7, 2026Hanshu Yao, Janfeng Zhong, Niu Lian +1Multimodal MemoryDocument QA

    1Harbin Institute of Technology, Shenzhen · 2Tsinghua Shenzhen International Graduate School, Tsinghua University

  14. Walkable to Whom? Capturing Subjective Variability in Walkability Perception Using Multimodal Deep Learning

    Aug 7, 2026Moloud Damandeh, Meead SaberiUser ModelingUrban Planning

    School of Civil and Environmental Engineering, University of New South Wales (UNSW), Sydney, NSW, Australia, and the Research Centre for Integrated Transport Innovation (rCITI)

  15. F2^2Agent: Financial Fusion of Agentic Intelligence for Multimodal Trading

    Aug 6, 2026Changshuo Liu, Yanzheng Jin, Shangfeng Cai +3Multimodal AgentsMultimodal Fusion

    National University of Singapore · Huazhong University of Science and Technology · Zhejiang University

  16. OPD-V: Visual On-Policy Self-Distillation with Modality Balance

    Aug 5, 2026Aniri, Jinhe Bi, Peng Liao +5Modality ImbalanceOn-Policy Self-Distillation

    2Ludwig Maximilian University of Munich · 3Munich Center for Machine Learning · 1National University of Singapore +1

  17. iStructTab: Structured Feature Sequencing for Multimodal Learning of Image and Tabular Data

    Aug 5, 2026Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Gyawali +2Sequence ModelingMultimodal Learning

    Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26506, USA · Scientific Computing and Imaging Institute & Department of Biomedical Informatics, The University of Utah, Salt Lake City, UT 84112, USA

  18. DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing

    Aug 3, 2026Sagnik Nandy, Samriddha Lahiry, Pragya Sur +1Latent Variable ModelsApproximate Message Passing

    Ohio State University · National University of Singapore · Harvard University

  19. SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

    Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Spiking TransformersSpiking Neural Networks

    School of Software Technology, Zhejiang University, Ningbo, China · NetEase Yidun AI Lab, Hangzhou, China

  20. CALM-AH: An ABAW11-Calibrated Multimodal Ensemble with Reliability-Gated Multi-Expert Consensus for Video-Level Ambivalence and Hesitancy Recognition

    Jul 31, 2026Wenzhuo Sun, Mingjian Liang, Richard Attfield +3Ensemble LearningMultimodal Emotion Recognition

    Monash University, Melbourne, Australia · Monash Suzhou Research Institute, Suzhou, China

  21. What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations

    Jul 29, 2026Kaizhen Tan, Xin Xu, Siru Tao +4Parameter IdentifiabilityPredictive Representation Learning

    New York University, New York, NY, USA · Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA

  22. Regularizing modality contribution drift in multimodal continual learning

    Jul 29, 2026Zhen Zhang, Jielei Chu, Bin Liu +1Multimodal Learning

    School of Computing and Artificial Intelligence, Southwest Jiaotong University

  23. From Cellular Responses to Pharmacological Domains: Multimodal Zero-Shot Drug Representation Learning

    Jul 28, 2026Jintao Huang, Lu Leng, Ziyuan YangCross-Modal RetrievalMolecular Property Prediction

    Nanchang Hangkong University, Nanchang, Jiangxi, China · Sichuan University, Chengdu, Sichuan, China

  24. Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification

    Jul 27, 2026Francisco Mena, Dino Ienco, Roberto Interdonato +2Missing-Modality LearningMultimodal Robustness

    GFZ Helmholtz Center for Geosciences, Potsdam, Germany · INRAE, UMR TETIS, University of Montpellier, Montpellier, France · INRIA, EVERGREEN, University of Montpellier, Montpellier, France +1

  25. Twins: Learn to Predict Unified Representations with Focal Loss

    Jul 24, 2026Kaixiong Gong, Xin Cai, Bin Lin +9Focal LossDiffusion Transformer

    The Chinese University of Hong Kong · Tencent, Hunyuan · City University of Hong Kong +2

  26. Generative and multimodal AI for materials prediction and design: Progress, challenges, and perspectives

    Jul 22, 2026Xianyuan Liu, Charles Anjah, Benjamin E. Jolly +9Materials Property PredictionMaterials Science

    School of Computer Science, University of Sheffield, Sheffield, UK · Centre for Machine Intelligence, University of Sheffield, Sheffield, UK · School of Chemical, Materials and Biological Engineering, University of Sheffield, Sheffield, UK +3

  27. Open Your Model's Eyes: Video and Context-Aware Multimodal Backchannel Prediction

    Jul 22, 2026Min-Jae Kim, Jun-Yeong Moon, Mujeen Sung +1Cross-Modal AlignmentMultimodal Learning

    1Korea University, Seoul, Republic of Korea · 2Kyung Hee University, Yongin, Republic of Korea

  28. A Unified Tokenization Framework for Pain Recognition using Heterogeneous 3D Modalities

    Jul 22, 2026Stefanos Gkikas, Christian Arzate Cruz, Valentina Becchetti +3Multimodal Learning

    Honda Research Institute Japan Wako City, Japan · Sapienza University of Rome Rome, Italy · University of Canberra Canberra, Australia

  29. Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

    Jul 20, 2026Tillmann Rheude, Roland Eils, Benjamin WildGeometric Representation LearningMultimodal Contrastive Learning

    Berlin Institute of Health, Charité - Universitätsmedizin Berlin · Department of Mathematics and Computer Science, Freie Universität Berlin · Intelligent Medicine Institute, Fudan University

  30. MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

    Jul 18, 2026Sana Tonekaboni, Viktoria Schuster, Caroline UhlerCross-Modal Representation LearningMultimodal Disentangled Representation Learning

    Massachusetts Institute of Technology, Cambridge, MA, USA · Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cambridge, MA, USA · Vector Institute, Toronto, Canada +1