Multimodal Learning

Follow this topic Ask about this topic

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

Data-Driven Priors for Uncertainty-Aware Risk Prediction of Clinical Deterioration using Multimodal Data
Mar 9, 2026L. Julián Lechuga López, Tim G. J. Rudner, Farah E. ShamoutBayesian Neural Networks Uncertainty Quantification
Tandon School of Engineering, New York University · New York University Abu Dhabi · University of Toronto
Toward Unified Multimodal Representation Learning for Autonomous Driving
Mar 9, 2026Ximeng Tao, Dimitar Filev, Gaurav PandeyMultimodal Pretraining Multimodal Contrastive Learning
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA
TS-MLLM: A Multi-Modal Large Language Model-based Framework for Industrial Time-Series Big Data Analysis
Mar 8, 2026Haiteng Wang, Yikang Li, Yunfei Zhu +3Multimodal Large Language Models Time Series Forecasting
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · School of Software, Beihang University, Beijing 100191, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China +3
VisualDeltas: Learning Preferences from Visual Quality Perturbations
Mar 7, 2026Hailiang Huang, Yihao Liu, Shengyue Guan +2Preference Learning Multimodal Learning
Peking University, Beijing, China · Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition
Mar 7, 2026Zhexian Huang, Bo Zhao, Hui Ma +5Multimodal Emotion Recognition Multimodal Learning
Great Bay University · The Chinese University of Hong Kong, Shenzhen · City University of Hong Kong (Dongguan) +1
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
Mar 6, 2026Lijiang Li, Zuwei Long, Yunhang Shen +6Masked Diffusion Models Unified Multimodal Models
Nanjing University · Tencent Youtu Lab · CASIA
MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs
Mar 6, 2026Zhi Lei, Chenxi Liu, Hao Miao +3Irregular Time-Series Modeling Multivariate Time Series Forecasting
East China Normal University, Shanghai, China · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China
UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark
Mar 5, 2026Yanlin Li, Minghui Guo, Kaiwen Zhang +13Multimodal Learning Multimodal Generation
NUS · SCUT · NTU +2
Clinical-Injection Transformer with Domain-Adapted MAE for Lupus Nephritis Prognosis Prediction
Mar 4, 2026Yuewen Huang, Zhitao Ye, Guangnan Feng +3Clinical Outcome Prediction Computational Pathology
Sun Yat-sen University · Department of Nephrology, Guangzhou Women and Children’s Medical Center, Guangzhou Medical University, Guangzhou, China
OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments
Mar 2, 2026Hymalai Bello, Lala Ray, Joanna Sorysz +2Human Activity Recognition Video Action Recognition
DFKI Kaiserslautern · RPTU Kaiserslautern-Landau · Korea University
Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
Mar 2, 2026Jiahan Chen, Da Li, Hengran Zhang +6Multimodal Embedding Multimodal Learning
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, University of Chinese Academy of Sciences, Beijing, China · Baidu Inc., Beijing, China
A multimodal slice discovery framework for systematic failure detection and explanation in medical image classification
Feb 27, 2026Yixuan Liu, Kanwal K. Bhatia, Ahmed E. FetitAlgorithmic Auditing Medical Image Classification
Department of Computing, Imperial College London, UK · Aival, London, UK
Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Feb 27, 2026Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1Multimodal Learning Unified Multimodal Models
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
WARM-CAT: Warm-Started Test-Time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning
Feb 26, 2026Xudong Yan, Songhe Feng, Jiaxin Wang +2Zero-Shot Learning Test-Time Adaptation
School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China
Towards Multimodal Domain Generalization with Few Labels
Feb 26, 2026Hongzhao Li, Hao Dong, Hualei Wan +3Multimodal Domain Generalization Multimodal Learning
Zhengzhou University · ETH Zürich · MBZUAI
Plug, Play, and Fortify: A Low-Cost Module for Robust Multimodal Image Understanding Models
Feb 26, 2026Siqi Lu, Wanying Xu, Yongbin Zheng +3Missing-Modality Learning Multimodal Learning
National University of Defense Technology
FewMMBench: A Benchmark for Multimodal Few-Shot Learning
Feb 25, 2026Mustafa Dogan, Ilker Kesen, Iacer Calixto +2Multimodal Model Evaluation Multimodal Large Language Models
Aselsan Research · Department of Computer Science, University of Copenhagen · Department of Medical Informatics, Amsterdam UMC, University of Amsterdam +4
Extending Sequence Length is Not All You Need: Effective Integration of Multimodal Signals for Gene Expression Prediction
Feb 25, 2026Zhao Yang, Yi Duan, Jiwei Zhu +3Genomics Multimodal Learning
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Zhongguancun Academy, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance +1
A Lightweight Vision-Language Fusion Framework for Predicting App Ratings from User Interfaces and Metadata
Feb 24, 2026Azrin Sultana, Firoz AhmedMultimodal Regression Multimodal Fusion
Department of Computer Science, American International University–Bangladesh
CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning
Feb 23, 2026Chunlei Meng, Guanhong Huang, Rong Fu +3Cross-Modal Representation Learning Multimodal Learning
Fudan University · Shantou University · University of Macau +1
Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection
Feb 21, 2026Wanqi Wang, Jingcai Guo, Yuxiang Cai +1Few-Shot Object Detection Object Detection
University of Chinese Academy of Sciences · The Hong Kong Polytechnic University · Zhejiang University +1
Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling
Feb 19, 2026Divyam Madaan, Sumit Chopra, Kyunghyun ChoMissing-Modality Learning Multimodal Learning
New York University. · New York University Grossman School of Medicine · Genentech +1
HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis
Feb 18, 2026J. Dhar, M. K. Pandey, D. Chakladar +4Multimodal Fusion Multimodal Learning
Indian Institute of Technology Ropar, India · RoentGen Health, India · Lulea University of Technology, Sweden +4
Breaking Data Efficiency Dilemma: A Federated and Augmented Learning Framework For Alzheimer's Disease Detection via Speech
Feb 16, 2026Xiao Wei, Bin Wen, Yuqin Lin +5Speech-Based Dementia Detection Data-Efficient Learning
Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · College of Computer and Data Science, Fuzhou University, Fuzhou, China +1
DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset
Jan 17, 2026Yiming Li, Dan Lin, Wenfei Liang +6Multimodal Dataset Human Activity Recognition
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · College of Computer Science and Technology, Harbin Engineering University · Singapore University of Technology and Design (SUTD), Singapore +1
ELF: A Family of Encoder-Free ECG-Language Models
Jan 5, 2026William Han, Tony Chen, Chaojing Duan +6Healthcare Electrocardiography
Carnegie Mellon University, USA · Allegheny Health Network, USA · University of California, Los Angeles, USA +1
From Words to Wavelengths: VLMs for Few-Shot Multispectral Object Detection
Dec 17, 2025Manuel Nkegoum, Minh-Tan Pham, Élisa Fromont +2VLM Adaptation Multispectral Object Detection
Univ Bretagne Sud, IRISA, UMR 6074, Vannes, France · ATERMES, Montigny-le-Bretonneux, France · Univ Rennes, IRISA, UMR 6074, Rennes, France +1
PrismSSL: One Interface, Many Modalities; A Single-Interface Library for Multimodal Self-Supervised Learning
Nov 21, 2025Melika Shirian, Kianoosh Vadaei, Kian Majlessi +3Multimodal SSL Self-Supervised Learning
dept. Computer Engineering University of Isfahan Isfahan, Iran · dept. Computer Engineering University of Texas at Dallas Isfahan, Iran · dept. Artificial Intelligence University of Isfahan Isfahan, Iran
Multimodal Feature Prototype Learning for Interpretable and Discriminative Cancer Survival Prediction
Oct 7, 2025Shuo Jiang, Zhuwen Chen, Liaoman Xu +6Computational Pathology Cancer Survival Prediction
Hangzhou Dianzi University, Hangzhou 310018, China · School of Information and Communication Technology, Griffith University, QLD 4215, Australia · Shenzhen Research Institute of Big Data, Shenzhen 518172, China +3
MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation
Sep 30, 2025Md Zubair, Hao Zheng, Grayson W. Armstrong +5Algorithmic Fairness Multimodal Classification
School of Computing and Informatics, University of Louisiana at Lafayette, Lafayette, LA, USA · Massachusetts Eye and Ear, Harvard Medical School, Boston, MA, USA · Louisiana Center for Health Innovation and College of Nursing & Health Sciences, University of Louisiana at Lafayette, Lafayette, LA, USA +2

Mar 9, 2026J/K move · Enter open · S save

Data-Driven Priors for Uncertainty-Aware Risk Prediction of Clinical Deterioration using Multimodal Data

L. Julián Lechuga López, Tim G. J. Rudner, Farah E. Shamout

Tandon School of Engineering, New York University · New York University Abu Dhabi · University of Toronto

Bayesian Neural Networks Uncertainty Quantification Selective Prediction Mortality Prediction Clinical Risk Prediction Multimodal Learning

Open paper AskPDF arXiv

Safe predictions are a crucial requirement for integrating predictive models into clinical decision support systems. One approach to improving trustworthiness is to enable models to express uncertainty about individual predictions. However, current machine learning models frequently lack reliable uncertainty estimation, hindering real-world deployment. This limitation is particularly evident in multimodal settings, where models must effectively integrate heterogeneous information. In this work, we propose MedCertAIn, a predictive uncertainty framework that leverages multimodal clinical data to improve model performance and reliability for in-hospital mortality risk prediction as an indicator of patient deterioration. We design data-driven priors over neural network parameters using a hybrid strategy that considers cross-modal similarity in self-supervised latent representations and modality-specific data corruptions. We train and evaluate the models with such priors using clinical time-series and chest X-ray images from the publicly available datasets MIMIC-IV and MIMIC-CXR. Our results show that MedCertAIn achieves competitive predictive performance and substantial gains in selective prediction compared with the evaluated deterministic and stochastic baselines. These findings highlight the promise of data-driven priors in advancing robust, uncertainty-aware AI tools for high-stakes clinical applications. Our implementation is publicly available at: https://github.com/jlaitue/medcertain.