Multimodal Learning

Follow this topic Ask about this topic

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models
May 15, 2026Yujun Tong, Dongliang Chang, Zijin Yin +3Unified Multimodal Models Multimodal Reasoning
Beijing University of Posts and Telecommunications · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance
A Mutual Information Lower Bound for Multimodal Regression Active Learning
May 14, 2026Leonardo Ferreira Guilhoto, Akshat Kaushal, Paris PerdikarisMixture Density Networks Active Learning
Graduate Group on Applied Mathematics & Computational Science · Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104 · Department of Mechanical Engineering and Applied Mechanics University of Pennsylvania Philadelphia, PA 19104
Compact Latent Manifold Translation: A Parameter-Efficient Foundation Model for Cross-Modal and Cross-Frequency Physiological Signal Synthesis
May 13, 2026Bo Cui, Xiaowen Song, Yaowen Zhang +4Time-Series Foundation Models Multimodal Learning
Department of Biomedical Signals and Systems University of Twente · Department of Applied Mathematics University of Twente
MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidence
May 12, 2026Yifan Chen, Fei Yin, Qingyan Bai +2Multimodal ICL Multimodal Reasoning
University of Cambridge · HKUST · Tsinghua University
SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations
May 12, 2026Souptik Sen, Raneen Younis, Zahra AhmadiMultimodal Learning Representation Learning
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Cross-Modal Learning Multimodal Large Language Models
StepFun-Audio Team
Resilient Vision-Tabular Multimodal Learning under Modality Missingness
May 12, 2026Camillo Maria Caruso, Valerio Guarrasi, Paolo SodaMissing-Modality Learning Multimodal Robustness
Research Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Roma, Italy, Europe · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden, Europe
Checkup2Action: A Multimodal Clinical Check-up Report Dataset for Patient-Oriented Action Card Generation
May 12, 2026Sike Xiang, Shuang Chen, Kevin Qinghong Lin +4Medical Report Generation Healthcare
Durham University Durham, UK · University of Oxford Oxford, UK
BEACON: A Multimodal Dataset for Learning Behavioral Fingerprints from Gameplay Data
May 11, 2026Ishpuneet Singh, Gursmeep Kaur, Uday Pratap Singh Atwal +3Biometric Identification Benchmark Design
Computer Science and Engineering Department Thapar Institute of Engineering and Technology Patiala, India · David R Cheriton School of Computer Science University of Waterloo, Canada
Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs
May 10, 2026Jiafeng Liang, Zhihao Zhu, Zihan Zhang +7Modality Reliance in Multimodal Models Causal Discovery
Harbin Institute of Technology · Harvard University · Peking University +2
FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning
May 10, 2026Xing Han, Shravan Chaudhari, Tanvi Ranade +2Cross-Modal Learning Healthcare
Department of Computer Science Johns Hopkins University · Department of ECE Johns Hopkins University · Department of Computer Science Johns Hopkins University, Bayesian Health
Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learning
May 9, 2026Nikhil J. Dhinagar, Vidhi Chhatbar, Chirag Jagad +6Representation Learning Spectral Methods
Imaging Genetics Center, Mark & Mary Stevens Neuroimaging & Informatics Institute, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA · Neuroscience Graduate Program, Mark & Mary Stevens Neuroimaging & Informatics Institute, Chan Division of Occupational Science & Occupational Therapy, Biomedical Engineering, University of Southern California, Los Angeles, CA, USA
PromptDx: Differentiable Prompt Tuning for Multimodal In-Context Alzheimer's Diagnosis
May 9, 2026Lujia Zhong, Yihao Xia, Shuo Huang +2Multimodal ICL Medical Diagnosis
Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California · Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California · Alfred E. Mann Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California
Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck
May 8, 2026Zihan Huang, Junda Wu, Tong Yu +6LLM Agent Skill Learning Multimodal Agents
UC San Diego · Adobe Research · UC Santa Barbara +1
Efficient Data Selection for Multimodal Models via Incremental Optimization Utility
May 8, 2026Jinhao Jing, Qiannian Zhao, Chao Huang +1Data Valuation Multimodal Learning
McNdroid: A Longitudinal Multimodal Benchmark for Robust Drift Detection in Android Malware
May 7, 2026Md Mahmuduzzaman Kamol, Jesus Lopez, Saeefa Rubaiyet Nowmi +5Concept Drift Android Malware Detection
Department of Computer Science, University of Texas at El Paso · CrowdStrike
Debiased Multimodal Personality Understanding through Dual Causal Intervention
May 7, 2026Yangfu Zhu, Zitong Han, Nianwen Ning +4Causal Representation Learning Algorithmic Fairness
Capital Normal University · Beijing, China · Henan University +2
EchoXFlow: A Beamspace Echocardiography Dataset for Cardiac Motion, Flow, and Function
May 6, 2026Elias Stenhede, Joanna Sulkowska, Eivind Bjørkan Orstad +2Medical Imaging Echocardiography
Medical Technology & E-Health, Akershus University Hospital, Norway · Department of Technology Systems, University of Oslo, Norway · Department of Cardiology, Akershus University Hospital, Norway
Gated Multimodal Learning for Interpretable Property Energy Performance Prediction and Retrofit Scenario Analysis
May 6, 2026Yunfei Bai, Aaron Tesfa Tsion, Raul Rosales +2Multimodal Model Interpretability Multimodal Fusion
Department of Engineering, King’s College London London, UK
Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration
May 5, 2026Xun Jiang, Yufan Gu, Disen Hu +5Multimodal Robustness Modality Imbalance
School of Computer Science and Technology, Tongji University · School of Computer Science and Engineering, University of Electronic Science and Technology of China · Independent Researcher +1
Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Experts
May 5, 2026Hahyeon Choi, Nojun KwakCross-Modal Representation Learning Sparse Mixture-of-Experts
Seoul National University, Seoul, South Korea.
Representation learning from OCT images
May 4, 2026Hedi Tabia, Désiré Sidibé, Nawres Khlifa +6Medical Imaging Medical Imaging Foundation Models
IBISC Univ. Evry Université Paris-Saclay · University of Tunis El Manar · ESIEE Paris - Université Gustave Eiffel +2
VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learning
May 3, 2026Zi-Yi Jia, Zi-Jian Cheng, Xin-Yue Zhang +4Benchmark Design Multimodal Model Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, China
Quantifying Multimodal Capabilities: Formal Generalization Guarantees in Pairwise Metric Learning
May 2, 2026Richeng Zhou, Xuelin Zhang, Liyuan LiuMetric Learning Statistical Learning Theory
LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations
May 1, 2026Huangbiao Xu, Huanqi Wu, Xiao Ke +1Missing-Modality Learning Multimodal Reasoning
Fujian Provincial Key Laboratory of Networking Computing and Intelligent Information Processing, College of Computer and Data Science, Fuzhou University, Fuzhou 350108, China · Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou 350108, China · Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China
Hypergraph and Latent ODE Learning for Multimodal Root Cause Localization in Microservices
May 1, 2026Xin Liu, Yuhang He, Sichen Zhao +2Microservice Architecture Root Cause Analysis
Independent Researcher New York, USA · Independent Researcher Chicago, USA · Northeastern University Boston, USA +2
Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction
Apr 30, 2026Mengfei Zhang, Jinlu Zhang, Zhigang TuHOI Generation Unified Multimodal Models
Wuhan University Wuhan, China · Peking University Beijing, China
GraphPL: Leveraging GNN for Efficient and Robust Modalities Imputation in Patchwork Learning
Apr 28, 2026Xingjian Hu, Zuoyu Yan, Jianhua Zhu +3Missing-Modality Learning Multimodal Learning
Wangxuan Institute of Computer Technology, Peking University, Beijing · Weill Cornell Medicine, Cornell University, New York · Biomedical Informatics, Stony Brook University, New York
Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval
Apr 28, 2026Guosheng Zhang, Linkai Liu, Keyao Wang +3Multimodal Grounding Cross-Modal Alignment
Baidu Inc.

May 15, 2026J/K move · Enter open · S save

Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models

Yujun Tong, Dongliang Chang, Zijin Yin, Xintong Liu+2

Beijing University of Posts and Telecommunications · Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance

Unified Multimodal Models Multimodal Reasoning Multimodal Generation Multimodal Foundation Models Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

The long-standing goal of multimodal AI is to build unified models in which visual understanding and visual generation mutually enhance one another. Despite recent works such as BAGEL, BLIP3o achieves remarkable progress; In practice, however, this unification remains one-directional: understanding routinely guides generation, yet how and why generation can support understanding is rarely investigated. We revisit this asymmetry and propose Generation-to-Understanding (G2U) synergy, where visual generation becomes an explicit intermediate reasoning step. Our framework enables a model to perform controlled generative acts, such as detail enhancement, context expansion or structural visualisation, to produce self-generated visual thoughts, which are then fed back into the model to refine perception without retraining or external tools. Through a comprehensive evaluation on twelve benchmarks, this reversed information flow consistently improves multimodal understanding. We show that generative fidelity bounds perceptual gain and that distinct families of edit prompts govern transfer efficiency. We further analyse whether models can decide what to imagine. While they can produce plausible edits, these self-generated visual thoughts lack stable task alignment, revealing that current large multimodal models fall short of true self-reflection. This work exposes a missing mechanism in unified cognition and suggests that imagination is not the end of understanding but its beginning.