Multimodal Learning

Ask about this topic

Overview

Multimodal learning asks how information from different sources can be combined to improve tasks such as diagnosis, retrieval, forecasting, and robot perception. Common approaches include feature fusion and cross-attention, contrastive alignment, self-supervised pretraining, and mixture-of-experts models that balance shared information with modality-specific cues. Foundation models increasingly support understanding and generation across several modalities, while domain-specific systems use paired or partially paired data and may use extra inputs during training even when they are unavailable at deployment. A central tension is that modalities can complement one another but can also be missing, noisy, misaligned, or unevenly used, so research focuses on robust fusion and evaluations that test whether models truly rely on the relevant evidence.

Latest papers

CardsList

338 papers38 papers in the last four weeks, down 5% on the four weeks before. 0.2% of all new papers.

  1. Monkey King Bang: A Unified Scientific Multimodal Foundation Model

    Jul 17, 2026Hesen Chen, Xinyu Su, Xiaomeng Yang +11AI for ScienceUnified Multimodal Models

    SAIS Team, Shanghai Academy of AI for Science

  2. Action QFormer: Structured Representation Shaping under Action Supervision in Vision-Language-Action Models

    Jul 16, 2026Yufeng Ji, Wenhao Tang, Haoyi Niu +3VLM AdaptationMultimodal Transformer

    Shanghai Qizhi Institute · The Chinese University of Hong Kong · Hong Kong Embodied AI Lab +2

  3. Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

    Jul 14, 2026Minh-Quan Le, Armand Comas, Alexandros Lattas +7Masked Diffusion ModelsMarkov Models

    Google · Stony Brook University · Google DeepMind

  4. ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions

    Jul 13, 2026Maria Teresa Parreira, Micol Spitale, Maia Stiber +5Human-Robot InteractionMultimodal Learning

    Cornell University Ithaca, NY, USA · Politecnico di Milano Milan, Italy · Microsoft Research Redmond, WA, USA +2

  5. Direct Image-to-Modern Vietnamese Translation of Han-Nom Manuscripts via Multimodal RLHF Preference Alignment

    Jul 13, 2026Thi Kim Trang Vo, Nghia Hieu Nguyen, Ha Minh TanDirect Preference OptimizationPreference Optimization

    University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

  6. One-Shot Multimodal Learning from Demonstration with Force-Constrained Elastic Maps

    Jul 10, 2026Brendan Hertel, Jonathan Spanos, Navya Garg +1Robot Skill LearningRobotic Manipulation

    Persistent Autonomy and Robot Learning (PeARL) Lab, University of Massachusetts Lowell, Lowell, MA 01854, USA.

  7. Reinforcing the Generation Order of Multimodal Masked Diffusion Models

    Jul 9, 2026Yidong Ouyang, Zhe Wang, Sourav Bhabesh +1Text-Image AlignmentRL for Diffusion Models

    University of California, Los Angeles · AGI Foundations for AWS

  8. The Importance of Encoder Choice:A Tabular-Image Study

    Jul 8, 2026Ilia Koloiarov, Diego Coello de Portugal Mecke, Vijaya Krishna Yalavarthi +2Representation LearningMultimodal Embedding

    ISMLL & VWFS DARC · University of Hildesheim · Hildesheim, Germany +1

  9. Integrating Neural Encoders in Bayesian Generalized Linear Mixed Models for Multimodal Data

    Jul 6, 2026Yuankang Zhao, Youngsoo Baek, Felipe A. Medeiros +2Bayesian InferenceHierarchical Bayesian Modeling

    Department of Biostatistics & Bioinformatics, Duke University · Department of Ophthalmology, University of Miami

  10. CALM: Interpretable Cross-Modal Alignment for Biomarker Discovery from Unpaired Data

    Jul 2, 2026Jueqi Wang, Zachary Jacokes, John Darrell Van Horn +3Cross-Modal AlignmentScientific Discovery

    Department of Electrical and Computer Engineering, Boston University · School of Data Science, University of Virginia · Department of Psychology, University of Virginia +2

  11. Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities

    Jun 29, 2026Taixi Chen, Nancy GuoMissing-Modality LearningPrompt Learning

    School of Computing State University of New York at Binghamton

  12. Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation

    Jun 29, 2026Chonghuinan Wang, Zhikai Chen, Chunwei Wang +9Unified Multimodal ModelsMultimodal Model Evaluation

    Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China +1

  13. LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion

    Jun 29, 2026Tianyi Zhang, Wei Shan, Yuan Zong +2Multimodal FusionMultimodal Learning

    Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China

  14. Primary ICD Category Prediction using LLM-based Probing

    Jun 27, 2026Chengyuan Liu, Xinyue Zhang, Yao Li +1Automated Medical CodingLinear Probing

    Department of Statistics, Pennsylvania State University, PA, USA · Department of Biostatistics, University of North Carolina at Chapel Hill, NC, USA · Department of Statistics and Operations Research, University of North Carolina at Chapel Hill, NC, USA

  15. Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection

    Jun 26, 2026Olivier Jiyoun Jung, Jonghyeon Park, Myungwoo OhSpeech-Based Dementia DetectionMultimodal Learning

    Division of Communication and Media, Ewha Womans University, South Korea · NAVER Cloud, South Korea

  16. Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards

    Jun 25, 2026Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar +5Unified Multimodal ModelsMultimodal Generation

    1Mohamed bin Zayed University of Artificial Intelligence · 4Linköping University · 3Australian National University +1

  17. OctoSense: Self-Supervised Learning for Multimodal Robot Perception

    Jun 25, 2026Anthony Bisulco, Jeremy Wang, Kostas Daniilidis +2Autonomous Driving DatasetsMasked Autoencoders

    1General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania · Computer Science Department, Brown University

  18. Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

    Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Unified Multimodal ModelsMultimodal Foundation Models

    Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China

  19. Utilizing Cognitive Signals Generated during Human Reading to Enhance Keyphrase Extraction from Microblogs

    Jun 25, 2026Xinyi Yan, Yingyi Zhang, Chengzhi ZhangDocument Information ExtractionMultimodal Learning

    Department of Information Management, Nanjing University of Science and Technology, Nanjing, 210094, China · Department of Archives and E-government, Soochow University, Suzhou, China

  20. Cross-Attention Multimodal Learning for Predicting Response to Neoadjuvant Imatinib in Gastrointestinal Stromal Tumors: A Multicenter Retrospective Study

    Jun 24, 2026Fariba Tohidinezhad, Douwe J. Spaanderman, Natalia Oviedo Acosta +14Clinical PredictionOncology

    Department of Radiology and Nuclear Medicine, Erasmus MC Cancer Institute, University Medical Center Rotterdam, Rotterdam, the Netherlands. · Department of Pathology, Erasmus MC Cancer Institute, University Medical Center Rotterdam, Rotterdam, the Netherlands. · Department of Surgical Oncology, Erasmus MC Cancer Institute, University Medical Center Rotterdam, Rotterdam, the Netherlands. +6

  21. Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition

    Jun 23, 2026Lars Doorenbos, Duc Manh Vu, Serdar Ozsoy +1OOD DetectionMultimodal Learning

    University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

  22. MMGist: A Comprehensive Multimodal Benchmark for 2027

    Jun 21, 2026Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong +3VLM EvaluationLarge Vision-Language Models

    1Shanghai Jiao Tong University · 2Sichuan University

  23. Morphology-Aware Multimodal Representation Learning for Insect Phylogenetic Reconstruction

    Jun 20, 2026Zixuan Liu, Kaijie Yu, Chun He +5Cross-Modal AlignmentPhylogenetic Inference

    Zhejiang Key Lab of Accessible Perception and Intelligent Systems, Zhejiang University, Hangzhou 310027, China · State Key Laboratory of Rice Biology and Breeding & Ministry of Agricultural and Rural Affairs Key Laboratory of Molecular Biology of Crop Pathogens and Insects, Institute of Insect Sciences, Zhejiang University, Hangzhou 310058, China · Rural Development Academy, Zhejiang University, Hangzhou 310058, China +1

  24. Synergistic Dual-Branch Adaptation for Multi-modal Generalized Category Discovery

    Jun 19, 2026Yuxun Qu, Minyu Zhou, Yongqiang Tang +2Cross-Modal AlignmentMultimodal Classification

    State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 101408, China

  25. Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders

    Jun 19, 2026Sergio Lanza, Jae Hee Lee, Stefan WermterVision-Language ModelsVLM Interpretability

    Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany

  26. TriMotion: Modality-Agnostic Camera Control for Video Generation

    Jun 18, 2026Seunghyun Shin, Jifei Song, Wooseok Jeon +2Camera-Controlled Video GenerationMultimodal Learning

    GIST · Huawei Noah’s Ark Lab · Yonsei University +1

  27. MixProLAP: Mixture-Induced Uncertainty Modeling for Probabilistic Language-Audio Pretraining

    Jun 18, 2026Yu Nakagome, Jaesong Lee, Soo-Whan ChungAudio-Text AlignmentMultimodal Pretraining

    LINE WORKS Corporation, Japan · NAVER Cloud Corporation, South Korea

  28. Pointwise is Pointless? A Multimodal Ablation Study for Precipitation Nowcasting with Graph Neural Networks

    Jun 16, 2026Ophélia Miralles, Máté Mile, Christoffer Artturi +2Precipitation ForecastingProbabilistic Forecasting

    Norwegian Meteorological Institute, Oslo, Norway

  29. BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics

    Jun 16, 2026Junfeng Xia, Wenhao Ye, Junxiang Zhang +3Medical Image SynthesisNeuroimaging

    Department of Biomedical Engineering, Southern University of Science and Technology, China · School of Biomedical Engineering, Shenzhen University, China

  30. Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration

    Jun 15, 2026Xiwen Wei, Mark Nutter, Madhusudhanan Srinivasan +1Unified Multimodal ModelsModality Imbalance

    The University of Texas at Austin · Advanced Micro Devices, Inc.