Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

772 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

    Jun 17, 2026Matheus B. Rocha, Gustavo B. Dettogni, Renato A. KrohlingMambaMultimodal Classification

    Labcin - Nature Inspired Computing Lab, Federal University of Espírito Santo, Vitória, Brazil · PPGI - Graduate Program in Computer Science, Federal University of Espírito Santo, Vitória, Brazil

  2. Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology

    Jun 16, 2026Tianyu Liu, Ziqing Wang, Zhaokang Liang +12Pathology Foundation ModelsClinical Prediction

    Program of Computational Biology and Bioinforamtics, Yale University, USA · Broad Institute of MIT and Harvard, USA · These authors contributed equally to this work. +12

  3. Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

    Jun 15, 2026Jingyu Hu, Giuseppe Tripodi, Reed Naidoo +2Computational PathologyMultimodal Robustness

    The Alan Turing Institute, London, United Kingdom · University of Manchester, Manchester, United Kingdom · The Institute of Cancer Research, London, United Kingdom +1

  4. AEF-Econ: Toward Plug-and-Play Socioeconomic Foundation Embeddings from AlphaEarth for Urban Remote Sensing

    Jun 15, 2026Shuyang Hou, Ziqi Liu, Haoyue Jiao +7Geospatial Foundation ModelsRemote Sensing

    State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan, China

  5. Propagating Structural Guidance: Synthesizing Fluorescein Angiography from Fundus Images and Sparse OCT Scans

    Jun 15, 2026Tengfei Ma, Ruiqi Wu, Chenran Zhang +6Medical ImagingCross-Modal Alignment

    School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Nanjing, China · Tianyuan Honors School, Nanjing Medical University, China +2

  6. MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs

    Jun 14, 2026Hangling XieMultimodal ICLMultimodal Large Language Models

    Nanjing University of Posts and Telecommunications, Wenyuan Road No.9, Xixia District, Nanjing City, Jiangsu Province, China

  7. Fusion-E2Pulse: A Multimodal Event-RGB Fusion Network for Non-contact Pulse Wave Reconstruction

    Jun 14, 2026Qian Feng, Hao Guo, Yan Niu +2Remote PhotoplethysmographyEvent-Based Vision

    College of Computer Science and Technology, Taiyuan University of Technology, China · College of Artificial Intelligence, Taiyuan University of Technology, China

  8. Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling

    Jun 13, 2026Zhemin Zhang, Weijie Chen, David Le +7Event Time PredictionMultimodal Robustness

    Arizona State University · Mayo Clinic

  9. Trimodal Glioma Representation Alignment via Volumetric Contrastive Learning

    Jun 12, 2026Denise Marini, Eleonora Grassucci, Danilo ComminielloCancer Survival PredictionCross-Modal Alignment

    Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy

  10. An Attention-based Model for Robust Forecasting with Missing Modality

    Jun 11, 2026Zhitian Zhang, Wenjie Zi, Yunduz Rakhmangulova +3Missing-Modality LearningHuman Trajectory Prediction

    1Simon Fraser University · 2RBC Borealis

  11. ProMUSE: Progressive Multi-modal Uncertainty-guided Staged Evidential Alzheimer Disease Classification

    Jun 11, 2026Long Doan, Branden Chen, Ethan Litton +6Efficient Multimodal InferenceEvidential Deep Learning

    Department of Computer Science, Kennesaw State University · Department of Data Science and Analytics, Kennesaw State University · Department of Information Technology, Kennesaw State University +2

  12. A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition

    Jun 11, 2026Haoran Zhang, Haokun Zhang, Pengyu Liu +3Micro-Gesture RecognitionUnsupervised Domain Adaptation

    School of Computer Science and Information Engineering, Hefei University of Technology (HFUT), Hefei, China · School of Computer Science, University of Auckland(UOA), Auckland, New Zealand

  13. SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs

    Jun 11, 2026Zhengyu Wu, Xu Wang, Hongchao Qin +4Spectral Graph EmbeddingGraph Representation Learning

  14. Latent World Recovery for Multimodal Learning with Missing Modalities

    Jun 10, 2026Hui Wang, Tianyu Ren, Joseph Butler +3Missing-Modality LearningCross-Modal Alignment

    Queen’s University Belfast, Belfast, United Kingdom

  15. From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion

    Jun 10, 2026Yuchen Xian, Yunqiu Xu, Yang He +1Multi-Source Image FusionImage Tokenization

    ReLER, The State Key Lab of Brain Machine Intelligence, Zhejiang University · College of Artificial Intelligence, Zhejiang University · National University of Singapore +2

  16. Bridging the Modality Gap in Forensic Image Retrieval

    Jun 10, 2026Ricardo González-Gazapo, Annette Morales-González, Yoanna Martínez-Díaz +2Multimodal IRDigital Forensics

    Advanced Technologies Application Center (CENATAV), Havana, Cuba · Centro de Sistemas Complejos, Facultad de Física, Universidad de La Habana, Havana, Cuba

  17. Missing-Token Prompted Reliability-Aware Fusion for Robust Polyglot Speaker Identification

    Jun 10, 2026Peng Jia, Li Dai, Jia Li +3Missing-Modality LearningMultimodal Robustness

    Hefei University of Technology Hefei, China · Intelligent Interconnected Systems Laboratory of Anhui Province Hefei University of Technology Hefei, China

  18. Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability

    Jun 10, 2026Kuo-En Hung, Hung-Yue Suen, Shih-Ching Yeh +1Multimodal FusionMultimodal Understanding

    Technology Application and Human Resource Development, National Taiwan Normal University, Taiwan and HRDA.pro, Taiwan · Technology Application and Human Resource Development, National Taiwan University, Taiwan · Computer Science and Information Engineering, National Central University, Taiwan +1

  19. Motion Reinforces Appearance: RGB-Skeleton Gated Residual Fusion for Micro-Gesture Online Recognition

    Jun 10, 2026Jialin Liu, Xinwen He, Pengyu Liu +3Micro-Gesture RecognitionGesture Recognition

    School of Computer Science and Information Engineering, Hefei University of Technology (HFUT), Hefei, China

  20. Information-Theoretic Decomposition for Multimodal Interaction Learning

    Jun 10, 2026Zequn Yang, Yake Wei, Haotian Ni +2Multimodal FusionMultimodal Learning

    Gaoling School of Artificial Intelligence, Renmin University of China, Beijing · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2

  21. Context-Aware Multimodal Claim Verification in Spoken Dialogues

    Jun 9, 2026Chaewan Chun, Delvin Ce Zhang, Dongwon LeeClaim VerificationAudio Understanding

    The Pennsylvania State University, USA · University of Sheffield, UK

  22. Multimodal Brain Tumour Classification Using Feature Fusion

    Jun 9, 2026Wajih ul Islam, Muhammad Yaqoob, Javed Ali Khan +1Multimodal ClassificationMedical Image Classification

    School of Physics, Engineering and Computer Science, University of Hertfordshire, UK

  23. Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

    Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Multimodal UnderstandingMultimodal Grounding

    National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5

  24. Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment

    Jun 9, 2026Jia Li, Qian Chen, Wei Wang +5Personality Modeling in Language ModelsMultimodal Fusion

    Hefei University of Technology Hefei, China · Intelligent Interconnected Systems Laboratory of Anhui Province Hefei University of Technology Hefei, China · Jianghuai Advanced Technology Center Anhui Provincial Industry Innovation Center of Humanoid Robots Anhui Provincial Key Laboratory of Humanoid Robots Hefei, China

  25. Vision-Assisted Foundation Model for Solving Multi-Task Vehicle Routing Problems

    Jun 9, 2026Shuangchun Gui, Zhiguang Cao, Wen Song +1Multi-Task LearningVehicle Routing

    College of Computing and Data Science, Nanyang Technological University, Singapore, and the Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore

  26. A Unified Multi-Modal Framework for Intelligent Financial Systems: Integrating Reinforcement Learning, High-Frequency Trading, and Game-Theoretic Approaches with Cross-Modal Sentiment Analysis

    Jun 9, 2026Fanrong Liu, Zhang Yuwei, Mingni LuoReinforcement LearningQuantitative Finance

    Henan University, International Eurasia College, Kaifeng, Henan, China · City University of Hong Kong, College of Business, Hong Kong, China · Northeastern University, School of Electronic and Information Engineering, Seattle, United States

  27. Fusing Satellite Imagery and Planimetric Maps for Cross-View Localization

    Jun 8, 2026Quang Long Ho Ngo, Zimin Xia, Alexandre AlahiMulti-Source Image FusionCross-View Geo-Localization

    1 ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Switzerland

  28. XMedFusion: A Knowledge-Guided Multimodal Perception and Reasoning Framework for Autonomous Medical Systems

    Jun 8, 2026Hamza Riaz, Arham Haroon, Maha Baig +3Radiology Report GenerationRadiology VLMs

    National University of Sciences and Technology (NUST), Islamabad, Pakistan · University of Staffordshire, Stoke-on-Trent, ST4 2DE, United Kingdom

  29. Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading

    Jun 8, 2026Eder del Blanco, David Gimeno-Gómez, Eva Navas +2Missing-Modality LearningMultimodal Fusion

    Aholab research group within the HiTZ Center at University of the Basque Country (UPV/EHU), Barrio Sarriena s/n, 48940 Leioa, Bizkaia, Spain · PRHLT research center, Universitat Polit`ecnica de Val`encia (UPV), Camino de Vera s/n, 46022, Val`encia, Spain

  30. Interpretable Crisis Behavior Analysis Using Mobility and Social Media Data

    Jun 8, 2026Muhammad Hamza Arshad Majeed, Sidahmed Benabderrahmane, Talal RahwanSocial Media AnalysisAssociation Rule Mining

    New York University (NYUAD), Division of Science, Computer Science Department