Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

    Mar 27, 2026Zizhao Chen, Ping Wei, Ziyang Ren +2Multimodal FusionFake News Detection

    State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University

  2. Good Scores, Bad Data: A Metric for Multimodal Coherence

    Mar 26, 2026Vasundra SrinivasanMultimodal Model EvaluationMultimodal Fusion

  3. Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference

    Mar 26, 2026Sk Miraj Ahmed, Xi Yu, Yunqi Li +2Hierarchical Representation LearningMultimodal Robustness

    Computing and Data Sciences, Brookhaven National Laboratory, Upton, NY 11973, USA · Rutgers University, New Brunswick, NJ, USA

  4. CSI-tuples-based 3D Channel Fingerprints Construction Assisted by MultiModal Learning

    Mar 26, 2026Chenjie Xie, Li You, Ruirong Chen +2Wireless CommunicationsRadio Propagation Modeling

    National Mobile Communications Research Laboratory, Southeast University, Nanjing 210096, China · Purple Mountain Laboratories, Nanjing 211111, China · Huawei Technologies Co., Ltd., Shenzhen 518129, China

  5. Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

    Mar 25, 2026Ciem Cornelissen, Sam Leroux, Pieter SimoensMultimodal SSLMultimodal Learning

    IDLab, Department of Information Technology, Ghent University - imec, Belgium

  6. AMIF: Authorizable Medical Image Fusion Model with Built-in Authentication

    Mar 25, 2026Jie Song, Jun Jia, Wei Sun +3Multimodal Fusion

    Macao Polytechnic University · Shanghai Jiao Tong University · East China Normal University +1

  7. DyMRL: Dynamic Multispace Representation Learning for Multimodal Event Forecasting in Knowledge Graph

    Mar 25, 2026Feng Zhao, Kangzheng Liu, Teng Peng +2Multimodal KGsTemporal KGs

    Natural Language Processing and Knowledge Graph Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Centre for Learning, Teaching and Technology, The Education University of Hong Kong, Hong Kong SAR, China

  8. Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval

    Mar 25, 2026Xintao Zong, Xian Zhong, Wenxuan Liu +3Image-Text RetrievalSpiking Neural Networks

    Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology · State Key Laboratory for Multimedia Information Processing, Peking University

  9. AD-Reasoning: Multimodal Guideline-Guided Reasoning for Alzheimer's Disease Diagnosis

    Mar 25, 2026Qiuhui Chen, Yushan Deng, Xuancheng Yao +1Alzheimer's Disease ClassificationMedical Diagnosis

    School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China

  10. HyDRA: Hybrid Domain-Aware Robust Architecture for Heterogeneous Collaborative Perception

    Mar 25, 2026Minwoo Song, Minhee Kang, Heejin AhnLate FusionHeterogeneous Collaborative Perception

    School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea

  11. Foundation Model Embeddings Meet Blended Emotions: A Multimodal Fusion Approach for the BLEMORE Challenge

    Mar 24, 2026Masoumeh Chapariniya, Aref Farhadipour, Sarah Ebling +2Emotion RecognitionMultimodal Emotion Recognition

    Department of Computational Linguistics, University of Zürich, Switzerland

  12. Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition

    Mar 24, 2026Saurabh Kataria, Xiao HuLate FusionSpeech Emotion Recognition

    School of Nursing, Emory University, Atlanta, USA

  13. ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval

    Mar 23, 2026Zhuocheng Zhang, Xingwu Zhang, Kangheng Liang +3Image RetrievalMultimodal Fusion

    Hunan University, Changsha, Hunan, China · University of Glasgow, Glasgow, United Kingdom

  14. Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion

    Mar 23, 2026Seunghan Lee, Jun Seo, Jaehoon Lee +7Cross-Modal LearningTime Series Forecasting

    LG AI Research Seoul, South Korea

  15. Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interaction

    Mar 23, 2026Shiyan Hu, Jianxin Jin, Yang Shu +3Multimodal Anomaly DetectionTime-Series Anomaly Detection

    East China Normal University

  16. SynLeaF: A Dual-Stage Multimodal Fusion Framework for Synthetic Lethality Prediction Across Pan- and Single-Cancer Contexts

    Mar 23, 2026Zheming Xing, Siyuan Zhou, Ruinan Wang +9Multimodal FusionCancer Genomics

    1*School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, Guang Dong 518055, China. · School of Biomedical Sciences, The University of Hong Kong, Hong Kong SAR, China. · 4Departments of Mathematics and Computer Science, Hong Kong Baptist University, Hong Kong SAR, China. +1

  17. Learning Progressive Adaptation for Multi-Modal Tracking

    Mar 22, 2026He Wang, Tianyang Xu, Zhangyong Tang +2Multimodal Visual Object TrackingMultimodal Fusion

    School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China · Centre for Vision, Speech and Signal Processing, University of Surrey, GU2 7XH Guildford, U.K.

  18. Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations

    Mar 22, 2026Tao Meng, Weilun Tang, Yuntao Shou +4Emotion Recognition in ConversationsMultimodal Emotion Recognition

    College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha, China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA

  19. Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition

    Mar 22, 2026Ying Liu, Yuntao Shou, Wei Ai +2Differential AttentionEmotion Recognition in Conversations

    College of Computer and Mathematics, Central South University of Forestry and Technology, 410004, Hunan, Changsha China · Department of Computer Science, State University of New York, New Paltz, New York, 12561, USA

  20. Weakly supervised multimodal segmentation of acoustic borehole images with depth-aware cross-attention

    Mar 21, 2026Jose Luis Lima de Jesus SilvaMultimodal Semantic SegmentationImage Segmentation

    Federal University of Bahia, Institute of Geosciences, Department of Geophysics, Salvador, BA 40170-115, Brazil · Grupo de Estudos e Aplicac¸ ˜ao de Inteligˆencia Artificial em Geof´ısica (GAIA), Federal University of Bahia, Salvador, BA 40170-115, Brazil

  21. Towards Practical Multimodal Hospital Outbreak Detection

    Mar 20, 2026Chang Liu, Jieshi Chen, Alexander J. Sundermann +5HealthcareMultimodal Fusion

    Auton Lab, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA · Microbial Genomic Epidemiology Laboratory, Center for Genomic Epidemiology, University of Pittsburgh, Pittsburgh, Pennsylvania, USA · Division of Infectious Diseases, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania, USA +1

  22. Probing and Inducing Clinical Modality Grounding in Brain Tumor Segmentation

    Mar 20, 2026Lokendra Kumar, Shubham Aggarwal3D Medical Image SegmentationHyper-Connections

  23. Unbiased Dynamic Multimodal Fusion

    Mar 20, 2026Shicai Wei, Kaijie Zhang, Luyi Chen +2Multimodal FusionUncertainty-Aware Multimodal Fusion

    University of Electronic Science and Technology of China

  24. A Multimodal Deep Learning Framework for Edema Classification Using HCT and Clinical Data

    Mar 20, 2026Aram Ansary Ogholbake, Hannah Choi, Spencer Brandenburg +10Medical Image ClassificationMultimodal Classification

  25. Pedestrian Crossing Intent Prediction via Psychological Features and Transformer Fusion

    Mar 20, 2026Sima Ashayer, Hoang H. Nguyen, Yu Liang +1Pedestrian Intention PredictionPedestrian Behavior Modeling

    The University of Tennessee at Chattanooga, Chattanooga, TN, USA

  26. Multi-Modal Building Change Detection for Large-Scale Small Changes: Benchmark and Baseline

    Mar 19, 2026Ye Wang, Wei Lu, Zhihui You +6Remote Sensing Change DetectionRemote Sensing Benchmarks

    MOE Key Lab of ICSP, Anhui Provincial Key Lab of Multimodal Cognitive Computation, IMIS Lab of Anhui Province, School of Computer Science and Technology, Anhui University, Hefei, China · School of Public Safety and Emergency Management, Anhui University of Science and Technology, Hefei 231131, China · College of Computing and Data Science, Nanyang Technological University, Singapore 639798 +3

  27. The Global-Local loop: what is missing in bridging the gap between geospatial data from numerous communities?

    Mar 19, 2026Clément Mallet, Ana-Maria RaimondMultimodal FusionRemote Sensing

    Univ Gustave Eiffel, Géodata Paris, IGN, LASTIG, F-77454 Marne-la-Vallée, France

  28. AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba

    Mar 19, 2026Yan Li, Yifei Xing, Xiangyuan Lan +3Multimodal Sentiment AnalysisMultimodal Fusion

    Pengcheng Laboratory, Shenzhen, China · Shaanxi University of Science & Technology, Xi’an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, China

  29. Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation

    Mar 18, 2026Haocheng Li, Juepeng Zheng, Shuangxi Miao +4Remote Sensing Image SegmentationMultimodal Fusion

    the College of Land Science and Technology, China Agricultural University, and also with the Key Laboratory of Remote Sensing for Agri-Hazards, Ministry of Agriculture and Rural Affairs, Beijing 100083, China · the School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China · Henan Polytechnic University, and also with the Key Laboratory of Spatio-Temporal Information and Ecological Restoration of Mines, Ministry of Natural Resources of the People’s Republic of China, Jiaozuo 454003, China +4

  30. Early Intervention for VFM-based Multimodal Medical Image Classification

    Mar 18, 2026Qijie Wei, Hailan Lin, Xirong LiCross-Modal Representation LearningVision Foundation Model Adaptation

    Renmin University of China, Beijing 100872, China