Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. AnyTrack: Unifying Visual Object Tracking with Any Modalities

    Aug 7, 2026Hao Li, Yunzhi Zhuge, Wenning Hao +4Multimodal Visual Object TrackingMissing-Modality Learning

    Army Engineering University of PLA Nanjing, China · Dalian University of Technology Dalian, China · National University of Defense Technology Nanjing, China

  2. CFGPNet: Cross-Attention-Based Fused Gradient Programmed Network Framework for Multispectral Object Detection

    Aug 6, 2026Nima Hatami, Karim Faez, Saeed Sharifian +1Multispectral Object DetectionEfficient Object Detection

    Department of Electrical Engineering, Amirkabir University of Technology, 15914, Tehran, Iran

  3. MIDAS: Mutual Information Disentanglement with Uncertainty-Aware Fusion for Incomplete Multimodal Sentiment Analysis

    Aug 6, 2026Yuhua Wen, Yingying Zhou, Qifei Li +4Missing-Modality LearningMultimodal Disentangled Representation Learning

    School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China · Zhongguancun Academy, Beijing 100094, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China +1

  4. URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation

    Aug 6, 2026Guoan Xu, Zhengxue Wang, Yang Xiao +3Multi-Scale Feature FusionMultimodal Fusion

    University of Technology Sydney Sydney, New South Wales, Australia · Nanjing University of Science and Technology Nanjing, China · Honor Device Co., Ltd. Beijing, China +1

  5. F2^2Agent: Financial Fusion of Agentic Intelligence for Multimodal Trading

    Aug 6, 2026Changshuo Liu, Yanzheng Jin, Shangfeng Cai +3Multimodal AgentsMultimodal Fusion

    National University of Singapore · Huazhong University of Science and Technology · Zhejiang University

  6. Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

    Aug 5, 2026Junlin Han, Shengbang Tong, David Fan +4Unified Multimodal ModelsMultimodal Pretraining

    1FAIR · University of Oxford · 2Reality Labs

  7. RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection

    Aug 5, 2026Zian Wang, Hangchuan Liang, Yuehua Chen +4Multi-Source Image FusionInfrared Object Detection

    Jilin University, China · Taiyuan University of Technology, China · Shenzhen University, China

  8. TRNet: Topography-Guided Frequency Rectification and Structure-Aware Decoding for Multimodal Paddy Rice Segmentation

    Aug 4, 2026Kaiwen Xiao, Chunlong Fu, Liping Zheng +1Agricultural Image AnalysisRemote Sensing Image Segmentation

    School of Computer Science, Sichuan University Jinjiang College, Meishan 620860, China

  9. Multimodal Skin Lesion Classification with Swin Transformer and Clinical Metadata Fusion

    Aug 4, 2026Nethmi Pathirana, Isuru Munasinghe, Dileeka AlwisSkin Lesion ClassificationMultimodal Classification

    School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka · Department of Electronic and Telecommunication Engineering, University of Moratuwa, Katubedda, Sri Lanka

  10. Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations

    Aug 4, 2026Chunlei Meng, Jacqueline J. Pang, Pengbin Feng +3Missing-Modality LearningMultimodal Sentiment Analysis

    The College of Intelligent Robotics and Advanced Manufacturing (CIRAM), Fudan University · Cornell University · University of Southern California (USC) +1

  11. Compass: Degradation-Simulated Reciprocal Learning with Lightweight Needle RWKV for Multimodal Crack Segmentation under Missing Modalities

    Aug 4, 2026Hui Liu, Chen Jia, Fan Shi +3Missing-Modality LearningEfficient Multimodal Inference

    Engineering Research Center of Learning-Based Intelligent System (Ministry of Education) Tianjin University of Technology Tianjin, China

  12. Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition

    Aug 4, 2026Suraj Kumar, Mohnish Raj, Soumi Chattopadhayay +2Missing-Modality LearningUncertainty Quantification

  13. Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding

    Aug 4, 2026Mohnish Raj, Suraj Kumar, Soumi Chattopadhayay +2Prototype LearningMultimodal Fusion

    Mohnish Raj, Suraj Kumar, Graduate Student Member, IEEE, Soumi Chattopadhayay, Senior Member, IEEE, Chandranath Adak, Senior Member, IEEE, Ayan Dutta

  14. DRPFNet: Dual-domain Residual Progressive Fusion Network for RGB-Thermal Object Detection

    Aug 4, 2026Zian Wang, Changchun LiMultispectral Object DetectionInfrared Object Detection

    College of Computer Science and Technology, Jilin University, China

  15. CIGTSurv: Clinical Information Guided Tri-modal Survival Prediction with Local Prototype Association and Global Feature Alignment

    Aug 4, 2026Jing Dai, Qibin Zhang, Weiwei Zhou +4Cancer Survival PredictionSurvival Analysis

    Cancer Hospital of Dalian University of Technology, Shenyang, China · School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China · Institute of Pathology, Technical University of Munich, TUM School of Medicine and Health, Munich, Germany +2

  16. DAIF: A Data-Driven Intermediate Fusion Framework for Multimodal Supervised Learning via Approximate Message Passing

    Aug 3, 2026Sagnik Nandy, Samriddha Lahiry, Pragya Sur +1Latent Variable ModelsApproximate Message Passing

    Ohio State University · National University of Singapore · Harvard University

  17. Uncertainty-Aware Crossmodal Fusion for Classification of Animal Behavior

    Aug 3, 2026Ehsan Yaghoubi, Florian HaselbeckUncertainty-Aware Multimodal FusionBioacoustics

    Weihenstephan-Triesdorf University of Applied Sciences, Department of Sustainable Agricultural and Energy Systems, Freising, Am Staudengarten 1, 85354, Germany

  18. Deep Multimodal Fusion Detection through Spatial Mask and Channel Competition

    Aug 3, 2026Guandi Wang, Ming Li, Yunsen Xing +1Cross-Modal LearningCross-Modal Attention

    KTH Royal Institute of Technology Stockholm, Sweden · University of Maryland, College Park, USA

  19. Parameter-Dynamic Adaptive Fusion and Calibration Network for RGBT Tracking

    Aug 3, 2026Zhaoding Ding, Chenglong Li, Jiandong Jin +2Visual Object TrackingMultimodal Fusion

    School of Artificial Intelligence, Anhui University · State Key Laboratory of Opt-Electronic Information Acquisition and Protection Technology, Anhui University · School of Computer Science and Technology, Anhui University

  20. SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

    Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Spiking TransformersSpiking Neural Networks

    School of Software Technology, Zhejiang University, Ningbo, China · NetEase Yidun AI Lab, Hangzhou, China

  21. Beyond Symmetric Fusion: Exploiting Task-Dependent Modality Strengths for RGB-Event Small Object Detection

    Aug 2, 2026Ziheng Wang, Chaolang Li, Yutong Yang +5Small Object DetectionDetection Transformer

    Sun Yat-sen University · Shenzhen University

  22. UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    Aug 2, 2026Ganzhong Luo, Yang Ren, Hanyong Wang +2Multimodal IRMultimodal Reranking

    School of Aeronautics and Astronautics, Sichuan University

  23. Understanding and Overcoming Cross-modal Fusion Bias in Multimodal Anomaly Detection From A Fisher Information Perspective

    Aug 2, 2026Kaifang Long, Lianbo Ma, Liming Liu +1Representational Similarity AnalysisMultimodal Anomaly Detection

    Northeastern University Shenyang, China · CATL Ningde, China

  24. WiFuse: An Attention Mechanism for Human Activity Recognition using Fused CSI Amplitude and Delay-Doppler Channel Features

    Aug 1, 2026Alison M. Fernandes, Hermes I. Del Monego, Bruno S. Chang +3Human Activity RecognitionMultimodal Fusion

    Universidade Tecnol´ogica Federal do Paran´a (UTFPR), Programa de P´os-Graduac¸˜ao em Engenharia El´etrica e Inform´atica Industrial (CPGEI-CT), Av. Sete de Setembro, 3165, Curitiba, 80230-901, Brazil · INESC TEC, Faculdade de Engenharia, Universidade do Porto, Rua Dr. Roberto Frias, Porto, 4200-465, Porto, Portugal

  25. RHEA: Reliability-Harmonized Reconstruction and Assignment for Robust Multimodal-Attributed Graph Clustering

    Aug 1, 2026Yinlin Zhu, Di Wu, Ziyu Han +4Multimodal RobustnessClustering

    Sun Yat-sen University, Guangzhou, China1 · Shandong University, Weihai, China2 · Beijing Institute of Technology, Beijing, China3

  26. Unleashing the Power of Text: Text-Guided Flow Matching for Image Fusion under Complex Degradations

    Aug 1, 2026Axi Niu, Jieheng Li, Kang Zhang +3Flow MatchingMulti-Source Image Fusion

    School of Computer Science, Northwestern Polytechnical University, Xi'an, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, Shaanxi, China · School of Electrical Engineering, KAIST, Daejeon, Republic of Korea +2

  27. Structured Proxy Features for Multimodal NSCLC Survival Prediction from Pretreatment CT

    Aug 1, 2026Huu Phong Nguyen, Delower Hossain, Ehsan Saghapour +2Cancer Survival PredictionOncology

    Department of Biomedical Informatics and Data Science, School of Medicine, The University of Alabama at Birmingham (UAB), Birmingham, AL, United States. · Department of Computer Science, The University of Alabama at Birmingham (UAB), Birmingham, AL, United States. · System Pharmacology and AI Research Center (SPARC), The University of Alabama at Birmingham (UAB), Birmingham, AL, United States.

  28. Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation

    Jul 31, 2026Weixiang Zhou, Xingguo Xu, Yuhao Wang +4Cross-Modal Representation LearningMultimodal Robustness

    School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China · Department of Radiology and Biomedical Imaging, University of California, San Francisco, 94107, USA +1

  29. DoubleHelix: Structured Cross-Modal Fusion for Audio-Visual Speech Recognition with LLMs

    Jul 31, 2026Ziwei Cheng, Zhenhua Tan, Zhuomin ZhuMultimodal RobustnessAudio-Visual Speech Recognition

    Software College of Northeastern University · Shenyang, Liaoning, China

  30. MIND: Multimodal Intent-Driven Network via Diffusion Transformers for Medical Image Fusion

    Jul 30, 2026Yunzhan Fu, Xiangyu Shen, Yifei Sun +3Multi-Source Image FusionDiffusion Transformer

    Transvascular Implantation Devices Research Institute, Zhejiang University, Hangzhou, China · Hangzhou Institute of Technology, Xidian University, Hangzhou, China · Zhejiang University, Hangzhou, China +3