Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. Beyond Ambiguous Visual Cues: Studying Physiological Disruptions and Cross-Modal Inconsistencies in Deepfake Videos

    Sep 11, 2026Chenxi Yang, Yassine Ouzar, Larbi BoubchirRemote PhotoplethysmographyDeepfake Detection

    LIASD Laboratory Shanghai Jiaotong University Shanghai, China · LIASD Laboratory University of Paris 8 Saint-Denis, France

  2. RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation

    Sep 11, 2026Hongbo Gao, Zhengyu Li, Xueru Nie +4Multimodal RobustnessSalient Object Detection

    State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, No. 92 West Dazhi Street, Nangang District, Harbin, Heilongjiang 150006, China · School of Computer Science, University of Sydney, Camperdown, Sydney, NSW 2006, Australia

  3. RoES: Rotational Equivariant Selective-frequency Fusion for Multimodal Images

    Sep 11, 2026Jiabao Wang, Wenjian Liu, Yaoming Cai +5Multi-Source Image FusionInfrared-Visible Image Fusion

    Zhongnan University of Economics and Law Wuhan, Hubei, China · City University of Macau Macau, China · University of Illinois Chicago Chicago, Illinois, United States +3

  4. Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning

    Sep 10, 2026Andrea Agostini, Simon Böhi, Moritz Vandenhirtz +7Masked AutoencodersHealthcare

    Department of Computer Science, ETH Zurich, Switzerland · Department of Biomedical Engineering, University of Basel, Switzerland

  5. Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework

    Sep 10, 2026Christian Gapp, Elias Tappeiner, Martin Welk +8Multimodal LearningMultimodal Fusion

    Institute of Biomedical Image Analysis UMIT TIROL – Private University for Health Sciences and Health Technology, Eduard-Wallnöfer-Zentrum 1, 6060 Hall in Tirol, Austria · VASCage – Centre on Clinical Stroke Research, 6020 Innsbruck, Austria · Department of Radiology, Medical University of Innsbruck, 6020 Innsbruck, Austria +2

  6. RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection

    Sep 10, 2026Xingyi He, Ziwei Wang, Dongrui WuNeural DecodingEEG Decoding

    School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, China

  7. The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods

    Sep 10, 2026Ioanna Kaffeza, Efthymios Georgiou, Alexandros PotamianosModality ImbalanceMultimodal Optimization

    PERSEE Center, Mines Paris-PSL University, Sophia-Antipolis, France · National Technical University of Athens, Athens, Greece · University of Bern, Bern, Switzerland +2

  8. Bidirectional Multimodal Fusion of Sky Images and Time-Series for Solar Forecasting with Large Language Models

    Sep 10, 2026Ken Chen, Maneesha Perera, Wei Wang +3Multivariate Time Series ForecastingMultimodal Large Language Models

    Department of Mechanical Engineering, The University of Melbourne, Melbourne, Australia · Department of Mechanical Engineering, Sri Lanka Institute of Information Technology, Sri Lanka

  9. Toward Interpretable Multimodal Fusion: Heat Conduction Modeling for Hyperspectral and LiDAR Joint Classification

    Sep 10, 2026Kan Wei, Jiahui Cui, Jing Yao +3Multimodal Model InterpretabilityHyperspectral Image Classification

    Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100190, China · Key Laboratory of Target Cognition and Application Technology, Beijing 100190, China · School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100190, China +3

  10. TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs

    Sep 9, 2026François G. Landry, Moulay A. AkhloufiPedestrian Intention PredictionMultimodal Fusion

    Perception, Robotics and Intelligent Machines Research Group (PRIME) Department of Computer Science, Universit´e de Moncton, Canada

  11. RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion with Matrix-Valued Trust for Multimodal Prediction under Modality Uncertainty

    Sep 9, 2026Yingfan Xu, Tieming Liu, Ye LiangMultimodal RobustnessUncertainty-Aware Multimodal Fusion

    School of Industrial Engineering and Management Oklahoma State University · Department of Statistics, Oklahoma State University

  12. OmniMed-FL: A Robust Multimodal Federated Learning Framework for Clinical Diagnosis

    Sep 9, 2026Ayush Debnath, Ruelia Saha, Sudip MisraMultimodal Federated LearningMedical Diagnosis

    Indian Institute of Technology Kharagpur, India · KTH Royal Institute of Technology, Sweden · SRM University-AP, India +1

  13. A Confidence-Aware Multimodal Fusion Framework for Industrial Human-Robot Collaboration

    Sep 9, 2026Xinyu Liu, Qiqi Dong, Boya Jia +2Intent ClassificationHuman-Robot Collaboration

  14. When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation

    Sep 9, 2026Yuchen Pei, Xiaoyu Hu, Yixiong Zou +5Cross-Modality Medical Image SegmentationMulti-Source Image Fusion

    Central China Normal University Wuhan, China · Huazhong University of Science and Technology Wuhan, China · Guangzhou University of Chinese Medicine Guangzhou, China +2

  15. Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry

    Sep 9, 2026Oriol Marín, Roger Marí, Gloria Haro +1Emotion Recognition in ConversationsMultimodal Emotion Recognition

    Eurecat, Centre Tecnològic de Catalunya, Barcelona, Spain · Universitat Pompeu Fabra, Barcelona, Spain · Serra Húnter Fellow Programme, Universitat Pompeu Fabra, Barcelona, Spain

  16. EEGBind: Detecting Source-Level Interictal Epileptiform Discharges via EEG-Centric Multimodal Binding

    Sep 9, 2026Muchen Li, Anglin Liu, Xuetian Gao +2ElectroencephalographyMultimodal Fusion

    Artificial Intelligence Thrust The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China · Ringgee Smart Technologies Co., Ltd. Fuzhou, Fujian, China

  17. VeriScene: Reconstructing Crime Scenes from Legal Evidence via World-Model Agent

    Sep 8, 2026Kevin Chuanpu Fu, Yongsen Zheng, Zee Kin Yeong +1Physical Consistency in Video GenerationDigital Forensics

    Nanyang Technological University, Singapore · Singapore Academy of Law, Singapore

  18. Explainable Temporal Attention-based Defect Detection For Fillet Joints in Real-Time Gas Metal Arc Welding Based on Multi-modal Data

    Sep 7, 2026Mobina Mobaraki, Mahyar Asadi, Klaske Van Heusden +1ManufacturingIndustrial Inspection

    University of British Columbia- Vancouver campus, 2329 West Mall, Vancouver, V6T 1Z4, British Columbia, Canada · Novarc Technologies Inc., 1225 E Keith Rd, North Vancouver, V7J 1J3, British Columbia, Canada · University of British Columbia, Kelowna campus, 3333 University Way, Kelowna, V1V 1V7, British Columbia, Canada

  19. Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection

    Sep 7, 2026Xuechao Zou, Yi Zhou, Kai Li +4Deepfake DetectionMultimodal Fusion

    Beijing Jiaotong University · Tsinghua University · Ant Group

  20. Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition

    Sep 7, 2026Peng Xie, Amr AlanwarUnified Multimodal ModelsMultimodal Fusion

    TUM School of Computation, Information and Technology, Department of Computer Engineering, Technical University of Munich

  21. RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems

    Sep 7, 2026Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong +3Efficient Multimodal InferenceMultimodal Emotion Recognition

    Faculty of Information Technology University of Science, VNU-HCM Ho Chi Minh City, Vietnam

  22. Fine-Grained Visual Preprocessing and Dual-Stream Temporal Modeling for Multimodal Sentiment Analysis on Social Media

    Sep 7, 2026Su Li, Yigong Zhang, Lei Xiong +1Sentiment AnalysisMultimodal Sentiment Analysis

    School of Information Engineering, Kunming University, No. 2 Puxin Road, Guandu District, Yangpu Campus, Kunming, 650214, Yunnan, China

  23. NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts

    Sep 7, 2026Kai Guo, Chuanbin Liu, Peng Hu +2Multimodal LearningCatastrophic Forgetting

    College of Computer Science, Sichuan University, Chengdu, China · School of Economics and Management, China University of Petroleum (Beijing), Beijing, China · School of Artificial Intelligence, Sichuan University, Chengdu, China +1

  24. LASSNet: Level-Aware Availability-Conditioned Spatial-Semantic Fusion for Brain Tumor Segmentation with Missing MRI Modalities

    Sep 6, 2026Haobin Chen, Ao Chang, Rundong Wang +3Missing-Modality Medical Image SegmentationMedical Image Segmentation

    School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui, 230026, China · Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE) Lab, YRD-RIGHT, Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, 215123, China · College of Information Science and Technology, Donghua University

  25. Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis

    Sep 6, 2026Xiaoran Kou, Jingyi Wu, Peng Sun +2Multimodal Sentiment AnalysisPrompt Tuning

    College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai 200433, China · Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China

  26. MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition

    Sep 5, 2026Pengfei Shao, Jisheng Dang, Jiawen Fang +7Multimodal Sentiment AnalysisMultimodal Emotion Recognition

    School of Information Science and Technology, Guangdong University of Foreign Studies, Guangdong, China · School of Information Science and Engineering, Lanzhou University, Lanzhou, China · School of Computer Science and Engineering, Sun Yat-sen University, Guangdong, China +2

  27. Real-World Multi-Modal and Longitudinal Lung Cancer Dataset

    Sep 4, 2026Rita Cordeiro Mendes, Maria Rita Fonseca Verdelho, Carlos Santiago +1Multimodal Survival AnalysisMultimodal Fusion

    Institute for Systems and Robotics, Instituto Superior Técnico, Portugal

  28. CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation

    Sep 4, 2026Mingyuan Meng, Shuchang Ye, Mingjian Li +3Cross-Modality Medical Image SegmentationMedical Image Segmentation

    Institute of Translational Medicine, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Academy & Zhongguancun Institute of Artificial Intelligence, Beijing, China · School of Computer Science, the University of Sydney, Sydney, NSW, Australia +1

  29. Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

    Sep 4, 2026Xu Lin, Ke Wang, Hui Kang +1Multimodal Emotion RecognitionMultimodal Fusion

    College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China

  30. Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG

    Sep 3, 2026Birgit Nierula, Karam Tomotaki-Dawoud, Mert Akguel +5Multimodal Emotion RecognitionMultimodal Fusion

    Fraunhofer HHI, Berlin, Germany · Humboldt University of Berlin, Berlin, Germany