Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning

    Jun 29, 2026Xinxin Chen, Yuchen Li, Zihan Wang +3Efficient Multimodal InferenceMulti-Agent Collaboration

    University of Chinese Academy of Sciences

  2. Cross-Modal Iteration Distillation for Robust IHD Screening: The IDNet Framework and A New Benchmark

    Jun 29, 2026Yongchang Gao, Junjie Pang, Shuaiyu Yang +5Cross-Modal Knowledge DistillationMedical Imaging

    University of Chinese Academy of Sciences, Beijing, China · MGI Tech Co., Ltd., Shenzhen, China · Shenzhen University, Shenzhen, China

  3. Traffic-CBM: A Structurally Interpretable Multimodal Framework for Encrypted Traffic Classification

    Jun 29, 2026Honglei Jin, Wenshuo Chen, Shaofeng Liang +6Encrypted Network Traffic AnalysisMultimodal Classification

    HKUST (GZ) Guangzhou, China · Shandong University Qingdao, China

  4. LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion

    Jun 29, 2026Tianyi Zhang, Wei Shan, Yuan Zong +2Multimodal FusionMultimodal Learning

    Key Laboratory of Child Development and Learning Science (Ministry of Education), School of Biological Sciences and Medical Engineering, Southeast University, Nanjing, China

  5. AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification

    Jun 28, 2026Chuxiao Zuo, Yao Zhu, Minqiang Xu +3Multimodal RobustnessSpeech Processing

    Honor Device Co., Ltd Nanjing, China · Honor Device Co., Ltd Shanghai, China

  6. CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

    Jun 28, 2026Yu Li, Yuenan Hou, Yingmei Wei +2Event-Based VisionEvent-Based Object Detection

    aLaboratory for Big Data and Decision, National University of Defense Technology, Changsha, Hunan, China · bShanghai AI Laboratory, Shanghai, Shanghai, China

  7. A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

    Jun 26, 2026Ahmed Mohamady, Robin Burchard, Kristof Van LaerhovenGated Multimodal FusionHuman Activity Recognition

    University of Siegen, Germany

  8. S2^2-VLA: State-Space Guided Vision-Language-Action Models for Long-Horizon Manipulation

    Jun 26, 2026Zhipeng Xie, Zongyi Han, Xiangyi Wei +3Long-Horizon Robotic ManipulationVision-Language-Action Models

    School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 2State Key Laboratory of Submarine Geoscience, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China

  9. Multi-Modal Conditioned High-Resolution Transformer for Urban Electromagnetic Field Map Prediction Download PDF

    Jun 26, 2026Do-Eon Kim, Dongryul Park, Seungyoung Ahn +3TransformerWireless Communications

    Soongsil University, Seoul 06978, Republic of Korea · Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, Republic of Korea · Department of Intelligent Semiconductors, Soongsil University, Seoul 06978, Republic of Korea

  10. Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI

    Jun 25, 2026Minh Duc Do, Tillmann Rheude, Noel Kronenberg +23D ViTsBrain Age Prediction

    1Berlin Institute of Health at Charité - Universitätsmedizin Berlin · 2Health Data Science Unit, Heidelberg University Hospital and BioQuant · 3Intelligent Medicine Institute, Fudan University +1

  11. Liquid Fusion of Heterogeneous Representations Towards General Salient Object Detection

    Jun 25, 2026Ke Chen, Ling Zhou, Guangqi Jiang +3Salient Object DetectionVisual SSMs

    School of Computer Science and Artificial Intelligence, Changzhou University, Changzhou, Jiangsu, 213159, China · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, 200082, China · Faculty of Data Science, City University of Macau, Avenida Padre Tomás Pereira2026 Taipa, Macao, 999078, China

  12. Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction

    Jun 25, 2026Xilai Li, Xiaosong Li, Haishu Tan +3Multi-Source Image FusionCross-Modal Attention

    Foshan University, Foshan, China · China University of Mining and Technology, Beijing, China · Kunming University of Science and Technology, Kunming, China

  13. When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

    Jun 25, 2026Jaden Moon, Arvind Pillai, Andrew CampbellUncertainty-Aware Multimodal FusionMultimodal Fusion

    Dartmouth College, United States

  14. Methane-Plume Segmentation From Hyperspectral Satellite Imagery Via Multimodal Deep Learning

    Jun 24, 2026Brayan Quintero, Jeferson Acevedo, Samuel Traslaviña +1Efficient Multimodal InferenceRemote Sensing Image Segmentation

    Department of Computer Science, Universidad Industrial de Santander Bucaramanga, 680002, Colombia

  15. SAC2^2-Net: Semantic Anchoring and Complementary-Consensus Fusion for Multimodal Micro-Expression Recognition

    Jun 24, 2026Xuepeng Zheng, Tong Chen, Chaoping Gui +3Cross-Modal AlignmentMultimodal Emotion Recognition

    College of Electronic and Information Engineering, Southwest University, Chongqing, 400715, China · Faculty of psychology, Southwest University, Chongqing, 400715, China · Chongqing Key Laboratory of Generic Technology and System of Service Robots, Southwest University, Chongqing, 400715, China +1

  16. Multi-Stream Temporal Fusion for Financial Fraud Detection

    Jun 23, 2026Mohammadamin Dashti Moghaddam, Nick SciarrilliTransformerFinancial Fraud Detection

    Amazon Web Services

  17. Enhancing Clinician Decision-Making via Uncertainty-Aware Multi-Expert Fusion for Stroke Rehabilitation

    Jun 23, 2026Tamim Ahmed, Thanassis RikakisClinical Decision-MakingMultimodal Model Interpretability

    Bioemdical Engineering, University of Soutehrn California,, Los Angeles, 90007, CA, USA.

  18. Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection

    Jun 23, 2026Tian Qiu, Jifeng Shen, Xin ZuoCross-Modal AttentionCross-Modal Alignment

    School of Electrical and Information Engineering, Jiangsu University, Zhenjiang, 212013, China · School of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang, 212003, China

  19. Beyond Time Series: Spatial Reasoning for Epidemic Forecasting via Multimodal Learning

    Jun 20, 2026Diana Guadalupe Gomez, Chenwei Wu, Zhiyi Wang +2Epidemic ForecastingSpatiotemporal Reasoning

    University of Michigan Ann Arbor, MI, USA

  20. A Controlled Study of CLIP-Based Body-Scene Fusion for Emotion Recognition in Context

    Jun 20, 2026Zubair Abbas, Muhammad Umair, Muqaddas HameedMultimodal FusionEmotion Recognition

    Department of Computer Science, University of Central Punjab, Lahore, Pakistan.

  21. AgroSense 2.0: Cross-Modal Transformer Fusion with Geospatial Raster Integration and Interpretable Multi-Task Learning for Precision Crop Recommendation

    Jun 20, 2026Vishal Pandey, Rishav Tewari, Ruzina Haque LaskarMultimodal FusionPrecision Agriculture

    Research Engineer London, UK · Independent Researcher Kolkata, IN · Scientist-B Centre for Development of Telematics Delhi, IN

  22. Bridging the Age Gap: Towards Detecting Neural Audio Codec Synthesized Elderly Speech Deepfake

    Jun 19, 2026Orchid Chetia Phukan, Girish, Mohd Mujtaba Akhtar +1Audio Deepfake DetectionDeepfake Detection

    BIIC Lab, NTHU, Taiwan

  23. A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage

    Jun 19, 2026Saifelden M. Ismail, Aser O. Ibrahim, Omar A. MahmoudPhishing DetectionMultimodal Fusion

    Department of Communications and Information Engineering Zewail City of Science and Technology, Giza, Egypt

  24. EnTrust: Modeling Inter-Modal Conflict for Trustworthy Multimodal Medical Image Analysis

    Jun 19, 2026Dwarikanath Mahapatra, Abhijit Das, Behzad Bozorgtabar +5Cross-Modality Medical Image SegmentationDiffusion Models for Medical Image Segmentation

    Khalifa University, Abu Dhabi, UAE. · MBZUAI, Abu Dhabi, UAE. · Aarhus University, Denmark. +5

  25. Demographic Metadata as Construct-Irrelevant Noise in DistilBERT-Based Automated Essay Scoring

    Jun 19, 2026Teik Peng Ch'ng, Hui Na ChuaAutomated Essay ScoringDemographic Bias in Language Models

    Sunway University School of Computing and Artificial Intelligence, Faculty of Engineering and Technology, Sunway University, Malaysia

  26. Towards Modality-imbalanced Federated Graph Learning: A Data Synthesis-based Approach

    Jun 18, 2026Zhengyu Wu, Hongchao Qin, Xunkai Li +3Multimodal Federated LearningMissing-Modality Learning

  27. Alzheimer's Disease Diagnosis using a Multimodal Approach with 3D MRI and PET

    Jun 18, 2026Loukas Ilias, Anthi-Maria Vozinaki, Christos Ntanos +1Medical Diagnosis3D Medical Imaging

    DSS Lab, School of ECE, NTUA, 15780 Athens, Greece

  28. Evidential Fusion Network for Multimodal Survival Prediction under Missing Modalities

    Jun 18, 2026Yucheng Xing, Hailan Mo, Zi Wang +2Missing-Modality LearningUncertainty Quantification

    National University of Singapore, Singapore · Imperial College London, London, UK

  29. Vortex: Multi-Modal Fusion System for Intelligent Video Retrieval

    Jun 18, 2026Duc-Tho Nguyen, Hieu-Hoc Tran-Minh, Khanh-Hoa Lam +4Multimodal IRInformation Retrieval

    University of Science, VNU-HCM, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

  30. An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

    Jun 17, 2026Matheus B. Rocha, Gustavo B. Dettogni, Renato A. KrohlingMambaMultimodal Classification

    Labcin - Nature Inspired Computing Lab, Federal University of Espírito Santo, Vitória, Brazil · PPGI - Graduate Program in Computer Science, Federal University of Espírito Santo, Vitória, Brazil