Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. VCP-DCN: Beyond Visual Concealed Property via Depth Collaborative Network for Camouflaged Object Detection

    Jul 30, 2026Songsong Duan, Xi Yang, Nannan WangCross-Modal AlignmentMultimodal Fusion

    the State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China

  2. Semantic-Aligned Structural Abstraction for Multimodal Sentiment Analysis

    Jul 30, 2026Wei Chen, Junkai Li, Tongguan Wang +4Multimodal Large Language ModelsMultimodal Classification

    Huazhong Agricultural University Wuhan, China · Hubei University Wuhan, China · Engineering Research Center of Intelligent Technology for Agriculture, Hubei Engineering Technology Research Center of Agricultural Big Data, Key Laboratory of Smart Farming for Agricultural Animals.

  3. DS@GT ARC at MEDIQA-CORE-Task-1 2026: Trimodal Model Fusion with Task-Specific Gates for Brain Tumor Subtype Classification

    Jul 30, 2026Hoang Thanh Thanh Truong, Charles R. ClarkMultimodal ClassificationBrain Tumor Classification

    Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332 · University of Florida, Stadium Rd, Gainesville, FL 32611

  4. TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification

    Jul 29, 2026Yu Chang, Anzhe Cheng, Chenwei Wu +7Sparse Mixture-of-ExpertsMultimodal Classification

    University of Southern California · University of Michigan · 3The University of British Columbia +1

  5. Multimodal fusion of visual and morphometric features for avian bone classification

    Jul 29, 2026Nevio Dubbini, Lisa Yeomans, Marco Pavia +4Multimodal ClassificationMultimodal Fusion

    University of Pisa, Italy · University of Copenhagen, Denmark · University of Turin, Italy +1

  6. Understanding Knowledge Transfer Mechanism in Heterogeneous MLLM Fusion: A Simple Linear Approach

    Jul 29, 2026Yinghao Hou, Jiahe Fan, Yuanhao Pu +2Multimodal Large Language ModelsMultimodal Reasoning

    University of Science and Technology of China

  7. Where Detectors Fail: Closing the Tail-Domain Gap with Expert-Guided Mutual Distillation

    Jul 29, 2026Xuan Feng, Guihong Liu, Tianlong Gu +5Fake News DetectionDomain Generalization

    1Jinan University · 2Guilin University of Electronic Technology · 3Nanyang Technological University +1

  8. PhysAgent: A Multi-Agent Framework for Reliable Remote Heart Rate Estimation

    Jul 29, 2026Yehui Yang, Bo Zhao, Junzhe Cao +4Remote PhotoplethysmographyHeart Rate Estimation

    1Great Bay University · 2Shenzhen University of Advanced Technology · 3Macao Polytechnic University +1

  9. Registration-Grounded Spectral Fusion for Unregistered WLI/NBI Endoscopic Lesion Segmentation

    Jul 29, 2026Pengyu Jie, Wanquan Liu, Rui He +5Cross-Modality Medical Image SegmentationMulti-Source Image Fusion

    School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, 518107, Guangdong, P.R. China · Department of Otolaryngology, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, 510000, P.R. China · School of Computer Science and Technology, Hainan University, Haikou, 570228, China +2

  10. DynaBridge: Dynamic Summary-Guided Cross-Task Multimodal Fusion for DASS-Structured Mental Health Assessment

    Jul 28, 2026Shiyu Teng, Haichen Yu, Jiaqing Liu +6Medical DiagnosisDepression Severity Estimation

    College of Information Science and Engineering Ritsumeikan University Ibaraki, Osaka, Japan · College of Computer Science and Technology Zhejiang University Hangzhou, Zhejiang, China

  11. Less is More: Modality-Decoupling for General AIGC Audio-Video Detection

    Jul 28, 2026Jielun Peng, Yabin Wang, Yaqi Li +3Audio Deepfake DetectionMultimodal Fusion

    Harbin Institute of Technology · University of Agder

  12. Balanced Soft mixture-of-expert model for Glaucoma Detection

    Jul 28, 2026Sai Venkatesh Chilukoti, Krishna Rauniyar, Min Shi +1Multimodal ClassificationMixture-of-Experts Models

    School of Computing and Informatics, University of Louisiana at Lafayette, 301 East Lewis Street, Lafayette, 70504, LA, USA.

  13. ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

    Jul 27, 2026Hangjie Yuan, Yichen Qian, Zhiwei Tang +21Medical Report GenerationMedical VQA

    1DAMO Academy, Alibaba Group, Hangzhou, China · 3Hupan Laboratory, Hangzhou, China · 2DAMO Academy, Alibaba Group, Beijing, China +7

  14. Spatio-Temporal Conditional Denoising Transformer for Modality-Missing RGBT Tracking

    Jul 27, 2026Andong Lu, Ziyi Zha, Jiandong Jin +4Missing-Modality LearningMultimodal Robustness

    School of Computer Science and Technology, Anhui University, China · School of Artificial Intelligence, Anhui University, China

  15. Mutual Modality Trust with Lightweight Reconstruction Regularization for Fine-grained Tire Pattern Recognition

    Jul 27, 2026Jianning Yang, Jie Fang, Xinda Ma +3Fine-Grained Image ClassificationMultimodal Fusion

    School of Communications and Information Engineering, Xi’an University of Posts and Telecommunications, Xi’an 710121, China · School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China · School of Artificial Intelligence, Hainan Normal University, Haikou 571158, China

  16. Color Fundus Photography Analysis: Co-evolution of Data, Preprocessing, and Modeling toward Multimodal AI

    Jul 27, 2026Yu Li, Wengan He, Wenhui Xu +7Medical Imaging Foundation ModelsMedical Image Analysis

    aZhuhai College of Science and Technology, Zhuhai, Guangdong, China · bZhuhai UNO technology Co., Ltd., Zhuhai, Guangdong, China

  17. DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering

    Jul 27, 2026Yue Zhang, Xiangyu Li, Wanshu Fan +2Visual Question AnsweringVision Transformer

    National and Local Joint Engineering Laboratory of Computer Aided Design, School of Software Engineering, Dalian University, Dalian, China · School of Computer Science and Technology, Dalian University of Technology, Dalian, China

  18. Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

    Jul 26, 2026Anh Ngo, Nicolas Rollet, Catherine Pelachaud +1Human-AI InteractionMultimodal Classification

    ALMAnaCH, INRIA Paris ISIR, Sorbonne University Paris, France · ALMAnaCH, INRIA Paris Télécom Paris, SES, Institut Polytechnique de Paris, I3-CNRS Paris, France · CNRS, ISIR, Sorbonne University Paris, France +1

  19. ConFusion: Continuous Fusion Space Learning for Fine-Grained Controllable Infrared and Visible Image Fusion

    Jul 26, 2026Guo Yurong, He Yufei, Li Yonghao +3Multi-Source Image FusionInfrared-Visible Image Fusion

    North China Electric Power University Baoding, Hebei, China · Beijing University of Posts and Telecommunications Beijing, China

  20. Token-Region Guided Cross-Attention Fusion for Multimodal Affect Interpretation

    Jul 26, 2026Musa Tur Farazi, Nufayer Jahan RezaSocial Media AnalysisMultimodal Classification

  21. Multi-Modal Object Re-Identification with Prompt-S6 and Semantic-Aware Knowledge Guidance

    Jul 26, 2026Weixiang Zhou, Jiabei Zuo, Yuhao Wang +3Cross-Modal AlignmentMultimodal Fusion

    School of Mathematical Sciences and the Key Laboratory for Computational Mathematics and Data Intelligence of Liaoning Province, Dalian University of Technology, Dalian, 116024, China · School of Computer Science and Technology, Dalian University of Technology, Dalian, 116024, China · School of Future Technology, School of Artificial Intelligence, Dalian University of Technology, Dalian, 116024, China +1

  22. Bitcoin Price Direction Prediction via Regime-Aware Multi-Modal Fusion of Social Sentiment and Technical Features

    Jul 25, 2026Muhammad Abdullah HaroonQuantitative FinanceTime Series Classification

    Department of Computer Science, FAST National University of Computer and Emerging Sciences (FAST-NUCES), Lahore, Pakistan

  23. Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating

    Jul 25, 2026Yuriko Kikuchi, Takato Hayashi, Ryusei Kimura +3Late FusionSpeech Processing

    Japan Advanced Institute of Science and Technology · Nomi, Ishikawa, Japan · NTT, Inc. +1

  24. Autoregressive EHR Foundation Models with Multimodal Inputs

    Jul 24, 2026Yuxuan Liu, Joshua Placidi, Jinpei Han +3Clinical Outcome PredictionMultimodal Token Compression

    Dept. of Computing, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom · School of Public Health, Imperial College London, London, United Kingdom +2

  25. Multimodal Surface EMG Hand Gesture Recognition Using Query-Based Transformers for Prosthetic Control

    Jul 24, 2026Federico Del Pup, Elisa Tentori, Manfredo AtzoriEMG-Based Gesture RecognitionHybrid CNN-Transformer Architectures

    Department of Information Engineering, University of Padua, Padua, 35131, Italy · Padova Neuroscience Center, University of Padua, Padua, 35129, Italy · Department of Biomedical Sciences, University of Padua, Padua, 35131, Italy +2

  26. Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features

    Jul 23, 2026Jisu Kim, Benjamin S. RigganBiometric IdentificationValence-Arousal Modeling

    University of Nebraska-Lincoln Lincoln, NE 68588

  27. Advancing All-Weather Building Damage Mapping to the Instance Level: Outcomes and Insights from the 2026 Bright Challenge

    Jul 23, 2026Hongruixuan Chen, He Huang, Haifeng Wang +19Disaster Damage AssessmentBuilding Damage Assessment

    RIKEN Center for Advanced Intelligence Project (AIP), RIKEN, Tokyo 103-0027, Japan · State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University, Wuhan 430079, China · Graduate School of Frontier Sciences, The University of Tokyo, Chiba 277-8561, Japan +3

  28. Adaptive Confidence-weighted Expansion for Trustworthy Multi-Omics Multimodal Fusion

    Jul 22, 2026Mohammad Raahemi, Ali Sekhavati, Alireza Maleki +1Probability CalibrationUncertainty Quantification

    Faculty of Engineering, University of Ottawa, Ottawa ON K1N 1A2, Canada · RIV Lab, Department of Computer Engineering, Bu-Ali Sina University, Hamedan, Iran · School of Computing and Communications, Lancaster University, Lancaster, UK

  29. Current Injection Spiking Neural Network for Infrared and Visible Image Fusion

    Jul 22, 2026Rui Zhao, Zhuoyuan Li, Wenrui Li +4Efficient Neural Network InferenceMulti-Source Image Fusion

    College of Computing and Data Science, Nanyang Technological University, Singapore · Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong · Department of Computer Science and Technology, Harbin Institute of Technology, Harbin, China +2

  30. Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention

    Jul 21, 2026Joy Dhar, Manish Kumar Pandey, Nayyar Zaidi +4Cross-Modal Representation LearningCross-Modal Attention

    Indian Institute of Technology Ropar Rupnagar, Punjab, India · RoentGen Health New Delhi, Delhi, India · Deakin University Melbourne, Victoria, Australia +3