Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers88 papers in the last four weeks, up 26% on the four weeks before. 0.5% of all new papers.

  1. AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition

    Apr 14, 2026Zeheng Wang, Zitong Yu, Yijie Zhu +9Multimodal Emotion RecognitionMultimodal RAG

    Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, Guangdong, China · Great Bay University, Dongguan, Guangdong, China · Guangdong Provincial Key Laboratory of Intelligent Information Processing & Shenzhen Key Laboratory of Media Security, Shenzhen University, Shenzhen, Guangdong, China

  2. TimeSAF: Towards LLM-Guided Semantic Asynchronous Fusion for Time Series Forecasting

    Apr 14, 2026Fan Zhang, Shiming Fan, Hua WangLLM-Based Time Series ForecastingLong-Term Time Series Forecasting

    Shandong Technology and Business University · Ludong University

  3. EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts

    Apr 14, 2026Runhe Zhou, Shanglin Li, Guanxiang Huang +5Multimodal LearningMultimodal Fusion

    Nanyang Technological University, Singapore · BIFOLD, Berlin Institute for the Foundations of Learning and Data, Berlin, Germany · ATR, Kyoto, Japan +2

  4. Cross-Attentive Multiview Fusion of Vision-Language Embeddings

    Apr 14, 2026Tomas Berriel Martins, Martin R. Oswald, Javier CiveraMulti-View Feature Fusion3D VLMs

    University of Zaragoza, Spain · University of Amsterdam, Netherlands

  5. Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

    Apr 14, 2026Kang He, Yuzhe Ding, Xinrong Wang +3Multimodal Sentiment AnalysisModality Imbalance

    Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University. · Shanghai Innovation Institute

  6. SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

    Apr 14, 2026Junbin Su, Ziteng Xue, Shihui Zhang +3Multimodal Visual Object TrackingEfficient Multimodal Inference

    School of Artificial Intelligence (School of Software), Yanshan University · School of Software, Beihang University · Hebei Key Laboratory of Computer Virtual Technology and System Integration +3

  7. Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

    Apr 14, 2026Hao Wang, Jiqing Zhang, Xin Yang +4Camouflaged Object DetectionMultimodal Learning

    Information Science and Technology College, Dalian Maritime University, Liaoning, 116026, China · Dalian University of Technology, Liaoning, 116024, China · Beijing University of Technology, Beijing, 100124, China

  8. OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion

    Apr 14, 2026Dongjian Yu, Weiqing Min, Qian Jiang +3Multimodal Fusion

    School of Software, Yunnan University, China · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China

  9. RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation

    Apr 14, 2026Guoan Xu, Yang Xiao, Guangwei Gao +3Multimodal Semantic SegmentationSemantic Segmentation

    Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW 2007, Australia · PCA Lab, Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Bionic Vision Systems Laboratory, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai 200050, China +2

  10. Why Your Tokenizer Fails in Information Fusion: A Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization

    Apr 13, 2026Xiangyu Zhang, Benjamin John Southwell, Siqi Pan +3Audio TokenizationMultimodal Fusion

    School of Electrical Engineering and Telecommunications, University of New South Wales, Sydney, Australia.

  11. Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection

    Apr 13, 2026Jiaqi Wu, Zhen Wang, Enhao Huang +5Multispectral Object DetectionMultimodal Fusion

    Department of Automation, Tsinghua University, BeiJing, China · School of Artificial Intelligence, China University of Mining Technology - Beijing, BeiJing, China · State Key Laboratory of Blockchain and Data Security, Zhejiang University, HangZhou, China +1

  12. Hyper-FEOD: Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE

    Apr 13, 2026Wei Bao, Yuehan Wang, Tianhang Zhou +1Event-Based Object DetectionMultimodal Fusion

    BNRist, THUIBCS, BLBCI, School of Software, Tsinghua University · State Key Laboratory of Heavy Oil Processing, China University of Petroleum (Beijing), Beijing, 102249, China

  13. MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes

    Apr 13, 2026Samir Wagle, Reewaj Khanal, Abiral AdhikariHate Speech DetectionMultimodal Sentiment Analysis

    Department of Computer Science and Engineering, Kathmandu University, Nepal

  14. A Dual Cross-Attention Graph Learning Framework For Multimodal MRI-Based Major Depressive Disorder Detection

    Apr 11, 2026Nojod M. Alotaibi, Areej M. AlhothaliDepression DetectionMultimodal Fusion

    Department of Computer Science, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

  15. Improving Pediatric Emergency Department Triage with Modality Dropout in Late Fusion Multimodal EHR Models

    Apr 10, 2026Tyler Yang, Romal MitrClinical TriageHealthcare

    Department of Computer Science, Stanford University · Department of Bioengineering, Stanford University

  16. Customized Fusion: A Closed-Loop Dynamic Network for Adaptive Multi-Task-Aware Infrared-Visible Image Fusion

    Apr 10, 2026Zengyi Yang, Yu Liu, Juan Cheng +3Infrared-Visible Image FusionMulti-Source Image Fusion

    Department of Biomedical Engineering, Hefei University of Technology · College of Automation, Chongqing University of Post and Telecommunications · Faculty of Information Engineering and Automation, Kunming University of Science and Technology

  17. MSCT: Differential Cross-Modal Attention for Deepfake Detection

    Apr 9, 2026Fangda Wei, Miao Liu, Yingxue Wang +3Deepfake DetectionMultimodal Deepfake Detection

    Beijing Institute of Technology, China · China Academy of Electronics and Information Technology, China

  18. Bridging MRI and PET physiology: Untangling complementarity through orthogonal representations

    Apr 8, 2026Sonja Adomeit, Kartikay Tehlan, Lukas Förner +7Medical ImagingMultimodal Fusion

    Dept. of diagnostic and interventional Radiology and Neuroradiology, University Hospital Augsburg, Germany · Digital Medicine, University Hospital Augsburg, Germany · Chair for Computer Aided Medical Procedures and Augmented Reality, Technical University of Munich, Germany +4

  19. USCNet: Transformer-Based Multimodal Fusion with Segmentation Guidance for Urolithiasis Classification

    Apr 8, 2026Changmiao Wang, Songqi Zhang, Yongquan Zhang +9Medical DiagnosisMedical Image Classification

    C. Wang and X. Wan are with Shenzhen Research Institute of Big Data, Shenzhen 518172, China · Y. Zhang, S. Zhang, and Y. Wang are with Zhejiang University of Finance and Economics, Hangzhou 310018, China · L. Liu is with Anhui University of Finance and Economics, Anhui 233000, China +4

  20. Multi-modal user interface control detection using cross-attention

    Apr 8, 2026Milad Moradi, Ke Yan, David Colwell +2Multimodal FusionCross-Modal Attention

    AI Research Lab, Tricentis, Vienna, Austria · AI Research Lab, Tricentis, Sydney, Australia · Institute of Artificial Intelligence, Center for Medical Statistics, Informatics, and Intelligent Systems, Medical University of Vienna, Vienna, Austria

  21. RASR: Retrieval-Augmented Semantic Reasoning for Fake News Video Detection

    Apr 8, 2026Hui Li, Peien Ding, Jun Li +5Multimodal RAGFake News Detection

    School of Informatics, Xiamen University, Xiamen, China · School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, China · School of Computer and Big Data, Minjiang University, Fuzhou, China +2

  22. Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition

    Apr 7, 2026Tianyi Liu, Yiming Li, Wenqian Wang +4Multimodal UnderstandingMultimodal Fusion

    Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore · Singapore University of Technology and Design, 8 Somapah Rd, 487372, Singapore · The Hong Kong Polytechnic University, 6/F, Core D, Lui Che Woo Building, Hung Hom, Hong Kong SAR, China

  23. Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approaches

    Apr 7, 2026Marcin Michał MirończukData FusionMultimodal Fusion

    National Information Processing Institute, al. Niepodległości 188b, 00-608, Warsaw, Poland

  24. Prior-guided Fusion of Multimodal Features for Change Detection from Optical-SAR Images

    Apr 7, 2026Xuanguang Liu, Lei Ding, Yujie Li +8Remote Sensing Change DetectionMultimodal Fusion

    Institute of Geospatial Information, Information Engineering University, Zhengzhou, China · Academy of Digital China (Fujian), Fuzhou University, Fuzhou, China · The School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen, China +1

  25. The Topology of Multimodal Fusion: Why Current Architectures Fail at Creative Cognition

    Apr 6, 2026Xiujiang TanMultimodal FusionMultimodal Reasoning

    Guangzhou Academy of Fine Arts, Guangzhou, China · Guangzhou Academy of Fine Arts

  26. Good Rankings, Wrong Probabilities: A Calibration Audit of Multimodal Cancer Survival Models

    Apr 5, 2026Sajad GhawamiCancer Survival PredictionMultimodal Survival Analysis

    Independent Researcher

  27. Solar-VLM: Multimodal Vision-Language Models for Augmented Solar Power Forecasting

    Apr 5, 2026Hang Fan, Haoran Pei, Runze Liang +3Solar Power ForecastingSpatiotemporal Forecasting

    School of Economic and Management, North China Electric Power University, 102206, Beijing, China · School of Control and Computer Engineering, North China Electric Power University, 102206, Beijing, China · Department of Electrical Engineering, Tsinghua University, 100084, Beijing, China +1

  28. CAGMamba: Context-Aware Gated Cross-Modal Mamba Network for Multimodal Sentiment Analysis

    Apr 4, 2026Minghai Jiao, Jing Xiao, Peng Xiao +6Multimodal Sentiment AnalysisMultimodal Fusion

    School of Computer Science and Engineering, Northeastern University, Shenyang, China · Xingning Power Supply Bureau, Guangdong Power Grid Co., Ltd., Meizhou, China

  29. MultiPress: A Multi-Agent Framework for Interpretable Multimodal News Classification

    Apr 4, 2026Tailong Luo, Hao Li, Rong Fu +7Multimodal ClassificationMultimodal Agents

    New York Institute of Technology · University of Arizona · University of Macau +3

  30. CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

    Apr 3, 2026Ankan Deria, Komal Kumar, Xilin He +4Vision-Language ModelsMultimodal Fusion

    Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, UAE