Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

772 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. RACANet: Reliability-Aware Crowd Anchor Network for RGB-T Crowd Counting

    Apr 27, 2026Jinghao Shi, Mengqi Lei, Kunliang He +3Cross-Modal AlignmentCrowd Counting

    School of Computer Science, China University of Geosciences, Wuhan, Wuhan, China

  2. Point Cloud Registration for Fusion between SPECT MPI and CTA Images

    Apr 27, 2026Ni Yao, Xiangyu Liu, Shaojie Tang +9Medical ImagingMedical Image Registration

    School of Computer Science and Artificial Intelligence, Zhengzhou University of Light Industry, Zhengzhou 450001, Henan, China · School of Automation, Xi’an University of Posts and Telecommunications, Xi’an, Shaanxi, 710121, China · Xi'an Key Laboratory of Advanced Control and Intelligent Process, Xi'an, Shaanxi, 710121, China +5

  3. CMGL: Confidence-guided Multi-omics Graph Learning for Cancer Subtype Classification

    Apr 27, 2026Boyang Fan, Hengchuang Yin, Siyu Yi +5OncologyCancer Subtype Classification

    College of Computer Science, Sichuan University, Chengdu, China · Xinjiang Technical Institute of Physics and Chemistry, Chinese Academy of Sciences, Urumqi, China · College of Mathematics, Sichuan University, Chengdu, China +1

  4. Open-Vocabulary Semantic Segmentation Network Integrating Object-Level Label and Scene-Level Semantic Features for Multimodal Remote Sensing Images

    Apr 27, 2026Jinkun Dai, Yuanxin Ye, Peng Tang +4Remote Sensing Image SegmentationText-Guided Image Segmentation

    Faculty of Geosciences and Engineering, Southwest Jiaotong University, Chengdu 611756, China · State-Province Joint Engineering Laboratory of Spatial Information Technology for High-Speed Railway Safety, Southwest Jiaotong University, Chengdu 611756, China · School of Artificial Intelligence, Wuhan University, Wuhan 430072, China +1

  5. Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis

    Apr 27, 2026Xuemei Qiu, Dawei Fan, Yebin Huang +2Cross-Modal AlignmentPrototype Learning

    College of Computer and Information Science, Fujian Agriculture and Forestry University, Fuzhou, 350002, China · Fujian Agriculture and Forestry University, Fuzhou, 350002, China · Department of Pathology, Clinical Oncology School of Fujian Medical University, and Fujian Cancer Hospital, Fuzhou, 350014, China +2

  6. LAVA: Layered Audio-Visual Anti-tampering Watermarking for Robust Deepfake Detection and Localization

    Apr 27, 2026Bokang Zeng, Zheng Gao, Xiaoyu Li +2Audio WatermarkingDeepfake Detection

    School of Computer Science and Engineering UNSW Sydney Sydney, Australia · School of Information and Communication Technology Griffith University Brisbane, Australia

  7. Modeling Induced Pleasure through Cognitive Appraisal Prediction via Multimodal Fusion

    Apr 26, 2026Nastaran Dab, Raziyeh Zall, Mohammadreza KangavariMultimodal Model InterpretabilityMultimodal Fusion

    Department of Computer Engineering, Iran University of Science and Technology, Tehran, Iran.

  8. Emotion-Conditioned Short-Horizon Human Pose Forecasting with a Lightweight Predictive World Model

    Apr 26, 2026Jingni Huang, Peter BloodsworthGated Multimodal FusionHuman Motion Generation

  9. A Heterogeneous Two-Stream Framework for Video Action Recognition with Comparative Fusion Analysis

    Apr 25, 2026Md. Afzalur Rahaman, Tahmid RahmanVideo Action RecognitionCross-Modal Attention

    Department of Computer Science & Engineering Hamdard University Bangladesh Munshiganj, Bangladesh

  10. Contrastive Learning for Multimodal Human Activity Recognition with Limited Labeled Data

    Apr 25, 2026Long Jing, Zhixiong Yang, Yajun Zhang +1Cross-Modal Representation LearningHuman Activity Recognition

    College of Computer Science, Northwest University, Xi’an, China · Xinjiang University, Urumqi, China

  11. DARC-CLIP: Dynamic Adaptive Refinement with Cross-Attention for Meme Understanding

    Apr 25, 2026Qiyuan JinVLM AdaptationHate Speech Detection

    The Hong Kong University of Science and Technology

  12. Text-Guided Multimodal Unified Industrial Anomaly Detection

    Apr 24, 2026Zewen Li, Shuo Ye, Zitong Yu +2Unified Multimodal ModelsIndustrial Anomaly Detection

    School of Computer Science & Software Engineering, Shenzhen University, China, 518060 · School of Computing and Information Technology, Great Bay University, Dongguan, 523000,China

  13. MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment

    Apr 23, 2026Juan Li, Chuanghao Ding, Xujie Zhang +1Cross-Modal LearningCross-Modal Alignment

    State Key Laboratory for Novel Software Technology, Nanjing University

  14. Semi-Supervised Flow Matching for Mosaiced and Panchromatic Fusion Imaging

    Apr 22, 2026Peiming Luo, Nan Wang, Litong Liu +4Flow MatchingMulti-Source Image Fusion

    Southeast University · Hunan University

  15. Structure-Semantic Decoupled Modulation of Global Geospatial Embeddings for High-Resolution Remote Sensing Mapping

    Apr 21, 2026Jienan Lyu, Miao Yang, Jinchen Cai +4Remote Sensing Image SegmentationGeospatial Foundation Models

    Sun Yat-Sen University · Tsinghua University

  16. A Proxy Consistency Loss for Grounded Fusion of Earth Observation and Location Encoders

    Apr 20, 2026Zhongying Wang, Kevin Lane, Levi Cai +2Geospatial Representation LearningAir Quality Prediction

    University of Colorado, Boulder

  17. DUALVISION: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning

    Apr 20, 2026Abrar Majeedi, Zhiyuan Ruan, Ziyi Zhao +3Cross-Modal LearningVisual Question Answering

    University of Wisconsin-Madison · Amazon · Northeastern University

  18. DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation

    Apr 19, 2026Nagur Shareef Shaik, Teja Krishna Cherukuri, Dong Hye YeMedical ImagingMedical Report Generation

    Department of Computer Science, Georgia State University, Atlanta, GA USA

  19. Multimodal Fusion of Histopathology Images and Electronic Health Records for Early Breast Cancer Diagnosis

    Apr 18, 2026Aditya Shribhagwan Khandelwal, Mohammad Samar Ansari, Asra AslamBreast CancerComputational Pathology

    University of Sheffield, UK · University of Chester, UK

  20. Dynamic Emotion and Personality Profiling for Multimodal Deception Detection

    Apr 18, 2026Li Zheng, Yanyi Luo, Hao Fei +5Multimodal Emotion RecognitionUncertainty-Aware Multimodal Fusion

    Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan, China · National University of Singapore, Singapore, Singapore

  21. TriTS: Time Series Forecasting from a Multimodal Perspective

    Apr 17, 2026Xiang AoLong-Term Time Series ForecastingCross-Modal Representation Learning

    School of Software Engineering, Beijing Jiaotong University Beijing 100044, China

  22. A Two-Stage Multi-Modal MRI Framework for Lifespan Brain Age Prediction

    Apr 17, 2026Dingyi Zhang, Ruiying Liu, Yun WangBrain Age PredictionNeuroimaging

    Department of Computer Science, Emory University, Atlanta, GA, USA · Department of Biomedical Informatics, Emory University, Atlanta, GA, USA

  23. Information Router for Mitigating Modality Dominance in Vision-Language Models

    Apr 17, 2026Seulgi Kim, Mohit Prabhushankar, Ghassan AlRegibVLM RobustnessVision-Language Models

    OLIVES at the Center for Signal and Information Processing CSIP, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA

  24. Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization

    Apr 17, 2026Habibeh Naderi, Behrouz Haji Soleimani, Stan MatwinCross-Modal LearningModality Imbalance

    Dalhousie University, Halifax NS, Canada

  25. SWNet: A Cross-Spectral Network for Camouflaged Weed Detection

    Apr 17, 2026Henry O. Velesaca, Luigi Miranda, Angel D. SappaAgricultural Image AnalysisWeed Detection

    ESPOL Polytechnic University, Campus Gustavo Galindo, 090902, Guayaquil, Ecuador · Software Engineering Department, Research Center for Information and Communication Technologies (CITIC-UGR), University of Granada, 18071, Granada, Spain · Computer Vision Center, Universitat Aut`onoma de Barcelona, 08193-Bellaterra, Barcelona, Spain

  26. Dual-Modal Lung Cancer AI: Interpretable Radiology and Microscopy with Clinical Risk Integration

    Apr 17, 2026Baramee Sukumal, Aueaphum AueawatthanaphisutMedical Image ClassificationExplainable Medical Image Analysis

    Hatyaiwittayalai School, Hat Yai, Songkhla, Thailand, 90110 · School of Information, Computer, and Communication Technology,2026 Sirindhorn International Institute of Technology, Thammasat University, Pathum Thani, Thailand

  27. EviDep: Uncertainty-Aware Multimodal Depression Estimation via Disentangled Evidential Learning

    Apr 17, 2026Fangyuan Liu, Sirui Zhao, Yangsong Zhang +5Disentangled Representation LearningEvidential Deep Learning

    School of Computer Science and Technology, University of Science and Technology of China, Hefei, China · aSchool of Computer Science and Technology, University of Science and Technology of China, Hefei, China · bSchool of Computer Science and Technology, Laboratory for Brain Science and Artificial Intelligence, Southwest University of Science and Technology, Mianyang, China +1

  28. ExoNet: Calibrated Multimodal Deep Learning for TESS Exoplanet Candidate Vetting using Phase-Folded Light Curves, Stellar Parameters, and Multi-Head Attention

    Apr 16, 2026Md. Rashadul IslamTime Series ClassificationAI-Assisted Scientific Research

    Dept. of Computer Science and Engineering Daffodil International University Dhaka, Bangladesh

  29. Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs

    Apr 16, 2026Ziyang Luo, Nian Liu, Junwei HanMultimodal Large Language ModelsMultimodal Reasoning

    Northwestern Polytechnical University Xi’an, Shaanxi, China

  30. M3R: Localized Rainfall Nowcasting with Meteorology-Informed MultiModal Attention

    Apr 15, 2026Sanjeev Panta, Rhett M Morvant, Xu Yuan +2Precipitation NowcastingSpatiotemporal Forecasting

    University of Louisiana at Lafayette, Lafayette, LA, USA · University of Delaware, Newark, DE, USA