Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

766 papers88 papers in the last four weeks, up 26% on the four weeks before. 0.5% of all new papers.

  1. I2^2-KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis

    Oct 22, 2026Xiaoting Xie, Feng Qiao, Xianwei Zhang +5Multimodal Sentiment AnalysisMultimodal Fusion

    School of Information Science and Engineering, Shandong University, Qingdao, Shandong, China · Department of Computer Science, University of Exeter, Exeter, UK · School of Computing Science, University of Glasgow, Glasgow, UK +1

  2. ICGF-CXRMate: Image-Conditioned Clinical Context Compression and Gated Fusion for Chest X-ray Report Generation

    Oct 22, 2026Vu Minh Tran, Thuc Huynh, Son Thanh Luu +1Gated Multimodal FusionRadiology Report Generation

    University of Information Technology · Viet Nam National University Ho Chi Minh City

  3. AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning

    Oct 22, 2026Chuan Han Li, Yun Chien ChengMultimodal Emotion RecognitionMultimodal Fusion

    Department of Mechanical Engineering, College of Engineering, National Yang Ming Chiao Tung University, Hsin-Chu, Taiwan · Department of Electrical Engineering, College of Engineering, National Taiwan University, Taipei, Taiwan

  4. Multi-Scale Latent Temporal Compression Attention and Video-Text Cross Attention Fusion for Video Action Detection

    Oct 21, 2026Yicheng Qiu, Keiji YanaiVideo Action RecognitionTemporal Video Understanding

    The University of Electro-Communications, Tokyo, Japan

  5. GAMF-Net: Grouped Availability-Aware Multimodal Fusion for Smartphone Skin Lesion Classification

    Oct 20, 2026Yue Zhuo, Jingjia Yan, Fangyu Liu +4Skin Lesion ClassificationMultimodal Classification

    Huizhou University, Huizhou 516007, Guangdong, China

  6. HypTI: Hyperbolic Geometry-Aware Multimodal Feature Fusion Framework for Dementia Detection Based on Tabular-Image Data

    Oct 20, 2026Jianfei Xu, Huizhi Liang, Judith Harrison +3Multimodal FusionMultimodal Classification

    Newcastle University, Newcastle upon Tyne NE1 7RU, UK

  7. Revealing Inter-Channel Modality Discriminability: Channel Selective Interaction Fusion Network for Optical-SAR Object Detection

    Oct 20, 2026Haojie Li, Yuanfei Huang, Lingfei Song +2Object DetectionMultimodal Object Detection

    School of Artificial Intelligence, Beijing Normal University · Beijing Key Laboratory of Artificial Intelligence for Education · Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education

  8. Progressive multi-scale tumor-state memory for incomplete multimodal brain tumor segmentation

    Oct 19, 2026Milad Rostamian, Mohammadreza Mohammadi, Mohsen SoryaniMissing-Modality Medical Image SegmentationMedical Image Segmentation

    School of Computer Engineering, Iran University of Science and Technology (IUST), Hengam Street, Resalat Square, 13114-16846, Tehran, Iran

  9. RGB--Event Object Detection Across Operating Frequencies via Reliability-Supervised Routing and Asymmetric-Resolution Fusion

    Oct 19, 2026Peijun Shi, Jiannan Zhao, Xiujin Zhu +3RGB-Event Sensor FusionReliability-Aware Multimodal Fusion

    Department of Electrical Engineering, Faculty of Engineering, Universiti Malaya, 50603 Lembah Pantai, Kuala Lumpur, Malaysia. · School of Electrical Engineering, Guangxi University, Nanning 530004, China · Computational Intelligence Laboratory, School of Computer Science, University of Lincoln, Lincoln LN6 7TS, U.K.

  10. CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings

    Oct 19, 2026Muntaqim Ahmed Raju, Priyanka Siddappa, Md Shifat Haider Al Amin +2Multimodal FusionChest X-Ray Classification

    University of Massachusetts Lowell, USA · NYC Health & Hospitals/Jacobi-NCB, USA

  11. SZED at ArGuard Shared Tasks: Multi-Task Label-Attention and Cross-Modal Fusion for Arabic Hateful Meme Detection

    Oct 18, 2026Selma Khelili, Zohra AbdeliHarmful Meme DetectionMulti-Label Classification

    ENSIA, Algeria

  12. Shared-Private Cross-Modal Discrepancy Learning for Imbalanced Multi-Source Audio-Visual Deepfake Detection

    Oct 18, 2026Qiwu Wen, Jahangir AlamMultimodal Deepfake DetectionMultimodal Fusion

    Luqia Technologies, 405 Ogilvy Ave., Suite 101, Montréal, Québec H3N 1M3, Canada

  13. DeliMF: Gated Cross-Attention Fusion of Structured Time Series and Clinical Text for Early Intensive Care Delirium Prediction

    Oct 17, 2026Yibo Yang, Zerui Huang, Haomiao Yang +1Clinical PredictionClinical Risk Prediction

    Independent Researcher, China · Xinjin District People’s Hospital, China · University of Exeter, United Kingdom

  14. Cross-Dimension and Cross-Modality Relational Tensor Representation Learning for Multimodal Health Data Modeling

    Oct 17, 2026Jiawei Guo, Meng ZhaoMultimodal LearningCross-Modal Representation Learning

    Department of Industrial and Systems Engineering, Lehigh University

  15. MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information

    Oct 17, 2026Xinyu Chen, Liangjian Wen, Jiang Duan +7Multimodal FusionModality Reliance in Multimodal Models

    School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid +4

  16. Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins

    Oct 16, 2026Yi Fang, Gaurav Srivastava, Sirui Ding +1Multimodal FusionMultimodal Model Evaluation

    Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

  17. Enhancing RAG-based Medical VQA with Gated Multimodal Representations

    Oct 16, 2026Daniela Jimenez-Pano, Hugo Jair Escalante, Delia Irazú Hernández-FaríasGated Multimodal FusionMedical VQA

    LabTL - INAOE, Puebla, Mexico · Institute for AAII, UTEP, El Paso, TX, USA · Department of Computer Science, UTEP, El Paso, TX, USA

  18. ODSS-Net: Orthogonally Decoupled Shared-Specific Representations for Multi-Modal Object Re-Identification

    Oct 16, 2026Zhifeng Ma, Yuhong Feng, Chongyi Huang +2Multimodal Disentangled Representation LearningMultimodal Fusion

    College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China · Tencent, Shenzhen, China

  19. FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

    Oct 15, 2026Xin Zhang, Robby T. TanMultimodal Semantic SegmentationMultimodal Fusion

    National University of Singapore

  20. Reliability-Aware Multimodal Sentiment Prediction Under Local Modality Missingness

    Oct 14, 2026Minghao Kong, Deng Yao, Xi Zhang +4Multimodal Sentiment AnalysisMissing-Modality Learning

    School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen, China · Department of Network Intelligence, Pengcheng Laboratory, Shenzhen, China · School of Mathematics, Sun Yat-sen University, Guangzhou, China +1

  21. Context-aware Bayesian Fusion for Multimodal Object Detection

    Oct 13, 2026Arunachalam Venkatachalam, Mhafuzul Islam, Marwan Yusuf +3Multimodal Object DetectionObject Detection

    imec Michigan, USA · imec and Ghent University Ghent, Belgium · imec Leuven, Belgium

  22. GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition

    Oct 13, 2026Ting Zhu, Huizhi Liang, Boguslaw Obara +2Multimodal Emotion RecognitionMultimodal Fusion

    Newcastle University, Newcastle upon Tyne, United Kingdom · Taiyuan University of Technology, Shanxi, China

  23. Parameter-Efficient Multimodal Brain Tumor Segmentation With Stage-Wise Shared Adapters

    Oct 13, 2026Milad Rostamian, Mohammadreza Mohammadi, Ender Konukoglu +1Brain Tumor SegmentationCross-Modality Medical Image Segmentation

    School of Computer Engineering, Iran University of Science and Technology (IUST), Tehran, Iran · Computer Vision Laboratory, ETH Zurich, 8092 Zurich, Switzerland

  24. Multimodal Fusion of RGB and Complementary Modalities for Semantic Segmentation

    Oct 13, 2026Hu Cao, Yifeng Cheng, Mu He +3Multimodal Semantic SegmentationMultimodal Fusion

    School of Automation, Southeast University, Sipailou 2, Nanjing, 210096, China. · Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich, Boltzmannstraße 15, Munich, 85748, Germany.

  25. Focus on What You Need: Global-to-Local Dynamic Attention for Multimodal Sarcasm Detection

    Oct 13, 2026Dehao Wu, Xingjie ZhuangSarcasm DetectionCross-Modal Attention

    Xiamen University

  26. Merit: Making every modality earn its contribution in multimodal attention decoding

    Oct 13, 2026Ali Alavi, Donald S. WilliamsonNeural DecodingEEG Decoding

    Department of Computer Science and Engineering The Ohio State University Columbus, OH 43210, USA

  27. Reference-Efficient Multimodal Bitstream-Corrupted Video Recovery via Depth-Conditioned Adaptation

    Oct 11, 2026Tianyi Liu, Jiaojiao Wang, Ye Lu +4Video RestorationMultimodal Fusion

    Nanyang Technological University, Singapore · School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin 150001, China +2

  28. Toward a Multimodal Foundation Model for Plasma State Reconstruction

    Oct 10, 2026Bin Xia, Chuanfei Dong, Boting Li +9Multimodal Foundation ModelsMissing-Modality Learning

    School of Physics, Georgia Institute of Technology, Atlanta, GA 30332, USA · Department of Astronomy, Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA · Princeton Plasma Physics Laboratory, Princeton, NJ 08540, USA +5

  29. HiPAD: A Unified Gait, Audio-Visual, and Personality Fusion Network for Depression Detection, with a Hierarchical Personality Ablation and the Calibration Limits of Domain-Adapted LLMs

    Oct 10, 2026Karol Chlasta, Piotr StruzikDepression DetectionDepression Severity Estimation

    Kozminski University Warsaw, Poland · WarsawIQ Warsaw, Poland

  30. BridgeWAM: Bridging Video and Action Experts via Latent Bridge Queries

    Oct 10, 2026Jiaxin Fang, Ruitong Liu, Zeyu Lai +6Vision-Language-Action ModelsEfficient Inference for World Action Models

    Fudan University · CocoMatrix · Peking University +3