Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

772 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. IB-HFN: Information Bottleneck-Driven SAR-Optical Fusion Network for High-Fidelity Cloud Removal

    Jun 8, 2026Haojun Guo, Fan Feng, Ziquan Wang +2Remote SensingMultimodal Fusion

    Institute of Geospatial Information, Information Engineering University, Zhengzhou 450001, China

  2. Heterogeneous 2D/1D Signal Representation Fusion for Underwater Acoustic Modulation Recognition Under Distribution Shift

    Jun 8, 2026Ronglai Qian, Liang An, Xiaoyan Wang +3Multimodal RobustnessMultimodal Fusion

    Key Laboratory of Underwater Acoustic Signal Processing of the Ministry of Education, Southeast University, Nanjing 210096, China

  3. Explicit Representation Alignment for Multimodal Sentiment Analysis

    Jun 8, 2026Baode Wang, Ziming Wang, Huacan Wang +2Cross-Modal AlignmentSentiment Analysis

    AgentAlpha

  4. Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

    Jun 8, 2026Siyuan Liu, Jinyang WuEfficient Multimodal InferenceMultimodal Large Language Models

    aSchool of Mechanics and Engineering Science, Peking University, China · bDepartment of Automation, Tsinghua University, China

  5. DifferSeg: Towards Diverse Multimodal Binary Segmentation via Differential Perception and Frequency Guidance

    Jun 8, 2026Qiangqiang Zhou, Jiawei Xu, Yong Chen +3Image SegmentationMultimodal Semantic Segmentation

    School of Artificial Intelligence, Jiangxi Normal University, Nanchang, China · Institute of AI Education, East China Normal University, Shanghai, China · Yale School of Medicine, Yale University, New Haven, CT, USA

  6. DSFNet: Learning Dual-Domain Spectral Operators for Multi-Modality Spatio-Temporal Forecasting in Urban Transportation Systems

    Jun 5, 2026Yongchao Li, Yang Li, Zhuoxuan Li +4Intelligent Transportation SystemsTime Series Forecasting

    School of Transportation, Southeast University, Nanjing 211189, China · Jiangsu Province Collaborative Innovation Center of Modern Urban Traffic Technologies, Nanjing 211189, China · Department of Data Science, City University of Hong Kong, Hong Kong 999077, China +6

  7. Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosis

    Jun 5, 2026Sanket Kachole, Siddhesh Thakur, Shubham Innani +4Missing-Modality LearningCancer Survival Prediction

    Division of Computational Pathology, Department of Pathology and Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, USA · IU Melvin and Bren Simon Comprehensive Cancer Center, Indianapolis, IN, USA · Departments of Biostatistics and Health Data Science; Radiology and Imaging Sciences; Neurological Surgery; Indiana University School of Medicine, Indianapolis, IN, USA +1

  8. PAMF: Prior-Aware Multimodal Fusion for Incomplete Time Series Data

    Jun 4, 2026Ziwen Kan, Wugeng Zheng, Tianlong Chen +1Missing-Modality LearningFlow Matching

    Department of Computer Science, University of Central Florida · Department of Computer Science, University of North Carolina at Chapel Hill

  9. TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models

    Jun 4, 2026Ziwen Kan, Yishuo Chen, Kecheng Li +7Irregular Time-Series ModelingMissing-Modality Learning

    Department of Computer Science, University of Central Florida · Department of Computer Science, University of North Carolina at Chapel Hill

  10. GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention

    Jun 4, 2026Giordano Cicchetti, Eleonora Grassucci, Danilo ComminielloCross-AttentionMultimodal Transformer

    Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy

  11. Step-adaptive multimodal fusion network with multi-scale cloud feature learning for ultra-short-term solar irradiance forecasting

    Jun 4, 2026Jingxin Zhang Xiaoqin WangSolar Irradiance ForecastingTime Series Forecasting

    School of Automation, Southeast University, Nanjing, 210096, China

  12. IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems

    Jun 4, 2026Tao Zhong, Jiajun Deng, Nikita Kuzmin +6Spoken Dialogue SystemsMultimodal Fusion

    The Chinese University of Hong Kong, China · AudioLab Hong Kong, Huawei Leibniz Research Center, China · Nanyang Technological University, Singapore

  13. To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection

    Jun 4, 2026Erfan Loweimi, Mengjie Qian, Kate Knill +7Cross-Modal RetrievalMissing-Modality Learning

    University of Cambridge, UK · Southwest Jiaotong University, China · University of Surrey, UK +1

  14. M2S-AVSR: Modality-aware Multi-view Self-supervised Representation for Robust Audio-Visual Speech Recognition

    Jun 4, 2026Fei Su, Cancan Li, Ming Li +1Multimodal RobustnessAudio-Visual Representation Learning

    School of Artificial Intelligence and the School of Computer Science, Wuhan University, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Artificial Intelligence, Wuhan University, China

  15. Beyond Missing Rates: Rethinking Incomplete Multi-View Clustering with Protocol Divergence

    Jun 3, 2026Haolu Liu, Xiyue Wang, Xuanting Xie +2Missing-Modality LearningLearning with Missing Data

  16. UniCA: Bi-directional Cross-Attention with Positive Similarity Loss for Robust Multi-Modal Retrieval

    Jun 3, 2026Yini Huang, Wenlong ZhangMultimodal IRCross-Modal Alignment

    The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Southern Medical University, Guangdong, China

  17. COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations

    Jun 3, 2026Zixu Li, Yupeng Hu, Zhiwei Chen +3Cross-Modal Representation LearningCross-Modal Alignment

    School of Software, Shandong University, Jinan, 250100, China · School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, 518000, China · Department of Data Science, City University of Hong Kong, Hong Kong 523808, China +1

  18. KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

    Jun 2, 2026Kevin Patel, Shashi Bhushan JhaFake News DetectionMultimodal Fusion

    Department of Computer Science, University of West Florida, Pensacola, FL, USA

  19. Visual Instruction Tuning Aligns Modalities through Abstraction

    Jun 2, 2026Luis Palacios, Lorenzo Basile, Diego Doimo +1VLM AdaptationVLM Interpretability

    Area Science Park, Trieste, Italy

  20. AQIFormer: A Transformer-Based Multi-View Architecture for Cross-City Air Quality Classification

    Jun 2, 2026Om Kathalkar, Nitin Nilesh, Sachin Chaudhari +1Multi-View LearningAir Quality Prediction

    IIIT Hyderabad · India

  21. Mixed-Modality Dual Face-Hair Retrieval

    Jun 2, 2026Quoc-Anh Bui-Huynh, Mai-Tuyen Lam, Dai-Anh-Tuan Nguyen +1Cross-Modal AlignmentCross-Modal Retrieval

    Vietnam National University, Ho Chi Minh City, Vietnam · University of Information Technology, VNU-HCM, Ho Chi Minh City, Vietnam

  22. Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network

    Jun 2, 2026Zhikang Li, Yan Wu, Xin Hu +2Multi-Scale Feature FusionCross-Modal Feature Matching

    Remote Sensing Image Processing and Fusion Group, School of Electronic Engineering, Xidian University, Xi’an 710071, China · National Key Laboratory of Radar Signal Processing, Xidian University, Xi’an 710071, China

  23. FAF-CD: Frequency-Aware Fusion for Change Detection under Imperfect Multimodal Remote Sensing

    Jun 2, 2026Yufan Wang, Sokratis Makrogiannis, Chandra KambhamettuMultimodal RobustnessFrequency-Domain Feature Learning

    University of South Florida · 2Delaware State University

  24. Hand Trajectory Fusion for Egocentric Natural Language Query Grounding

    Jun 1, 2026Enmin Zhong, Carlos R. del-Blanco, Fernando Jaureguizar +1Temporal Video GroundingEgocentric Video Understanding

    Grupo de Tratamiento de Im´agenes (GTI), Information Processing and Telecommunications Center , ETSI Telecomunicaci´on, Universidad Polit´ecnica de Madrid, Spain

  25. QUIVER: Quantum-Informed Views for Enhanced Representations in Large ML Models

    Jun 1, 2026Aritra Bal, Michael Binder, Markus Klute +2Quantum Machine LearningMultimodal Fusion

    Institute of Theoretical Physics (ITP), KIT, Karlsruhe, DE · Institute of Experimental Particle Physics (ETP), Karlsruhe Institute of Technology (KIT), Karlsruhe, DE · Blackett Laboratory, Imperial College of Science and Technology, London, UK +1

  26. Parameter-efficient Dual-encoder Architecture with Differentiable Choquet Integral Fusion for Underwater Acoustic Classification

    Jun 1, 2026Amirmohammad Mohammadi, Joshua Peeples, Alexandra Van DineUnderwater AcousticsMultimodal Fusion

    Department of Electrical and Computer Engineering, Texas A&M University, College Station, Texas, 77843, USA · Massachusetts Institute of Technology Lincoln Laboratory, Lexington, Massachusetts, 02421, USA

  27. Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals

    Jun 1, 2026Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang +2Multimodal RobustnessMultimodal Fusion

    Adelaide University Adelaide, Australia · Shandong University Jinan, China

  28. CL-DMDF:Dynamic Multimodal Data Fusion Model Based on Contrastive Learning

    Jun 1, 2026Dong Li, Lingling Zhang, Binghao Han +2Missing-Modality LearningCross-Modal Attention

    School of Information, Liaoning University, Shenyang, China · School of Computer Science and Engineering, Northeastern University, Shenyang, China

  29. Turing Patterns for Multimedia: Reaction-Diffusion Multi-Modal Fusion for Language-Guided Video Moment Retrieval

    Jun 1, 2026Xiang Fang, Wanlong Fang, Wei Ji +1Temporal Video GroundingReaction-Diffusion Systems

    Nanyang Technological University · Singapore, Singapore · Nanjing University +2

  30. NeuroAlign: Hierarchical Multimodal Fusion of Dynamic and Structural Neuroimaging for MCI Analysis

    May 31, 2026Xiongri Shen, Zhenxi Song, Jiaqi wang +13Feature AttributionFunctional Connectivity

    Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, 518055, China · School of Intelligence Science and Engineering, College of Artificial Intelligence, Harbin Institute of Technology, Shenzhen, 518055, China · School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China +5