Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

    Jul 8, 2026Kanglei Zhou, Ruizhi Cai, Xinning Wang +4Multimodal RobustnessCross-Modal Alignment

    Dept. of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China · State Key Lab of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Dept. of Rheumatology and Immunology, Children’s Hospital, Capital Institute of Pediatrics, Beijing, China

  2. AT-Attn: Temporal-Aware Cross-Attention for Longitudinal Multimodal Alzheimer's Disease Diagnosis

    Jul 8, 2026Xinyue Du, Yibo Liu, Zhenglei Zhou +3Alzheimer's DiseaseLongitudinal Medical Image Analysis

    School of Information Science and Engineering, East China University of Science and Technology, China · School of Computer Science, Shanghai Jiao Tong University, China · Tencent, China

  3. Multimodal Spatiotemporal-Frequency Fusion with Peak Enhancement for Cellular Traffic Forecasting

    Jul 8, 2026Qingzhong Li, Yue Hu, Hui Ma +4Exogenous Time Series ForecastingTime Series Forecasting

    Xinjiang Key Laboratory of Intelligent Computing and Smart Applications, School of Software, Xinjiang University, Urumqi 830017, China. · College of Geography and Remote Sensing Sciences, Xinjiang University, Urumqi 830046, China.

  4. General Incomplete Multimodal Learning via Dynamic Quality Perception

    Jul 8, 2026Xiangyu Meng, Shicai WeiMissing-Modality LearningMultimodal Robustness

    University of Electronic Science and Technology of China

  5. TMF-RSE: Tri-Modal Fusion with Regional Semantics and Evidential Uncertainty for Lung Severity Scoring

    Jul 7, 2026Fadi Abdeladhim Zidi, Salah Eddine Bekhouche, Abdellah Zakaria Sellam +3Clinical PredictionMultimodal Fusion

    Institute of Applied Sciences and Intelligent Systems (ISASI), CNR, 73100 Lecce, Italy · Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain · University of Salento, 73100 Lecce, Italy

  6. Multimodal Molecular Representation Learning with Graph Neural Networks, Deep & Cross Networks, and SMILES Embeddings

    Jul 7, 2026Qiwei Han, Chi Zhou, Ruobing Wang +1Molecular Property PredictionMultimodal Fusion

    Department of Chemistry, Duke University, Durham, NC 27708, USA · Department of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332, USA

  7. Deep Learning for Semen Analysis in Male Infertility: Computer Vision, Multimodal Fusion, and Clinical Translation

    Jul 6, 2026Runwei Guan, Shaofeng Liang, Jiacheng Weng +10Multimodal FusionMicroscopy Image Analysis

    Thrust of AI, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Department of Oncology, Suzhou Xiangcheng People’s Hospital, Suzhou, China · FertiTech AI, Shanghai, China +7

  8. FUSE: FK-Steered Multi-Modal Flow Matching for Efficient Simulation-Based Posterior Estimation

    Jul 6, 2026Weichen Qin, Yufan Xie, Peihao Wang +8Flow MatchingSimulation-Based Inference

    ShanghaiTech University, Shanghai, China · The University of Texas at Austin, Austin, TX, USA · Center for Gravitational Wave Experiment, National Microgravity Laboratory, Institute of Mechanics, Chinese Academy of Sciences, Beijing 100190, China +2

  9. Beyond Modality Fusion: Deep Ensembles for Multimodal Classification

    Jul 6, 2026Ilya Burenko, Dmitry VetrovModality ImbalanceMultimodal Classification

    ScaDS.AI, Technische Universität Dresden · Dresden, Germany · Constructor University +1

  10. EVAS: Efficient Multimodal Temporal Forgery Localization via Audio-Visual Synergy and Steered Boundary Calibration

    Jul 5, 2026Shen Shen, Quan Zhang, Dan Jiang +1Audio-Visual Representation LearningDigital Forensics

    Soochow University · Tsinghua University

  11. Explainable AI for Screening Abuse-Related Trauma in Bangladeshi Children: A Training-Free Multimodal Framework Evaluated on Noise-Aware Synthetic Data

    Jul 4, 2026Salma Hoque Talukdar Koli, Fahima Haque Talukder JelyExplainable Artificial IntelligenceMental Health

    Dept. of Computer Science & Engineering RTM Al-Kabir Technical University Sylhet-3100, Bangladesh · Dept. of Computer Science & Engineering North East University Bangladesh Sylhet, Bangladesh

  12. Cross-Modal Fusion of OCT and OCT angiography enface for Improved Diagnostics of Diabetic Retinopathy

    Jul 4, 2026Rashadul Hasan Badhon, Atalie Carina Thompson, Jennifer I. Lim +2Diabetic Retinopathy GradingOptical Coherence Tomography

  13. Q-TriM: Question-Guided Tri-Modal Attention for Audio-Visual Question Answering

    Jul 4, 2026SungHun Kim, SeungJun BaekMultimodal QACross-Modal Attention

    Dept. of Computer Science and Engineering Korea University Seoul, South Korea

  14. InfraNet: Quality-Aware RGB Guidance for Efficient Infrared Object Detection

    Jul 4, 2026Zichao Feng, Haodong Zhu, Jingying Yang +8Efficient Multimodal InferenceMultimodal Robustness

    Beihang University, Beijing, China · Zhongguancun Academy, Beijing, China · Communication University of China, Beijing, China +1

  15. Can Conversational Temporal Dynamics Improve Depression Detection in Dyads? A Preliminary Investigation in Multi-Modality Perspectives

    Jul 4, 2026Hanie Kang, Huang-Cheng Chou, Sudarsana Reddy Kadiri +1Depression DetectionMultimodal Fusion

    1Thomas Lord Department of Computer Science, University of Southern California, USA · 2Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California, USA

  16. Multimodal Knowledge Aggregation for Microbial Keratitis: Geographic Confounding and Shortcut Risk in Pooled Multicountry Data

    Jul 4, 2026Yiqing Wang, Maria A. Woodward, Ziyun Yang +11Multimodal ClassificationMultimodal Fusion

    Department of Biomedical Engineering, Duke University, Durham, NC, USA. · Kellogg Eye Center, Department of Ophthalmology and Visual Sciences, University of Michigan, Ann Arbor, MI, USA. · Department of Cornea and Refractive Surgery Services, Aravind Eye Care System, Madurai, Tamil Nadu, India. +4

  17. Jointly Improving Dialect Identification and ASR in Indian Languages using Multimodal Feature Fusion

    Jul 3, 2026Saurabh Kumar, Amartyaveer, Prasanta Kumar GhoshMulti-Task LearningMultimodal Fusion

    Department of Electrical Engineering, Indian Institute of Science, Bangalore, India

  18. Echoes of Unrest: A Multimodal NLP Framework for Early Warning of Fake News and Violence-Driven Mob Activity

    Jul 2, 2026Md. Maruf Bangabashi, Tahmid Hasan, Golam Mahmud +3Social Media AnalysisFake News Detection

  19. Multimodal Fusion for Fine-Grained Classification of Breast Fibroadenoma and Phyllodes Tumors

    Jul 2, 2026Chuxi Nan, Di Wu, Hongming Guo +4Multimodal ClassificationUltrasound Imaging

    School of Integrated Circuits, Wuxi Vocational College of Science and Technology · School of Advanced Technology, Xi’an Jiaotong-Liverpool University · Shanghai Gem Science and Technology Co., Ltd. +2

  20. Multi-modal Rail Crossing Safety Analysis

    Jul 1, 2026Paimon Goulart, Chansong Lim, Nícolas Roque dos Santos +4Multimodal ClassificationMultimodal Fusion

    University of California, Riverside Riverside, California, USA · Riverside County Transportation Commission Riverside, California, USA

  21. QuaMoE-DRF: Proactive Beam and Rate Adaptation via Multimodal Dynamic Radio Map Forecasting in ISAC Networks

    Jul 1, 2026Zhihan Zeng, Kaihe Wang, Zhongpei Zhang +1Integrated Sensing and CommunicationWireless Communications

    National Key Laboratory of Wireless Communications, University of Electronic Science and Technology of China, Chengdu 611731, China · University of Electronic Science and Technology of China, Chengdu 611731, China · College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China +2

  22. Learning When to Listen: Gated Affect Fusion for Human Motion Prediction

    Jul 1, 2026Jingni HuangCross-Modal LearningGated Multimodal Fusion

    University of Oxford

  23. Leveraging Multimodality for Real-Time Classification of Transients and Variables found by the Zwicky Transient Facility

    Jun 30, 2026Ved G. Shah, Nabeel Rehemtulla, Adam A. Miller +13Hierarchical ClassificationTime Series Classification

    Department of Physics and Astronomy, Northwestern University, 2145 Sheridan Road, Evanston, IL 60208, USA · Center for Interdisciplinary Exploration and Research in Astrophysics (CIERA), 1800 Sherman Ave, Evanston, IL 60201, USA · NSF-Simons AI Institute for the Sky (SkAI), 172 E. Chestnut St., Chicago, IL 60611, USA +8

  24. MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs

    Jun 30, 2026Zhongyang Li, Yaqian Li, Faming Fang +6Multimodal Large Language ModelsMultimodal Fusion

    Li Auto Inc. · East China Normal University.

  25. Evidence Triangulation for Multimodal Fact-Checking in the Wild

    Jun 30, 2026Stefanos-Iordanis Papadopoulos, Zacharias Chrysidis, Christos Koutlis +2Automated Fact-CheckingMultimodal Fusion

    Information Technologies Institute, Centre for Research & Technology, Hellas, Greece · Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki, Greece · 1Information Technologies Institute, Centre for Research & Technology, Hellas, Greece

  26. MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration

    Jun 30, 2026Gabriel ClingerCross-Modal LearningMultimodal Fusion

    The George Washington University

  27. Building a Multimodal Dataset of Academic Paper for Keyword Extraction

    Jun 30, 2026Jingyu Zhang, Xinyi Yan, Yi Xiang +2Multimodal DatasetMultimodal Fusion

    Department of Information Management, Nanjing University of Science and Technology, China · Department of Archives and E-government, Soochow University, China

  28. Dual Sparse Aggregation Transformer for Multispectral Object Detection

    Jun 30, 2026Wencong Wu, Xiuwei Zhang, Hanlin Yin +2Multispectral Object DetectionDetection Transformer

    School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China · School of Computer Science, the Shaanxi Provincial Key Laboratory of Speech and Image Information Processing and the National Engineering Laboratory for Integrated Aerospace-Ground-Ocean Big Data Application Technology, Northwestern Polytechnical University, Xi’an 710072, China · School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China

  29. PromptGNN-sim: Deep Fusion and Alignment of GNN and LLMs for Text-Attributed Graph Learning

    Jun 29, 2026Zhifei Hu, Alexandra I. CristeaText-Attributed Graph LearningMultimodal Fusion

    Durham University, Durham, United Kingdom

  30. Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion

    Jun 29, 2026Chao Tian, Zikun Zhou, Chao Yang +2Multispectral Object DetectionEfficient Object Detection

    Harbin Institute of Technology, Shenzhen, Guangdong, China