Multimodal Fusion

Ask about this topic

Overview

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

CardsList

771 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

  1. Multimodal rumor detection enhanced by external evidence and forgery features

    Jan 21, 2026Han Li, Hua SunFake News DetectionCross-Modal Alignment

    Information Engineering School of Dalian Ocean University,Dalian Liaoning, China

  2. From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion

    Jan 15, 2026Cheng Chen, Yuyu Guo, Pengpeng Zeng +4Vision-Language ModelsVLM Adaptation

    Tongji University, China · Ant Group, China · Shanghai Innovation Institute, China +1

  3. Imaging-anchored multiomics in cardiovascular disease: integrating cardiac imaging, bulk, single-cell, and spatial transcriptomics

    Jan 10, 2026Minh H. N. Le, Thanh-Huy Nguyen, Tao Li +7Multimodal FusionSpatial Transcriptomics

    International Ph.D. Program in Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan · AIBioMed Research Group, Taipei Medical University, Taipei, Taiwan · Computational Biology Department, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA +6

  4. FEA-SLT: A Gloss-Free End-to-End Framework for Facial-Expression-Aware Sign Language Translation

    Jan 7, 2026Guobin Tu, Di WengSign Language TranslationMultimodal Fusion

    School of Software Technology, Zhejiang University

  5. RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarios

    Dec 31, 2025Tianyi Zhao, Jiawen Xi, Linhui Xiao +4VLM RobustnessVisual Grounding

    Institute of Artificial Intelligence, State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, Shanghai, China

  6. GPF-Net: Gated Progressive Fusion Learning for Polyp Re-Identification

    Dec 25, 2025Suncheng Xiang, Xiaoyang Wang, Junjie Jiang +2Multi-Scale Feature FusionMultimodal Fusion

    Shanghai Jiao Tong University · Peking University · Shanghai Fifth People’s Hospital

  7. Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Video

    Dec 13, 2025Daniel Adebi, Sagnik Majumder, Kristen GraumanRelative Pose EstimationCamera Pose Estimation

    The University of Texas at Austin, Austin TX 78712, USA

  8. A Spatial-Spectral-Frequency Interactive Network for Multimodal Remote Sensing Classification

    Oct 6, 2025Hao Liu, Yunhao Gao, Wei Li +3Remote Sensing Image UnderstandingFrequency-Domain Feature Learning

    Department of Information Engineering and Computer Science, University of Trento, 38123 Trento, Italy · School of Information and Electronics, Beijing Institute of Technology, Beijing 100081, China · Beijing Key Laboratory of Fractional Signals and Systems, Beijing Institute of Technology, Beijing 100081, China +3

  9. Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

    Sep 23, 2025Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual UnderstandingMultimodal Semantic Segmentation

    Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

  10. Diff-Feat: Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification

    Sep 19, 2025Tian Lan, Yiming Zheng, Jianxin YinMultimodal ClassificationMulti-Label Classification

    School of Statistics, Renmin University of China · Center for Applied Statistics and School of Statistics, Renmin University of China

  11. Invisible Yet Detected: PelFANet with Attention-Guided Anatomical Fusion for Pelvic Fracture Diagnosis

    Sep 17, 2025Siam Tahsin Bhuiyan, Rashedur Rahman, Sefatul Wasi +4Medical Image AnalysisMedical Image Classification

    Center for Computational & Data Sciences, Independent University, Bangladesh · Department of Computer Science and Engineering, Independent University, Bangladesh · Advanced Medical Engineering Research Institute, University of Hyogo, Japan +1

  12. SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation

    Aug 20, 2025Marshall Thomas, Edward Fish, Richard BowdenSign Language TranslationMultimodal Fusion

    University of Surrey, Guildford, GU2 7XH, United Kingdom

  13. GlaBoost: A Multimodal Structured Framework for Glaucoma Risk Stratification

    Aug 3, 2025Cheng Huang, Zeyu Han, Weizheng Xie +4Multimodal FusionClinical Risk Prediction

    Department of Computer Science, Southern Methodist University, Dallas, TX 75205, USA · Department of Ophthalmology, UT Southwestern Medical Center, Dallas, TX 75390, USA · High Performance Computing Program, National Aeronautics and Space Administration, Washington, DC 20546, USA

  14. The NTNU System at the S&I Challenge 2025 SLA Open Track

    Jun 5, 2025Hong-Yun Lin, Tien-Hong Lo, Yu-Hsuan Fang +4Audio-Language ModelsAutomated Evaluation

    Department of Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · Institute of AI Interdisciplinary Applied Technology, National Taiwan Normal University, Taiwan

  15. The P3^3 Dataset: Pixels, Points and Polygons for Multimodal Building Vectorization

    May 21, 2025Raphael Sulzer, Liuyun Duan, Nicolas Girard +1Remote Sensing Image SegmentationBuilding Footprint Extraction

    Université Côte d’Azur, INRIA, Sophia-Antipolis, France · LuxCarta Technology, Mouans-Sartoux, France

  16. Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing

    May 15, 2025Chunyu Ye, Yunhao Zhang, Jingyuan Sun +3Brain-Computer InterfacesNeural Decoding

    1State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Department of Computer Science, The University of Manchester +1

  17. Batch Augmentation with Unimodal Fine-tuning for Multimodal Fusion of Large Language Models

    May 10, 2025H M Dipu Kabir, Subrota Kumar Mondal, Mohammad Ali MoniMultimodal FusionMultimodal Classification

    AI and Cyber Futures Institute, Charles Sturt University, Australia · Rural Health Research Institute, Charles Sturt University, Australia · School of Computer Science and Engineering, Macau University of Science and Technology, Macao

  18. Uncertainty-Weighted Fusion of Image and Synthetic Event for Video Anomaly Detection

    May 5, 2025Sungheon Jeong, Jihong Park, Mohsen ImaniUncertainty-Aware Multimodal FusionMultimodal Anomaly Detection

    University of California, Irvine · MOLOCO

  19. A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language

    Dec 23, 2024Kris KraackMultimodal Emotion RecognitionMultimodal Fusion

    College of Computing, Georgia Institute of Technology, Atlanta, GA, USA.

  20. A Multimodal Feature Distillation with Mamba-Transformer Network for Brain Tumor Segmentation with Incomplete Modalities

    Apr 22, 2024Ming Kang, Fung Fung Ting, Shier Nee Saw +3Cross-Modal Knowledge DistillationCross-Modality Medical Image Segmentation

    School of Information Technology, Monash University, Malaysia Campus, Subang Jaya 47500, Malaysia · Augmented Intelligence and Multimodal analytics (AIM) for Health Lab, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia · Monash Medical AI Group, Monash University, Clayton VIC 3800, Australia +1

  21. GestaltMML: Enhancing Rare Genetic Disease Diagnosis through Multimodal Machine Learning Combining Facial Images and Clinical Text

    Dec 23, 2023Da Wu, Zhanliang Wang, Hongzhuo Chen +12Rare Disease DiagnosisMultimodal Classification

    Raymond G. Perelman Center for Cellular and Molecular Therapeutics, Children’s Hospital of Philadelphia, Philadelphia, PA 19104, USA · Department of Mathematics, University of Pennsylvania, Philadelphia, PA 19104, USA · Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, NY 10032, USA +8