Multimodal Fusion

Follow this topic Ask about this topic

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

772 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

Uncovering Trajectory and Topological Signatures in Multimodal Pediatric Sleep Embeddings
May 13, 2026Scott Ye, Harlin LeeMultimodal Embedding Clinical Prediction
Department of Radiology, University of California, San Francisco · School of Data Science and Society, University of North Carolina at Chapel Hill
Multimodal Graph-based Classification of Esophageal Motility Disorders
May 13, 2026Alexander Geiger, Lars Wagner, Daniel Rueckert +3Multimodal Graph Learning Multimodal Fusion
Technical University of Munich, School of Medicine and Health, TUM University Hospital Rechts der Isar, Research Group MITI, Munich, Germany. · Technical University of Munich, School of Computation, Information and Technology, Chair for AI in Healthcare and Medicine, Munich, Germany. · Imperial College London, Department of Computing, London, UK. +2
Supervised Deep Multimodal Matrix Factorization for Interpretable Brain Network Analysis
May 13, 2026Amjad Seyedi, Lifang He, Songlin Zhao +2Neural Network Interpretability Multimodal Graph Learning
Dept. of Mathematics & Operational Research University of Mons, Mons, Belgium · Dept. of Computer Science & Engineering Lehigh University, Bethlehem, PA, USA · Dept. of Industrial & Systems Engineering Lehigh University, Bethlehem, PA, USA
Machine Learning-Driven Multimodal Spectroscopic Liquid Biopsy for Early Multicancer Detection
May 13, 2026Alejandro Leonardo García Navarro, Javier Cachón Ortiz, Javier González Colsa +2Multimodal Classification Oncology
Signal Processing Group Gregorio Marañón Health Research Institute Madrid, Spain · Amber Health Solutions Madrid, Spain
StampFormer: A Physics-Guided Material-Geometry-Coupled Multimodal Model for Rapid Prediction of Physical Fields in Sheet Metal Stamping
May 13, 2026Jiajie Luo, Mohamed Mohamed, Osama Hassan +8Neural Surrogate Modeling Surrogate Modeling
Multitask Multimodal Fusion with Tabular Foundation Models for Peak and Durability Prediction of Pertussis Booster Response
May 13, 2026Divya SitaniMulti-Task Learning Clinical Prediction
Berlin, Germany
Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion
May 12, 2026Hassan Keshvarikhojasteh, Josien P. W. Pluim, Mitko VetaRepresentation Learning Cancer Survival Prediction
Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
Resilient Vision-Tabular Multimodal Learning under Modality Missingness
May 12, 2026Camillo Maria Caruso, Valerio Guarrasi, Paolo SodaMissing-Modality Learning Multimodal Robustness
Research Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Roma, Italy, Europe · Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Umeå, Sweden, Europe
Towards Visually Grounded Multimodal Summarization via Cross-Modal Transformer and Gated Attention
May 12, 2026Abid Ali, Diego Molla-Aliod, Usman NaseemCross-Modal Alignment Determinantal Point Process
School of Computing, Macquarie University, Sydney, Australia
MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation
May 11, 2026Ziyi Wang, Xianping Ma, Ziyao Wang +2Remote Sensing Image Understanding Remote Sensing Image Captioning
The Chinese University of Hong Kong (Shenzhen) · Southwest Jiaotong University
Simultaneous Long-tailed Recognition and Multi-modal Fusion for Highly Imbalanced Multi-modal Data
May 11, 2026Heegeon Yoon, Heeyoung KimClass-Imbalanced Learning Long-Tailed Classification
Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology (KAIST), 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Republic of Korea
Adaptive Context Matters: Towards Provable Multi-Modality Guidance for Super-Resolution
May 11, 2026Jinyi Luo, Minghao Liu, Yifan Li +2Mixture of Experts Multimodal Fusion
Wangxuan Institute of Computer Technology, Peking University
Fashion130K: An E-commerce Fashion Dataset for Outfit Generation with Unified Multi-modal Condition
May 11, 2026Yu He, Ting Zhu, Yichun Liu +6Conditional Image Generation Cross-Modal Alignment
Hyperbolic Distillation: Geometry-Guided Cross-Modal Transfer for Robust 3D Object Detection
May 11, 2026Kanglin Ning, Wenrui Li, Houde Quan +3Cross-Modal Knowledge Distillation 3D Object Detection
Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China · Suzhou Research Institute of HIT
M $^2$ FedAQI: Multimodal Federated Learning for Air Quality Prediction on Heterogeneous Edge Devices
May 10, 2026Manjil Nepal, Kimsie Phan, Tamoghna Ojha +2Multimodal Federated Learning Air Quality Prediction
Department of Computer Science and Engineering, SRM University-AP, India · Department of Mathematics and Computing, IIT (ISM) Dhanbad, India · Department of Mathematics and the Department of Computer Science, University of Central Florida, USA
Cross-Source Supervision for Bone Infection Segmentation in Dual-Modality PET-CT
May 10, 2026Zonglin Yang, Xiaolei Diao, Jishizhan Chen +5Cross-Modality Medical Image Segmentation PET/CT Imaging
Shanghai Maritime University Shanghai, CN · University College London London, UK · Shanghai Sixth People’s Hospital Shanghai, CN +2
Learning predictive models for combinations of heterogeneous proteomic data sources
May 9, 2026Michal Valko, Richard Pelikan, Miloš HauskrechtMedical Diagnosis Multimodal Classification
Computer Science Department University of Pittsburgh, Pittsburgh, Pennsylvania
MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment
May 9, 2026Qiqi Li, Pengfei Wang, Hongyu Chen +1Multimodal Fusion Action Quality Assessment
Qiushi Academy for Advanced Studies (QAAS), Zhejiang University, Hangzhou, Zhejiang, 310007, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang, 310007, China · School of Software Technology, Zhejiang University, Hangzhou, Zhejiang, 310007, China +3
MicroFuse: Protein-to-Genome Expert Fusion for Microbial Operon Reasoning
May 9, 2026Seungik ChoGenomics Multimodal Fusion
Department of Physics and Astronomy, Rice University, Texas, USA.
EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing
May 9, 2026Huilai Li, Xiaomeng Di, Ying Xing +3Audio-Visual Understanding Weakly Supervised Learning
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications, Beijing 100876, China
EyeCue: Driver Cognitive Distraction Detection via Gaze-Empowered Egocentric Video Understanding
May 8, 2026Lang Zhang, JinYi Yoon, Matthew Corbett +2Egocentric Video Understanding Multimodal Fusion
Virginia Tech · Inha University · Army Cyber Institute at West Point
UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition
May 8, 2026Shuai Zhang, Zhecheng Shi, Zhuxiao Li +4Disentangled Representation Learning Multimodal Semantic Segmentation
1HKUST(GZ) · 2HKUST
PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition
May 8, 2026Yuchen He, Jing ZhangAudio-Visual Understanding Referring Expression Segmentation
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
DPG-CD: Depth-Prior-Guided Cross-Modal Joint 2D-3D Change Detection
May 8, 2026Luqi Zhang, Zhen Dong, Bisheng YangRemote Sensing Image Understanding Remote Sensing Change Detection
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, No.129, Luoyu Road, Wuhan, Hubei, 430079, China
Multimodal Emotion Recognition via Causal-Diffusion Bridge (Affect-Diff)
May 7, 2026Ankit SanjyalClass-Imbalanced Learning Multimodal Emotion Recognition
Department of Computer Science – Data Science Fordham University New York, USA
Enhancing Eye Movement Biometrics for User Authentication via Continuous Gaze Offset Score Fusion
May 7, 2026Hashim Aziz, Mehedi Hasan Raju, Oleg V. KomogortsevBiometric Identification Eye Tracking
Texas State University · San Marcos, Texas, USA
PRISM: Iterative Cross-Modal Posterior Refinement for Dynamic Text-Attributed Graphs
May 7, 2026Trimble Chang, Yihang Liu, Mingjing Han +1Cross-Modal Representation Learning Temporal Link Prediction
College of Artificial Intelligence Nankai University
Fusion in Your Way: Aligning Image Fusion with Heterogeneous Demands via Direct Preference Optimization
May 7, 2026Weijian Su, Songqian Zhang, Yuqi Han +3Diffusion Model Alignment Multi-Source Image Fusion
School of Computer Science and Technology, Dalian University of Technology · Key Laboratory of Social Computing and Cognitive Intelligence (Dalian University of Technology), Ministry of Education · Institute of Image Processing and Understanding, North Minzu University
EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation
May 7, 2026Ziheng Xi, Jiayi Yu, Yitao Wang +3EMG-Based Gesture Recognition 3D Hand Pose Estimation
Department of Automation, Tsinghua University
Adaptive Physical-Facial Representation Fusion via Subject-Invariant Cross-Modal Prompt Tuning for Video-Based Emotion Recognition
May 7, 2026Xiwen Luo, Jia Li, Rencheng Song +2Multimodal Emotion Recognition Invariant Representation Learning
Department of Biomedical Engineering, and Anhui Province Key Laboratory of Measuring Theory and Precision Instrument, Hefei University of Technology, Hefei, 230009 China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601 China

May 13, 2026J/K move · Enter open · S save

Uncovering Trajectory and Topological Signatures in Multimodal Pediatric Sleep Embeddings

Scott Ye, Harlin Lee

Department of Radiology, University of California, San Francisco · School of Data Science and Society, University of North Carolina at Chapel Hill

Multimodal Embedding Clinical Prediction Multimodal Fusion

Open paper AskPDF arXiv

While generative models have shown promise in pediatric sleep analysis, the latent structure of their multimodal embeddings remains poorly understood. This work investigates session-wide diagnostic information contained in the sequences of 30-second pediatric PSG epochs embedded by a multimodal masked autoencoder. We test whether augmenting embeddings with PHATE-derived per-epoch coordinates and whole-night movement descriptors, persistent homology summaries of the embedding cloud, and EHR yields task-relevant signals. Simple linear and MLP models, chosen for interpretability rather than state-of-the-art performance, show that geometric, topological, and clinical features each provide complementary gains. For binary predictions, feature importance is task-dependent, and more expressive late-fusion models generally perform better, with AUPRC improving from 0.26 to 0.34 for desaturation, 0.31 to 0.48 for EEG arousal, 0.09 to 0.22 for hypopnea, and 0.05 to 0.14 for apnea. We also report Brier score and Expected Calibration Error, where the full fusion model yields the best calibration across all four binary tasks. Our study reveals that latent geometry/topology and EHR offer complementary, interpretable signals beyond embeddings, improving calibration and robustness under extreme imbalance.