Multimodal Fusion
Overview
Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.
Latest papers
810 papers50 papers in the last four weeks, down 43% on the four weeks before. 0.4% of all new papers.
DiFuse: Fusing Native Diffusion Representations for Spatial Reasoning in Unified Multimodal Models
MOSAIC-Flow: Orthogonal-Axis and Status-Conditioned Fusion for Missing-Sequence Completion in Brain Tumor MRI
MemoryFusion: Cross-Temporal Memory Learning for Multimodal Video Fusion
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
Alpha-TCM: A Nine-Layer Framework for TCM-Guided Multimodal Inference and Structured-Prior Auditing
Brain-Behavior Modeling from Multimodal MRI using Opportunistic Pretraining and Common-Cohort Adaptation
3D-SceneQ: Empowering 3D LLM With Query-Guided Adaptive Pruning and Multi-Modal Feature Enhancement
MM-Net: A Multimodal Multi-Task Network for Sleep Staging and Sleep Apnea Assessment After Ischemic Stroke
When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis
When Fusion Order Matters: Insights on Progressive Multimodal Representation Learning for Cross-modal Retrieval
Efficient Multi-modal Video Fusion Network with Dynamic Region-Guided Temporal Interaction
I-KAN: Interpretable Intra- and Inter-Modal Multi-Pathway Representation Learning for Multimodal Sentiment Analysis
ICGF-CXRMate: Image-Conditioned Clinical Context Compression and Gated Fusion for Chest X-ray Report Generation
AGF-Net: A Robust Multimodal Emotion Recognition Model for Asynchronous Audio-Visual Signals Based on Predictive Dynamic Fusion with Negative Learning
Multi-Scale Latent Temporal Compression Attention and Video-Text Cross Attention Fusion for Video Action Detection
GAMF-Net: Grouped Availability-Aware Multimodal Fusion for Smartphone Skin Lesion Classification
HypTI: Hyperbolic Geometry-Aware Multimodal Feature Fusion Framework for Dementia Detection Based on Tabular-Image Data
Revealing Inter-Channel Modality Discriminability: Channel Selective Interaction Fusion Network for Optical-SAR Object Detection
Progressive multi-scale tumor-state memory for incomplete multimodal brain tumor segmentation
RGB--Event Object Detection Across Operating Frequencies via Reliability-Supervised Routing and Asymmetric-Resolution Fusion
CheXFuse: Guided Differential Joint Attention Fusion for Robust Multimodal Chest X-Ray Classification in Data-Scarce Medical Settings
SZED at ArGuard Shared Tasks: Multi-Task Label-Attention and Cross-Modal Fusion for Arabic Hateful Meme Detection
Shared-Private Cross-Modal Discrepancy Learning for Imbalanced Multi-Source Audio-Visual Deepfake Detection
[CLS]-augmented audio and visual tokens into a shared stream and source-specific private streams before the first MSCT block, while retaining its internal multi-scale self-attention (MSSA), differential cross-modal attention (DCA), and feed-forward operations. The shared stream models cross-source reusable evidence, while the private streams retain source-dependent residual variations. A domain-aware gated residual fusion module injects selected private evidence into the shared representation for the main fused prediction. Experiments under two controlled source-composition protocols show improved macro and worst-source AUC compared with the MRDF and MSCT baselines. These results support shared-private cross-modal discrepancy learning in the evaluated two-source setting with known source identity.