Multimodal Fusion

Follow this topic Ask about this topic

Multimodal fusion studies how to combine inputs such as images, text, audio, medical scans, sensor streams, and structured records so models can use their complementary information. Common approaches include early or late fusion, cross-attention and contrastive alignment, and adaptive gates or mixtures of experts that weight modalities by task, sample, or estimated reliability. Methods are tested across settings such as BraTS medical segmentation, CMU-MOSI and CMU-MOSEI sentiment analysis, RGB-thermal detection, and clinical prediction, with growing attention to missing, noisy, or misaligned inputs. A central open tension is that additional modalities can help when they add useful evidence, but can also introduce noise or let a dominant modality suppress weaker ones; robust gains and generalization beyond benchmark conditions remain difficult to establish.

Latest papers

772 papers73 papers in the last four weeks, down 12% on the four weeks before. 0.4% of all new papers.

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning
May 31, 2026Zhiqiang Zhou, Xuezhen XieCross-Attention Multimodal Fusion
Hunan Chemical Industry Vocational and Technical College
Reusing Fusion-Time Spectral Reliability for Adaptive Fusion and Expert Routing in RGB-Infrared Object Detection
May 31, 2026Yefeng WuMultimodal Robustness Adaptive Model Routing
Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition
May 31, 2026Wanlong Fang, Tianle Zhang, Wen Tao +1Multimodal Understanding Multimodal Grounding
College of Computing and Data Science, Nanyang Technological University, Singapore · Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore · Centre of AI in Medicine (C-AIM), Nanyang Technological University, Singapore
Enhancing Protein-Protein Interaction Prediction with Hierarchical Motif-based Multimodal Protein Embedding
May 30, 2026Zaifei Yang, Samuel Ping-Man Choi, James KwokHierarchical Representation Learning Multimodal Embedding
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong. · Lee Shau Kee School of Business and Administration, The Hong Kong Metropolitan University, Ho Man Tin, Kowloon, Hong Kong.
UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment
May 29, 2026Zheng Wang, Shuo Wang, Junhong WangUnsupervised Domain Adaptation Multimodal Emotion Recognition
Institute of Advanced Technology, University of Science and Technology of China, Hefei, 230026, China · Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230026, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230026, China
Geometry-based Schrödinger Bridges for Trustworthy Multimodal Fusion
May 29, 2026Jiayu Xiong, Jing Wang, Qi Zhang +2Multimodal Robustness Multimodal Fusion
Department of Computer Science and Techonology, Huaqiao University · Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Huaqiao University · Tongji University +1
IAF-Net: Illumination-Adaptive Fusion for Low-Light Urban Road Segmentation
May 29, 2026Bingtao Wang, Daojie Peng, Fulong Ma +2Autonomous Driving Perception Multimodal Fusion
The Shandong University · The Hong Kong University of Science and Technology (Guangzhou)
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
May 28, 2026Feng Han, Zhixiong Zhang, Zheming Liang +2Cross-Modal Learning VLM Robustness
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
Multimodal Music Recommendation System using LLMs
May 28, 2026Srikar Prabhas Kandagatla, Sreehitha R. Narayana, Chandana Magapu +6Sequential Recommendation Recommender Systems
University of Massachusetts Amherst, Massachusetts, USA · Dolby Laboratories Atlanta, Georgia, USA · Adobe Research San Jose, California, USA +2
MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding
May 28, 2026Abdulkadir Gokce, Badr AlKhamissi, Martin SchrimpfNeuroimaging Neural Encoding
NeuroAI Lab, EPFL
MMTM: Tri-Modal Topic Modeling for Long-Form Video via Similarity-Gated Fusion
May 28, 2026Ali Abusaleh, Bhuvanesh Verma, Alexander MehlerTopic Modeling Long-Video Understanding
Text Technology Lab (TTLab), Goethe University Frankfurt
Lightweight Complementary-Cue Fusion for Robust Video Face Forgery Detection
May 27, 2026Sunghwan Baek, Tariq Anwaar, Karanveer Singh +1Frequency-Domain Feature Learning Deepfake Detection
Carnegie Mellon University
Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach
May 27, 2026Anderson AugusmaMultimodal Emotion Recognition Privacy-Preserving ML
Thèse de Doctorat de l’Université Grenoble Alpes
A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis
May 27, 2026Jianheng Dai, Jiazhang Liang, Sijie MaiModality Imbalance Gradient Interference
School of Computer Science, South China Normal University, Guangzhou, Guangdong, China
GS-FUSE: Granger-Supervised Gated Fusion and Multi-Granularity Alignment for Event-Driven Financial Forecasting
May 27, 2026Yang Zhang, En Chun, Ziyun Mao +2Financial Forecasting Time Series Forecasting
Southwestern University of Finance and Economics Chengdu, China
Particle-Lund Multimodality in Jet Taggers
May 26, 2026Loukas Gouskos, Benedikt MaierUnified Multimodal Models Jet Tagging
Brown University, Providence, USA · Imperial College of Science, Technology and Medicine, London, United Kingdom
PolyFusionAgent: A Multimodal Foundation Model and Autonomous AI Assistant for Polymer Property Prediction and Inverse Design
May 26, 2026Manpreet Kaur, Xingying Zhang, Qian LiuAI Agents for Scientific Discovery Materials Property Prediction
Department of Applied Computer Science, The University of Winnipeg, 515 Portage Avenue, Winnipeg, MB R3B 2E9, Canada. · Department of Mechanical Engineering, University of Manitoba, Winnipeg, MB R3T 2N2, Canada.
Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery
May 25, 2026Niels Sombekke, Rob G. J. Wijnhoven, Martin R. OswaldRemote Sensing Image Understanding Multi-Source Image Fusion
University of Amsterdam (UvA), Amsterdam, The Netherlands · Spotr, The Hague, The Netherlands
Prospective evaluation of multimodal respiratory failure prediction: Do chest X-rays improve performance beyond EHR signals?
May 25, 2026Xiaolei Lu, Shamim NematiClinical Outcome Prediction Multimodal Fusion
Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos
May 25, 2026Bonan Ding, Umair Nawaz, Ufaq Khan +5Multimodal Understanding Cross-Modal Learning
Mohamed bin Zayed University of Artificial Intelligence · Tianjin University · Chongqing University +1
STaT: Resolving Shape Distortion in Non-Stationary Time Series via Tri-Modal Synergy
May 25, 2026Hui Cheng, Jinsheng Guo, Zhenhao Weng +2Non-Stationary Time Series Forecasting Multimodal Fusion
Hefei University of Technology Hefei, China
DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion
May 25, 2026Xingyuan Li, Haoyuan Xu, Shulin Li +3Temporal Consistency Multimodal Fusion
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · School of Software Technology & DUT-RU International School of ISE, Dalian University of Technology, Dalian, China · College of Information Science and Technology, Dalian Maritime University, Dalian, China
A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning
May 25, 2026Loukas Ilias, Dimitris AskounisSpeech-Based Dementia Detection Speech Processing
Decision Support Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece
Does Seeing More Mean Knowing More? Mono-Anchored Advantage Normalization for Multi-Source Visual Reasoning
May 25, 2026Fanhu Zeng, Zhicong Luo, Zefan Wang +3Visual Reasoning Multimodal Reasoning
Tsinghua University · Northwest Polytechnical University · Beijing Jiaotong University
MTLLFM: Multimodal-Temporal Laughter Localization: UR-FUNNY-Temporal and SMILE-Temporal Benchmarks with an Adaptive Multimodal Fusion Model
May 25, 2026Eyal Hanania, Nadav Kirsch, Daniel Arkushin +4Audio-Visual Understanding Weakly Supervised Learning
WSC-Sports
Toward Native Multimodal Modeling: A Roadmap
May 25, 2026Siyu An, Junru Lu, Junnan Dong +18Unified Multimodal Models Multimodal Generation
Tencent Youtu Lab
Multimodality Stacking with Blockwise missing values and application to the PIONeeR biomarkers study for prediction of resistance to immunotherapy
May 24, 2026Mohamed Boussena, Florence Monville, Jacques Fieschi-Meric +8Missing-Modality Learning Cancer Survival Prediction
Inria – Inserm team COMPO, COMPutational pharmacology and clinical Oncology, Centre Inria Sophia Antipolis - M´editerran´ee, Centre de Recherches en Canc´erologie de Marseille, Inserm U1068, CNRS UMR7258, Institut Paoli-Calmettes, Pharmacy faculty, Aix-Marseille University · Veracyte SAS, Marseille, France · Assistance Publique-Hˆopitaux de Marseille (APHM), Marseille, France +4
AdaFuse-Det: Adaptive Cross-Modal Fusion of Event Cameras for Robust Object Detection in Low-Light RGB Imagery
May 23, 2026Raju Imandi, Chethana B, Bharatesh Chakravarthi +3Event-Based Object Detection Object Detection
SRM University AP, India · Aptiv, Bengaluru, India · Arizona State University, USA +2
Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs
May 23, 2026Hao Yan, Xuanru Wang, Jun Yin +3Graph Neural Networks Multimodal Graph Learning
School of Computer Science and Engineering, Central South University, Changsha, China · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China · School of Information and Communication Technology, Griffith University, Brisbane, Australia
PoseRefer: Pathway-Local Parameters for Semantically Grounded Reference Resolution
May 23, 2026Anna DeichlerMultimodal Grounding Multimodal Fusion
KTH Royal Institute of Technology, Stockholm, Sweden

May 31, 2026J/K move · Enter open · S save

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

Zhiqiang Zhou, Xuezhen Xie

Hunan Chemical Industry Vocational and Technical College

Cross-Attention Multimodal Fusion Feature Fusion Multimodal Understanding

Open paper AskPDF arXiv

The choice between cross-attention and concatenation for multimodal fusion remains governed by practitioner intuition rather than principled understanding. In this paper, we demonstrate that feature alignment quality, not data scale alone, is the primary determinant of which fusion strategy excels. Through controlled experiments on Flickr8k using two feature extraction backbones (ResNet18 and CLIP ViT-B/32), we show that concatenation outperforms cross-attention by 4.1-5.1 percentage points across all tested scales (2048-16384 samples) when features are pre-aligned by a vision-language pretraining objective. We provide a theoretical explanation grounded in sample complexity analysis: concatenation requires O(d_v + d_t) samples to learn its fusion projection, while cross-attention requires O(d_v * d_t) samples to learn bilinear attention weights, over 256 times as many for 512-dimensional CLIP features. When features are already aligned, the approximation error gap between the two methods vanishes, and concatenation's sample efficiency dominates at all practical dataset sizes. An alignment degradation study confirms a monotonic trend: as feature alignment degrades, concatenation's advantage grows from 1.3% to 2.8%. These findings provide a principled decision framework for fusion method selection in multimodal systems, with direct implications for the design of Multimodal Large Language Models.