Unbiased Dynamic Multimodal Fusion

Shicai Wei; Kaijie Zhang; Luyi Chen; Tao He; Guiduo Duan

cs.CVMar 20, 2026

Unbiased Dynamic Multimodal Fusion

Authors: Shicai Wei, Kaijie Zhang, Luyi Chen, Tao He, Guiduo Duan

Organizations: University of Electronic Science and Technology of China

Abstract

Traditional multimodal methods often assume static modality quality, which limits their adaptability in dynamic real-world scenarios. Thus, dynamical multimodal methods are proposed to assess modality quality and adjust their contribution accordingly. However, they typically rely on empirical metrics, failing to measure the modality quality when noise levels are extremely low or high. Moreover, existing methods usually assume that the initial contribution of each modality is the same, neglecting the intrinsic modality dependency bias. As a result, the modality hard to learn would be doubly penalized, and the performance of dynamical fusion could be inferior to that of static fusion. To address these challenges, we propose the Unbiased Dynamic Multimodal Learning (UDML) framework. Specifically, we introduce a noise-aware uncertainty estimator that adds controlled noise to the modality data and predicts its intensity from the modality feature. This forces the model to learn a clear correspondence between feature corruption and noise level, allowing accurate uncertainty measure across both low- and high-noise conditions. Furthermore, we quantify the inherent modality reliance bias within multimodal networks via modality dropout and incorporate it into the weighting mechanism. This eliminates the dual suppression effect on the hard-to-learn modality. Extensive experiments across diverse multimodal benchmark tasks validate the effectiveness, versatility, and generalizability of the proposed UDML. The code is available at https://github.com/shicaiwei123/UDML.

Figures & tables

Figure 1: Visualization of dynamic multimodal methods for audio-visual classification on the CREMA-D dataset. (a) Visual weighting coefficients obtained using different uncertainty estimation methods, such as energy score (ES) [ 47 ] and probabilistic embedding (PE) [ 8 ] , and the proposed UDML, as varying levels of noise ( $\sigma$ ) are injected into the visual modality. (b) Performance Comparison of different methods under static and dynamic weighting when noise ( $\sigma=5$ ) is injected into the visual modality.

Figure 2: The framework of the unbiased dynamic multimodal fusion. It consists of two parts: 1) noise-aware uncertainty estimator, which measures the modality quality; 2) modality-dependency calculator, which quantifies the model’s dependency on each modality.

Methods	MVSA-Single		CREMA-D		Kinetics-Sounds
Methods	Acc	F1	Acc	F1	Acc	F1
Late Fusion [ 26 ]	76.88	75.72	58.83	59.43	64.97	65.21
MMBT [ 49 ]	78.50	-	64.25	65.43	65.89	-
TMC [ 11 ]	76.06	74.55	65.43	66.10	66.12	66.51
ITIN [ 33 ]	75.19	74.97	-	-	-	-
MVCN [ 41 ]	76.06	74.55	-	-	-	-

Table 1: Performance comparison of different methods on multimodal classification tasks on MVSA-Single, CREMA-D, and Kinetics-Sounds datasets.

Methods	CMU-MOSI			CMU-MOSEI
Methods	Acc7	F1	Corr	Acc7	F1	Corr
MIB [ 51 ]	48.6	85.3	0.798	54.1	86.2	0.790
HMA [ 23 ]	45.3	85.6	0.782	52.8	85.4	0.787
MIM [ 10 ]	47.0	85.9	0.805	52.5	86.3	0.792
GCNet [ 4 ]	44.9	85.1	-	51.5	85.2	-
ConFEDE [ 44 ]	42.3	85.5	0.784	54.9	85.8	0.780

Table 2: Performance comparison of different methods for multimodal regression tasks on CMU-MOSI and CMU-MOSEI datasets.

Datasets	Methods	Clean	Salt		Gaussian
Datasets	Methods	Acc@ $\epsilon$ =0	Acc@ $\epsilon$ =5	Acc@ $\epsilon$ =10	Acc@ $\epsilon$ =5	Acc@ $\epsilon$ =10
MVSA-Single	Late fusion [ 26 ]	76.88	67.88	55.43	63.46	55.16
	MMBT [ 49 ]	78.50	74.07	51.26	71.99	55.35
	TMC [ 11 ]	74.88	68.02	56.62	66.72	60.36
	QMF [ 47 ]	78.07	73.90	60.41	73.85	61.28
	EAU [ 8 ]	79.15	74.81	61.04	73.89	62.04

Table 3: Performance comparison of different methods when 50% of the modalities suffer from salt and Gaussian noise, respectively. The mean and variance of the noise are 0 and

\epsilon

, respectively.

Dataset	MVSA	CREMA-D	Kinetics-Sounds
Settings	Acc	Acc	Acc
Full	77.87	67.53	66.36
- NUE	74.56	64.60	64.33
- MC	75.44	64.10	65.41
- POS	76.16	66.49	64.92

Table 4: Ablation of the UDML method when 50% of the modalities suffer from Gaussian noise (

\epsilon=5

). We report the result of complete UDML (Full), the UDML without noise-aware uncertainty estimator (-NUE), the UDML without Modality-dependency Calculator (-MC), and the UDML without Progressive Optimization Strategy(POS).

Datasets	MVSA	CREMA-D	Kinetics-Sounds
Structure	Acc	Acc	Acc
MMTM	78.86	51.86	56.70
MMTM †	85.68	56.88	62.72
mmFormer	79.69	59.69	64.72
mmFormer †	85.33	63.14	67.51

Table 5: Performance evaluation after integrating the UDML method into CNN-based and Transformer-based architectures. † indicates that the UDML is applied.

Figure 3: Visualization of dynamic multimodal fusion for audio-visual classification on the CREMA-D dataset as varying levels of noise ( $\sigma$ ) are injected into the visual modality.

Explore similar work

Dynamic Alignment and Calibration for Multimodal Learning
Oct 6, 2026Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu +1Multimodal Robustness Cross-Modal Alignment
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu,China · Tianyijiaotong Technology Ltd.,China · School of Computer Science and Technology, Hainan University, Haikou, China
General Incomplete Multimodal Learning via Dynamic Quality Perception
Jul 8, 2026Xiangyu Meng, Shicai WeiMissing-Modality Learning Multimodal Robustness
University of Electronic Science and Technology of China
CL-DMDF:Dynamic Multimodal Data Fusion Model Based on Contrastive Learning
Jun 1, 2026Dong Li, Lingling Zhang, Binghao Han +2Missing-Modality Learning Cross-Modal Attention
School of Information, Liaoning University, Shenyang, China · School of Computer Science and Engineering, Northeastern University, Shenyang, China

Oct 6, 2026J/K move · Enter open · S save

Dynamic Alignment and Calibration for Multimodal Learning

Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu, Xiaoshuang Shi

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu,China · Tianyijiaotong Technology Ltd.,China · School of Computer Science and Technology, Hainan University, Haikou, China

Multimodal Robustness Cross-Modal Alignment Uncertainty-Aware Multimodal Fusion Multimodal Contrastive Learning

Open paper AskPDF arXiv

Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise variations, potentially leading to unreasonable over-alignment; and (ii) confidence- or uncertainty-aware fusion methods often fail to adequately account for feature magnitude and confidence differences across modalities. For modality pairs with significant feature magnitude differences or small confidence gaps, it might be unreliable to strictly align fusion weights according to confidence. To address these issues, we propose an Alignment- and Calibration-driven Multimodal Learning framework (ACML). Specifically, ACML incorporates a dynamic cross-modal triplet alignment module, which enforces strong semantic consistency for high-confidence positive pairs while encouraging diverse representation learning between high- and low-confidence positive pairs according to their confidence gaps. Additionally, ACML introduces a difference-aware attention calibration strategy that adaptively adjusts attention regularization based on feature magnitude and confidence differences across modalities, thereby mitigating biases caused by unreasonable fusion constraints. Extensive experiments on multiple multimodal benchmark datasets demonstrate that ACML consistently achieves superior performance and robustness over recent state-of-the-art methods.

Abstract

Explore similar work

Dynamic Alignment and Calibration for Multimodal Learning

General Incomplete Multimodal Learning via Dynamic Quality Perception

CL-DMDF:Dynamic Multimodal Data Fusion Model Based on Contrastive Learning