PDMP: Rethinking Balanced Multimodal Learning via Performance-Dominant Modality Prioritization
Authors: Shicai Wei, Chunbo Luo, Qiang Zhu, Yang Luo
Organizations: Laboratory of Intelligent Collaborative Computing, University of Electronic Science and Technology of China. · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Sichuan, China. · Peng Cheng Laboratory, Shenzhen, China.
Multimodal learning has attracted increasing attention due to its practicality. However, it often suffers from insufficient optimization, where the multimodal model underperforms even compared to its unimodal counterparts. Existing methods attribute this problem to the imbalanced learning between modalities and solve it by gradient modulation. This paper argues that balanced learning is not the optimal setting for multimodal learning. On the contrary, imbalanced learning driven by the performance-dominant modality that has superior unimodal performance can contribute to better multimodal performance. And the under-optimization problem is caused by insufficient learning of the performance-dominant modality. To this end, we propose the Performance-Dominant Modality Prioritization (PDMP) strategy to assist multimodal learning. Specifically, PDMP firstly mines the performance-dominant modality via the performance ranking of the independently trained unimodal model. Then PDMP introduces asymmetric coefficients to modulate the gradients of each modality, enabling the performance-dominant modality to dominate the optimization. Since PDMP only relies on the unimodal performance ranking, it is independent of the structures and fusion methods of the multimodal model and has great potential for practical scenarios. Finally, extensive experiments on various datasets validate the superiority of PDMP.
Figures & tables
Fig. 1: The visualization on two audio-visual datasets: CREMA-D and AVE. (a) and (b) present the performance curves of the multimodal models under different values of w . w is the optimization dependency coefficient (see Eq. 5 ), which measures the relative contribution of the audio and visual modalities during training. ‘vanilla’ means the default w without any gradient modulation. (c) and (d) present the unimodal performance curves.
Fig. 2: Illustration of back-propagation processing for balanced learning and the proposed performance-dominant modality prioritization. Balanced learning mines the optimization-dominant modality and aligns its gradient go with the gradient gr of the remaining modality. In contrast, performance-dominant modality prioritization mines the performance-dominant modality and makes its gradient gp larger than the gradient gr of the remaining modality, guiding it to dominate the optimization.
Fig. 3: The visualization on the CREMA-D datasets. (a) presents the performance of the audio and visual branches in the multimodal model. (b) presents the performance of the audio and visual modality in the unimodal model.
Dataset
CREMA-D
KS
AVE
Methods
Acc
macro F1
Acc
macro F1
Acc
macro F1
Audio-only
57.27
57.89
48.67
48.89
62.16
58.54
Visual-only
62.17
62.78
52.36
52.67
31.40
29.87
Concatenation
58.83
59.43
64.97
65.21
66.15
62.46
Grad-Blending
68.81
69.34
67.31
67.68
67.40
63.87
OGM-GE
64.34
64.93
66.35
66.76
65.62
62.97
TABLE I: Comparison with existing modulation strategies on CREMA-D, Kinetics-Sounds, and AVE datasets. The proposed PDMP achieves the best performance. Bold and underline mean the best and second-best results, respectively.
Dataset
CEFA
UCF101
Methods
Acc
macro F1
Acc
macro F1
Concatenation
64.91
65.26
80.41
79.40
Grad-Blending
65.68
66.11
81.73
80.84
OGM-GE
66.33
66.76
81.15
80.36
AGM
67.65
67.98
81.55
80.36
PMR
68.38
68.89
81.36
80.37
TABLE II: (Left) : Comparison with imbalanced multimodal learning methods on CEFA dataset with three modalities. (Right) : Comparison with imbalanced multimodal learning methods on the UCF101 dataset with 101 categories. Bold and underline mean the best and second-best results, respectively.
Dataset
CREMA-D
AVE
KS
Method
Acc
Acc
Acc
Audio-only
57.27
62.16
48.67
Visual-only
62.17
31.40
52.36
Concatenation
58.83
66.15
64.97
Summation
62.24
67.25
64.53
Film
56.21
59.18
57.35
TABLE III: Performance on CREMA-D, AVE and KS datasets with various fusion methods. Combined with PDMP, conventional fusion methods consistently gain considerable improvement. † indicates PDMP is applied.
Fig. 4: Visualization of optimization dependency coefficient w on CREMA-D (a) and AVE (b) dataset, respectively. Here, s0 and s1 in Eq. 5 represent the logit output of audio and visual modalities, respectively. ‘Vanilla’ denotes the conventional multimodal model trained without PDMP.‘PDMP’ denotes the model trained with PDMP, respectively.
Dataset
CREMA-D
AVE
KS
Method
Acc
Acc
Acc
SE-Fusion
61.22
58.03
59.15
SE-Fusion†
67.86
64.87
65.27
mmFormer
59.92
60.14
64.26
mmFormer†
68.37
67.49
68.18
TABLE IV: Performance on different datasets with CNN-based and transformer-based architectures, respectively. † indicates PDMP strategy is applied, which achieves better performance.
γp
1
5
10
15
20
25
CREMA-D
58.83
70.22
76.76
80.21
79.26
78.36
AVE
66.15
67.55
69.73
71.28
70.64
69.85
KS
64.97
68.39
72.18
70.92
69.66
68.77
TABLE V: Ablation experiments of γp on different datasets. The fusion method here is concatenation.
Scale
20%
40%
60%
80%
100%
Audio
26.9
31.8
36.4
44.6
48.7
Visual
22.7
36.7
43.2
49.7
52.6
TABLE VI: Experiments on KS with different subset scales. n% indicates the proportion of the training dataset. The fusion method here is concatenation.
Dataset
CREMA-D
AVE
KS
Learning Rate
Acc
Acc
Acc
0.001
58.83
66.15
64.97
0.015
63.45
69.16
68.13
0.001†
80.21
71.28
72.18
TABLE VII: Ablation experiments of learning rate on different datasets. † indicates PDMP is applied. γr is set as 15