Balancing Multimodal Domain Generalization via Gradient Modulation and Projection
Authors: Hongzhao Li, Guohao Shen, Shupan Li, Mingliang Xu, Muhammad Haris Khan
Organizations: School of Computer and Artificial Intelligence, Zhengzhou University · Engineering Research Center of Intelligent Swarm Systems, Ministry of Education · National Supercomputing Center in Zhengzhou · Shandong Bosuan Zhixin Information Technology Co., Ltd · Mohamed Bin Zayed University of Artificial Intelligence
Multimodal Domain Generalization (MMDG) leverages the complementary strengths of multiple modalities to enhance model generalization on unseen domains. A central challenge in multimodal learning is optimization imbalance, where modalities converge at different speeds during training. This imbalance leads to unequal gradient contributions, allowing some modalities to dominate the learning process while others lag behind. Existing balancing strategies typically regulate each modality's gradient contribution based on its classification performance on the source domain to alleviate this issue. However, relying solely on source-domain accuracy neglects a key insight in MMDG: modalities that excel on the source domain may generalize poorly to unseen domains, limiting cross-domain gains. To overcome this limitation, we propose Gradient Modulation Projection (GMP), a unified strategy that promotes balanced optimization in MMDG. GMP first decouples gradients associated with classification and domain-invariance objectives. It then modulates each modality's gradient based on semantic and domain confidence. Moreover, GMP dynamically adjusts gradient projections by tracking the relative strength of each task, mitigating conflicts between classification and domain-invariant learning within modality-specific encoders. Extensive experiments demonstrate that GMP achieves state-of-the-art performance and integrates flexibly with diverse MMDG methods, significantly improving generalization across multiple benchmarks.
Figures & tables
Figure 1: Performance of state of the art MMDG methods, RNA-Net ( Planamente et al. 2024 ) , MOOSA ( Dong, Chatzi, and Fink 2024 ) , SimMMDG ( Dong et al. 2023 ) , and CMRF ( Fan et al. 2024 ) , evaluated with and without our proposed GMP strategy.
Domain
Method
Video
Audio
Video-Audio
Source
Uni-video
74.55
-
-
Uni-audio
-
52.35
-
Base
70.28
48.63
76.31
OGM-GE
72.66
50.33
77.69
Grad-Blending
72.31
50.62
78.03
GMP(Ours)
72.69
50.52
78.05
Table 1: Performance comparison on source and target sets using the EPIC-Kitchens dataset ( Kay et al. 2017 ) . Source results are averaged over validation sets (of the seen domains), and target results are averaged over test sets from the corresponding unseen domains. Uni-video and Uni-audio denote independently trained single-modality models.
Figure 2: The overall training strategy of our proposed GMP.
Method
EPIC-Kitchens
HAC
Base
55.06
61.86
Grad-Blending
55.49
62.66
OGM-GE
55.71
62.83
AGM
55.39
62.16
CGGM
55.30
62.80
DRB
54.88
61.92
Table 2: Comparison of our proposed GMP strategy with existing gradient strategies. All methods use concatenation fusion as the base.
Method
EPIC-Kitchens
HAC
RNA-Net
60.31
62.88
MOOSA
60.29
67.39
SimMMDG
60.81
67.82
CMRF
60.60
67.66
RNA-Net†
60.92
65.21
MOOSA†
61.79
68.75
Table 3: Integration with existing MMDG methods. Methods marked with † indicate integration of GMP.
Method
EPIC-Kitchens
HAC
Base
55.06
61.86
IGDM-only
55.98
63.05
CAGP-only
55.34
63.41
Full
57.36
64.91
Unified Modulation
54.97
62.50
w/o ktm
55.19
62.33
Table 4: Ablation study results demonstrating the effectiveness of individual components and strategies.
Figure 3: t-SNE plots on the HAC dataset (target domain A) compare the Base and GMP methods in classification and domain invariance.
Method
EPIC-Kitchens
HAC
Activation-based
52.23
57.49
Concatenation
55.06
61.86
Summation
52.94
58.51
FiLM
53.16
59.35
Activation-based†
55.69
61.33
Concatenation†
57.36
64.91
Table 5: Combined with GMP, conventional fusion methods consistently gain considerable improvement. † indicates GMP strategy is applied.
Figure 4: Hyperparameter sensitivity analysis of GMP.
Figure 5: Discrepancy ratio trajectories ( ρtm , σtm ) on EPIC-Kitchens. GMP mitigates cross-modal imbalance by driving ratios toward 1, achieving balanced semantic and domain optimization.
Deploying multimodal models in real-world scenarios requires generalization to new environments where recording conditions differ from training, a challenge known as multimodal domain generalization (MMDG). Standard architectures employ separate encoders for each modality and a fusion module, training the system end-to-end by optimizing on the fused features. In this paper, we identify that such joint optimization causes encoders to exploit cross-modal co-occurrences, statistical relationships between modalities that arise from source-specific recording conditions, rather than learning domain-invariant features. We term this failure mode Fusion Overfitting. To address this, we propose Modality-Entropy Regularization for Domain Generalization (MER-DG), which maximizes the entropy of each encoder's feature distribution to preserve feature diversity. MER-DG is architecture-agnostic and integrates into existing multimodal frameworks as an additive loss term. Extensive experiments on EPIC-Kitchens and HAC benchmarks demonstrate average improvements of approximately 5% over standard fusion and approximately 2% over state-of-the-art methods.
Yavuz Yarici, Ghassan AlRegib
1OLIVES at the Center for Signal and Information Processing CSIP, School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA.
Multimodal learning often suffers from modality imbalance, where the joint optimization process is dominated by a single modality. Existing methods typically estimate modality imbalance from score disparities derived from prediction uncertainty or optimization statistics. However, due to distinct prediction uncertainty and learning dynamics across modalities, direct comparison of such scores may misinterpret intrinsic modality differences as progress gaps, leading to biased imbalance estimation. In this paper, we propose Function-Space Guided Multimodal Optimization (FGMO), which leverages a function-space progress signal to assess modality-wise optimization progress and coordinate optimization across modalities to alleviate modality imbalance. Specifically, we introduce Functional Progress Estimation (FPE) to measure each modality's update-induced function-space response and calibrate it against a loss-aligned unimodal reference, producing a comparable progress signal. Based on this signal, Functional Response Control (FRC) redistributes modality-level function-space budgets and realizes the target responses through tensor-wise learning-rate adjustment. Theoretical analysis establishes a one-step target-contraction property of FRC under bounded controller-state mismatch, and extensive experiments demonstrate the effectiveness of FGMO across multiple multimodal benchmarks.
Multimodal learning often suffers from modality imbalance, where modalities that converge faster dominate optimization while others remain undertrained. Existing approaches typically mitigate this issue by strengthening the weak modality or adjusting optimization gradients. However, such strategies mainly compensate for optimization rate discrepancies, often at the expense of the strong modality's optimization capacity, without analyzing how these discrepancies arise at the modality level. Based on theoretical insights and empirical observations, we argue that the discrepancy of learning pace arises from differences in the mapping difficulty between modality-specific feature space and the shared label space. To address this issue, we propose Balanced Multimodal Label Reshaping (BMLR), the first method that promotes multimodal balance from the label-side design. BMLR reshapes the cross-modal label space to equalize mapping difficulty across modalities, thereby facilitating modality interaction and injecting richer inter-class information into each modality. Extensive experiments across multiple architectures demonstrate that BMLR consistently improves multimodal performance and exhibits strong compatibility with diverse model designs. The source code will be released soon.