Probe-Guided Gradient Balancing for Multimodal Learning

Vu Vo; Haytham M. Fayek; Thuy Nguyen

cs.CVNov 2, 2026

Probe-Guided Gradient Balancing for Multimodal Learning

Authors: Vu Vo, Haytham M. Fayek, Thuy Nguyen

Organizations: School of Science, Engineering & Technology, RMIT University · School of Computing Technologies, RMIT University

Abstract

Jointly trained multimodal networks frequently underutilize weaker modalities, as faster-learning modalities dominate the shared fusion objective and suppress gradients reaching slower ones. Most existing approaches throttle dominant modalities, whereas directly boosting weaker modalities often fails because monitoring signals derived from the joint loss remain dominated by stronger modalities. We propose Probe-Guided Gradient Boosting (PGGB), a gradient modulation approach that boosts weaker modalities while preserving the strength of dominant ones. PGGB attaches lightweight linear classifiers to stop-gradient features from each modality to estimate representation quality independently. The gap between these scores, termed the utilization gap, provides an imbalance signal used to adaptively scale weaker-modality gradients through bounded and smoothed modulation. PGGB is composable with throttling methods, as it leaves the loss function and fusion architecture unchanged. Experiments across eight benchmarks spanning four domains and 2-4 modalities demonstrate that PGGB outperforms state-of-the-art methods on highly imbalanced multimodal datasets while remaining competitive under low or negligible imbalance. We further establish bounded scaling, self-attenuation, and a descent bound for standard SGD under smoothness and bounded-variance assumptions. Code is available at https://github.com/VuVoCodes/PGGB.

Figures & tables

Figure 1: An Illustration of probe-guided gradient boosting on 2 modalities. Forward path (top): features

\mathbf{z}_{m}

are concatenated and classified via the shared fusion classifier

g

, producing

\mathcal{L}

. Probe monitoring (bottom): stop-gradient features

\operatorname{sg}[\mathbf{z}_{m}]

are evaluated by linear probes along solid per-modality probe-flow lines (audio in blue, visual in orange), yielding per-modality accuracies

\bar{P}_{m}

. The boost block computes the relative-gap weight

w_{m}

and a smoothed scale

\bar{s}_{m}

(EMA-smoothed, refreshed only at probe-evaluation events every

K

steps and held constant in between), clipped at

s_{\max}

. The actuation rail (thick green) applies

\bar{s}_{m}

to the per-modality gradient

\nabla_{\!\theta_{m}}\mathcal{L}

. Probes never backpropagate into encoders, so the monitoring signal is structurally decoupled from the fusion objective.

Method	CREMA-D	AVE	KS	CMU-MOSI	Twitter15	Sarcasm	CH-SIMS	BraTS 2021
Baseline	61.59 $\pm$ 0.80	86.54 $\pm$ 0.42	79.05 $\pm$ 0.40	72.42 $\pm$ 0.48	66.60 $\pm$ 0.57	82.40 $\pm$ 0.17	70.42 $\pm$ 0.29	85.77 $\pm$ 0.62
OGM-GE	69.14 $\pm$ 1.13	86.96 $\pm$ 0.71	77.25 $\pm$ 0.79	72.68 $\pm$ 0.89	66.33 $\pm$ 0.21	81.81 $\pm$ 0.23	72.47 $\pm$ 0.70	86.21 $\pm$ 0.17
MMPareto	65.51 $\pm$ 0.87	86.37 $\pm$ 0.33	78.21 $\pm$ 0.38	72.68 $\pm$ 0.79	66.81 $\pm$ 0.47	82.27 $\pm$ 0.12	70.20 $\pm$ 0.71	—
AGM	57.42 $\pm$ 0.73	84.42 $\pm$ 0.26	77.85 $\pm$ 0.30	72.50 $\pm$ 0.62	66.79 $\pm$ 0.47	82.32 $\pm$ 0.16	69.28 $\pm$ 0.67	—
G-Blend	61.10 $\pm$ 1.87	87.09 $\pm$ 0.29	77.75 $\pm$ 1.13	72.36 $\pm$ 0.66	66.66 $\pm$ 0.57	82.35 $\pm$ 0.26	70.15 $\pm$ 0.66	—
CGGM	50.22 $\pm$ 1.39	76.72 $\pm$ 0.42	73.18 $\pm$ 0.27	59.45 $\pm$ 0.36	62.41 $\pm$ 0.36	80.71 $\pm$ 0.27	68.05 $\pm$ 0.46	81.13 $\pm$ 1.34

Table 1: Main results across eight benchmarks. We report accuracy (%) for classification datasets and average Dice (%) for BraTS 2021. All values are mean

\pm

std over 5 seeds. Bold indicates the best result per dataset. Imbalance categorization is described in Section 4.1 . Baseline denotes joint training with unmodified gradients, not a published OGM-GE result. All methods are re-run under the matched pipeline of Section 4.1 , in which CREMA-D uses 3 frames. OGM-GE’s published 1-frame CREMA-D result (

61.9\%

) [ 25 ] is reproduced at

62.47\pm 1.42\%

in Table 2 .

Method	Accuracy (%)
Baseline	60.30 $\pm$ 0.70
PGGB ( $\alpha{=}0.5$ )	60.46 $\pm$ 0.85
OGM-GE only	62.47 $\pm$ 1.42
PGGB+OGM-GE ( $\alpha{=}0.5$ )	62.37 $\pm$ 0.57
PGGB+OGM-GE ( $\alpha{=}0.75$ )	62.69 $\pm$ 0.22

Table 2: Ablation on CREMA-D (1-frame). Accuracy (%), mean

\pm

std over 5 seeds. The upper row reports the baseline. Lower rows isolate single-mechanism interventions (boost only, OGM-GE only) and their combinations.

Regime	Trainable	Asymmetry	Adequate data	Compl.	$\boldsymbol{\Delta}$
CREMA-D	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$+9.86$ ***
AVE (pretrained)	$\checkmark$	$\checkmark$ (pretrain)	$\checkmark$	$\checkmark$	$+0.69$
AVE (from-scratch)	$\checkmark$	$\times$	$\times$ ( $\sim 4$ K)	$\checkmark$	$-4.56$ ***
Food101 (frozen)	$\times$	—	$\checkmark$	$\checkmark$	$-1.64$

Table 3: Operating conditions across four regimes. Columns: trainable encoders, modality asymmetry (pretraining or LR differences), adequate data-to-architecture ratio, and complementary modalities.

\Delta

: accuracy change (pp), PGGB+OGM-GE vs. baseline. Significance: ***

p<0.001

(Welch’s t-test, 5 seeds per condition). AVE (from-scratch) is the clearest failure case (

-4.56

pp), with the pretrained

\leftrightarrow

scratch contrast localizing sensitivity to the OGM-GE component. Per-regime analysis in Appendix B.3 .

Panel of Figure 2 — Figure 2: Operating-conditions ablation (5 seeds). (a) PGGB alone self-attenuates toward baseline when Table 3 conditions are violated, whereas PGGB+OGM-GE inherits OGM-GE’s regime-dependent failure modes (AVE from-scratch is the clearest case). (b) With data and architecture fixed, flipping initialization from pretrained to random shifts PGGB+OGM-GE from $+0.69$ to $-4.56$ pp ( $p{=}0.0001$ ) while PGGB alone stays positive, localizing sensitivity to the OGM-GE component.

Figure 3: Per-modality diagnostics on CREMA-D (5 seeds). (a) Post-hoc linear-probe accuracy

P_{m}^{\text{post}}

(Appendix B.13 ). Dotted line marks chance (

1/6

). AGM and MILES collapse the weak probe. (b) Utilization gap

|P_{a}^{\text{post}}-P_{v}^{\text{post}}|

. PGGB+OGM-GE attains

2.31

pp (

5.4\times

reduction vs. baseline gap). Error bars:

\pm 1

std.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Accuracy (%)
Baseline	59.20 $\pm$ 1.27
OGM only	63.74 $\pm$ 0.45
OPM only	65.43 $\pm$ 0.24
OPM+OGM	69.84 $\pm$ 1.47
PGGB+OGM-GE ( $\alpha{=}0.75$ )	71.77 $\pm$ 0.91

Appendix

Table 4: Comparison with OPM [ 33 ] on CREMA-D (3-frame) under OPM’s native single-layer fusion architecture. Accuracy (%), mean

\pm

std over 5 seeds. All rows use identical optimizer, schedule, and data pipeline. OPM hyperparameters follow the original paper.

Method	CREMA-D	AVE	KS	CMU-MOSI	Twitter15	Sarcasm	CH-SIMS
Baseline	61.93 $\pm$ 0.94	86.29 $\pm$ 0.29	78.40 $\pm$ 0.50	70.16 $\pm$ 0.65	51.33 $\pm$ 4.18	81.81 $\pm$ 0.21	60.05 $\pm$ 0.58
OGM-GE	69.52 $\pm$ 1.02	86.48 $\pm$ 0.85	—	—	49.98 $\pm$ 5.12	81.09 $\pm$ 0.40	61.46 $\pm$ 1.61
MMPareto	66.10 $\pm$ 0.74	85.91 $\pm$ 0.42	77.06 $\pm$ 0.52	70.69 $\pm$ 0.80	55.36 $\pm$ 2.65	81.61 $\pm$ 0.18	58.79 $\pm$ 2.93
AGM	57.45 $\pm$ 0.66	84.05 $\pm$ 0.38	77.24 $\pm$ 0.38	70.11 $\pm$ 1.24	54.15 $\pm$ 2.01	81.66 $\pm$ 0.24	59.00 $\pm$ 2.01
G-Blend	61.50 $\pm$ 1.90	86.79 $\pm$ 0.35	77.15 $\pm$ 1.04	70.50 $\pm$ 0.85	56.29 $\pm$ 1.56	81.64 $\pm$ 0.30	58.60 $\pm$ 1.75
CGGM	—	—	—	—	39.02 $\pm$ 0.28	80.07 $\pm$ 0.25	—

Appendix

Table 5: Supplementary F1-macro results (%, mean

\pm

std over 5 seeds). F1 computed at the epoch selected by best test accuracy, matching the Table 1 protocol. Bold indicates the best result per column. Ties within 0.05 pp are bolded jointly. ‘—’ denotes runs where training logs were not retained. BraTS is excluded as it uses Dice, not classification F1.

Panel of Figure 4 — Figure 4: Hyperparameter sensitivity on CREMA-D (3-frame, single seed). (a) Accuracy vs boost strength $\alpha$ — plateaus within noise above $\alpha{=}1.0$ . (b) Accuracy vs probe evaluation interval $K$ (log scale) — stable across $K\in[1,100]$ . (c) Accuracy vs boost cap $s_{\max}$ — mild increase from $1.5$ to $3.0$ .

Setting	CH-SIMS	CMU-MOSI
Baseline (Table 1 )	$70.42\pm 0.29$	$72.42\pm 0.48$
$\alpha=0.25$	$70.90\pm 0.55$	$71.89\pm 0.81$
$\alpha=0.5$ (default)	$69.80\pm 0.80$	$71.89\pm 0.82$
$\alpha=0.75$	$70.37\pm 0.41$	$71.92\pm 0.78$
$\alpha=1.0$	$70.42\pm 0.87$	$71.86\pm 0.91$
$\alpha=1.5$	$70.02\pm 0.50$	$71.86\pm 0.78$

Appendix

Table 6: Boost-strength sweep on the two high-imbalance sentiment benchmarks, boost alone without the OGM-GE composition (accuracy %, mean

\pm

std over 5 seeds, 100 epochs, Table 1 pipeline). The first row restates the joint-training baseline of Table 1 for reference. No entry is marked as best: every

\alpha

setting lies within one seed standard deviation of the baseline on both benchmarks, so none of these differences is resolved at this sample size.

Configuration	s/epoch	Overhead	Peak GPU memory
Baseline (joint training)	$17.43$	reference	$\approx 12.2$ GB
PGGB (probes + boost)	$17.71$	$+1.61\%$	$\approx 12.2$ GB
PGGB+OGM-GE	$18.71$	$+7.34\%$	$\approx 12.2$ GB

Appendix

Table 7: Wall-clock and peak-memory cost on CREMA-D (3-frame), single RTX 4090, mean seconds per epoch over 7 epochs under an identical data pipeline. Relative overhead is measured against the baseline row. Peak GPU memory ranged from

12.1

12.3

GB across all three configurations.

Benchmark	Optimizer	Modality	Applied $\bar{s}_{m}$	Grad. ratio	Update ratio
CREMA-D	SGD	visual	$1.64$	$1.52$	$1.50$
CMU-MOSI	Adam	text	$1.12$	$1.06$	$0.97$
		audio	$1.48$	$1.38$	$1.17$
		visual	$1.31$	$1.31$	$1.11$
CMU-MOSI	SGD (control)	text	$1.45$	$1.53$	$1.47$
		audio	$1.29$	$1.27$	$1.18$

Appendix

Table 8: Transmission of the applied boost to the parameter update. Each entry is a ratio of means over matched post-warmup steps between an

\alpha=0.75

run and its matched

\alpha=0

run, so the applied-scale column is the mean boost scale

\bar{s}_{m}

itself. The CREMA-D row is that run’s most-boosted modality, the visual encoder. The two CMU-MOSI blocks use identical data, architecture, and seed and differ only in the optimizer, the SGD block being an untuned control (see below). Values near the applied scale indicate transmission, values near

1

indicate absorption.

Optimizer	$\alpha=0$	$\alpha=0.75$	Effect (pp)	$p$
Adam (reported setting)	$72.68\pm 0.99$	$72.60\pm 1.05$	$-0.09$	$0.90$
SGD (control, untuned)	$55.89\pm 1.03$	$57.58\pm 1.04$	$+1.69$	$0.032$

Appendix

Table 9: Accuracy consequence of the boost on CMU-MOSI at full budget, composed contrast (

\alpha=0.75

against

\alpha=0

), five matched seeds per cell. Effect is the mean paired difference over the shared seeds and may differ from the difference of the rounded means.

p

-values are from a paired

t

-test over those seeds. The Adam means reproduce the CMU-MOSI OGM-GE and PGGB+OGM-GE entries of Table 1 , with spreads given here as the sample standard deviation over five seeds rather than the population standard deviation of Table 1 , which is smaller by a factor

\sqrt{4/5}

. The SGD rows are an untuned control, so their absolute level is not comparable with the Adam rows.

Figure 5: Probe-signal stability on CREMA-D (PGGB+OGM-GE, seed 42, 500 probe-eval checkpoints). For each modality, the EMA-smoothed estimator

\bar{P}_{m}

(red dashed) is substantially less noisy than the per-batch sample

P_{m}

(grey), and evolves smoothly with training. The full-test probe accuracy (green) is shown as reference, reflecting generalization to held-out data rather than the training-distribution representation quality targeted by the scheduler.

Modality	$P_{m}$ (batch)	$\bar{P}_{m}$ (EMA)	Reduction
Audio	$0.31$	$0.99$	n/a (saturated)
Visual	$9.32$	$4.36$	$-53\%$

Appendix

Table 10: Per-batch and EMA-smoothed probe accuracy std (pp), computed over the final 200 probe-eval checkpoints. The

19\times

steady-state bound (§ 3.2 ) is unconditional under i.i.d. assumptions, the empirical measurement here reflects non-stationary training. Visual (boost-driving signal) shows the expected reduction direction. Audio inversion (EMA std

>

batch std) reflects training-phase saturation: late-training batch-level audio is near the training-distribution ceiling, while the EMA integrates earlier, non-saturated samples.

Figure 6: Early and full-run probe trajectory on CREMA-D (PGGB+OGM-GE, seed 42). Both panels plot the EMA-smoothed per-modality probe accuracy

\bar{P}_{m}

, which is the quantity the scheduler consumes. (a) The first fifteen probe events, spanning the first three epochs, with the first event annotated. The ordering of the two modalities is inverted at that event alone, where both probes sit at chance level on the 32-sample evaluation half. (b) The same two curves over the full 100-epoch run, with the utilization gap

\delta

(Eq. ( 6 )) shaded. Audio leads at each of the remaining 499 events.

Method	Base alone	PGGB + Base	$\Delta$ mean (pp)	$\Delta$ std
OGM-GE	$69.14\pm 1.13$	$71.45\pm 1.71$	$\mathbf{+2.31}$	$+0.58$
MMPareto	$65.51\pm 0.87$	$66.00\pm 1.25$	$+0.49^{\dagger}$	$+0.38$
G-Blend	$61.10\pm 1.87$	$61.99\pm 0.99$	$+0.89^{\dagger}$	$\mathbf{-0.88}$ ( $-47\%$ )
AGM	$57.42\pm 0.73$	$58.17\pm 1.79$	$+0.75^{\dagger}$	$+1.06$
CGGM	$50.22\pm 1.39$	$50.32\pm 1.03$	$+0.10^{\dagger}$	$-0.36$

Appendix

Table 11: Composability with existing balancing methods on CREMA-D (3-frame, mean

\pm

std over 5 seeds). All rows use identical PGGB hyperparameters. Positive

\Delta

indicates composition does not hurt the base method’s mean accuracy. † : within noise band (

n{=}5

\sigma\approx 1

pp, 80%-power MDE

\approx 2

pp), not statistically distinguishable from zero.

Dataset	PGGB+MMPareto	PGGB+AGM	PGGB+G-Blend	PGGB+CGGM
AVE	$+1.01$	$+1.26$	$-0.25$	$-0.25$
Food101	$-0.09$	$-0.23$	$-0.09$	$+4.41^{\dagger}$

Appendix

Table 12: Composability across datasets:

\Delta

accuracy (pp) of PGGB+X versus X alone, mean over 5 seeds. AVE has high audio-visual modality imbalance, Food101 uses frozen pre-extracted features (operating-regime sensitivity reported in § 4.3.1 ). † PGGB recovers some CGGM degradation but absolute accuracy (

53.00\%

) remains far below baseline (

85.75\%

), so the boost does not rescue catastrophic regression.

Seed	$\alpha{=}0$	$\alpha{=}0.75$	$\Delta$ (pp)
Original five seeds
42	67.88	71.37	$+3.49$
123	68.15	72.04	$+3.89$
456	69.35	73.66	$+4.31$
789	69.49	71.24	$+1.75$
1024	70.83	68.95	$-1.88$

Appendix

Table 13: Per-seed test accuracy (%) for the isolation contrast on CREMA-D (3-frame):

\alpha{=}0

with probes active against

\alpha{=}0.75

, both composed with OGM-GE, 15 seeds per arm under the Table 1 protocol. Batches are listed in the order in which they were run.

\Delta

is the seed-matched difference. The aggregate row gives the mean with the sample standard deviation (

n-1

denominator), which is not the population standard deviation convention of Table 1 .

Method	Audio (strong)	Visual (weak)	Gap (pp)
Baseline	58.66 $\pm$ 1.60	46.13 $\pm$ 3.72	12.53
OGM-GE	58.04 $\pm$ 1.09	51.67 $\pm$ 1.85	6.37
MMPareto	60.03 $\pm$ 0.73	52.63 $\pm$ 5.92	7.39
AGM	55.32 $\pm$ 0.36	17.72 $\pm$ 0.73	37.61
G-Blend	57.77 $\pm$ 1.78	37.04 $\pm$ 2.43	20.73
MILES	59.81 $\pm$ 1.53	19.62 $\pm$ 10.42	40.19

Appendix

Table 14: Post-hoc linear-probe accuracy on CREMA-D (%, mean

\pm

std over 5 seeds). Probes are trained for 50 epochs on frozen encoder features from each method’s saved checkpoint, matching the protocol in Figure 3 . Gap is the absolute difference of column means. Chance level is

1/6\approx 16.67\%

for CREMA-D’s 6-class classification.

Dataset	Method	Text (strong)	Audio	Vision	Gap (pp)
CH-SIMS	Baseline	64.95	51.29	59.78	13.66
CH-SIMS	PGGB ( $\alpha{=}0.5$ )	66.43	53.17	60.09	13.26
CMU-MOSI	Baseline	70.99	48.86	54.49	22.13
CMU-MOSI	PGGB ( $\alpha{=}0.5$ )	71.72	49.24	53.47	22.48

Appendix

Table 15: Post-hoc linear-probe accuracy on the text-dominant sentiment benchmarks (%, mean over 5 seeds). Probes are trained on frozen encoder features from each method’s saved final checkpoint, following the protocol of Table 14 . Gap is the strongest minus the weakest per-modality probe accuracy. Differences between methods are small on these benchmarks, so we do not mark a best entry. The gap for the PGGB+OGM-GE composition on CH-SIMS is reported in the text.

Dataset	$\boldsymbol{\delta}$ (mean $\pm$ std)
CREMA-D	$0.268\pm 0.029$
CMU-MOSI	$0.222\pm 0.047$
CH-SIMS	$0.178\pm 0.086$
Twitter15	$0.150\pm 0.096$
Sarcasm	$0.076\pm 0.008$
Kinetics-Sounds	$0.038\pm 0.013$

Appendix

Table 16: Per-dataset utilization gap

\delta

at the end of baseline training (mean

\pm

std over 5 seeds), measured as the EMA-smoothed probe accuracy gap (Eq. ( 6 )).

Dataset	42	123	456	789	1024	Seeds above threshold
	Seed
CREMA-D	0.245	0.289	0.260	0.241	0.307	5/5
CMU-MOSI	0.273	0.191	0.273	0.195	0.178	5/5
CH-SIMS	0.112	0.154	0.118	0.322	0.184	3/5
Twitter15	0.221	0.161	0.019	0.098	0.260	3/5
Sarcasm	0.070	0.083	0.074	0.068	0.086	0/5

Appendix

Table 17: Per-seed utilization gap

\delta

at the final epoch of baseline training, for the five seeds used throughout the paper. The final column counts the seeds above the

\delta>0.15

threshold of Section 4.1 . Seed means and standard deviations are in Table 16 .

Dataset	Epoch 25	Epoch 50	Epoch 75
CREMA-D	0.473	0.455	0.313
CMU-MOSI	0.198	0.219	0.226
CH-SIMS	0.171	0.178	0.176
Twitter15	0.148	0.154	0.152
Sarcasm	0.100	0.085	0.075
Kinetics-Sounds	0.016	0.007	0.030

Appendix

Table 18: Utilization gap

\delta

read at intermediate epochs of baseline training (seed mean over the same five seeds as Table 17 ). Final-epoch values are those in Table 16 .

Dataset	MSLR	PGGB ( $\alpha{=}0.5$ )	PGGB+OGM-GE ( $\alpha{=}0.75$ )	Best
CREMA-D	63.79 $\pm$ 0.47	62.72 $\pm$ 1.65	71.45 $\pm$ 1.71	PGGB+OGM-GE
AVE	84.72 $\pm$ 0.59	87.41 $\pm$ 0.26	87.23 $\pm$ 0.58	PGGB
Kinetics-Sounds	78.56 $\pm$ 0.38	79.17 $\pm$ 0.97	77.33 $\pm$ 0.64	PGGB
CMU-MOSI	71.84 $\pm$ 0.76	71.89 $\pm$ 0.82	72.60 $\pm$ 0.94	PGGB+OGM-GE
CH-SIMS	70.99 $\pm$ 1.03	69.80 $\pm$ 0.80	72.43 $\pm$ 0.65	PGGB+OGM-GE
Twitter15	67.25 $\pm$ 0.66	66.85 $\pm$ 0.50	66.59 $\pm$ 0.51	MSLR (sub- $\sigma$ )

Appendix

Table 19: Cross-class comparison with MSLR [ 37 ] (5 seeds, mean

\pm

std). MSLR runs the fixed-ratio Keep variant, ratios per dataset listed above. PGGB family results reproduced from Table 1 for reference. Bold indicates the best per-row mean.

Modality Imbalance

Modality imbalance studies how unequal signal quality, learning dynamics, or model attention make multimodal systems over-rely on some inputs and neglect others.

See all 32 papers

Mar 15, 2026

Balancing Multimodal Domain Generalization via Gradient Modulation and Projection

Multimodal Domain Generalization (MMDG) leverages the complementary strengths of multiple modalities to enhance model generalization on unseen domains. A central challenge in multimodal learning is optimization imbalance, where modalities converge at different speeds during training. This imbalance leads to unequal gradient contributions, allowing some modalities to dominate the learning process while others lag behind. Existing balancing strategies typically regulate each modality's gradient contribution based on its classification performance on the source domain to alleviate this issue. However, relying solely on source-domain accuracy neglects a key insight in MMDG: modalities that excel on the source domain may generalize poorly to unseen domains, limiting cross-domain gains. To overcome this limitation, we propose Gradient Modulation Projection (GMP), a unified strategy that promotes balanced optimization in MMDG. GMP first decouples gradients associated with classification and domain-invariance objectives. It then modulates each modality's gradient based on semantic and domain confidence. Moreover, GMP dynamically adjusts gradient projections by tracking the relative strength of each task, mitigating conflicts between classification and domain-invariant learning within modality-specific encoders. Extensive experiments demonstrate that GMP achieves state-of-the-art performance and integrates flexibly with diverse MMDG methods, significantly improving generalization across multiple benchmarks.

Hongzhao Li, Guohao Shen, Shupan Li +2

School of Computer and Artificial Intelligence, Zhengzhou University · Engineering Research Center of Intelligent Swarm Systems, Ministry of Education · National Supercomputing Center in Zhengzhou +2

Multimodal Domain Generalization Domain Generalization Cross-Domain Generalization Multimodal Optimization Gradient Interference

Ask PDF

Oct 2, 2026

Balancing Multimodal Learning via Functional Progress

Multimodal learning often suffers from modality imbalance, where the joint optimization process is dominated by a single modality. Existing methods typically estimate modality imbalance from score disparities derived from prediction uncertainty or optimization statistics. However, due to distinct prediction uncertainty and learning dynamics across modalities, direct comparison of such scores may misinterpret intrinsic modality differences as progress gaps, leading to biased imbalance estimation. In this paper, we propose Function-Space Guided Multimodal Optimization (FGMO), which leverages a function-space progress signal to assess modality-wise optimization progress and coordinate optimization across modalities to alleviate modality imbalance. Specifically, we introduce Functional Progress Estimation (FPE) to measure each modality's update-induced function-space response and calibrate it against a loss-aligned unimodal reference, producing a comparable progress signal. Based on this signal, Functional Response Control (FRC) redistributes modality-level function-space budgets and realizes the target responses through tensor-wise learning-rate adjustment. Theoretical analysis establishes a one-step target-contraction property of FRC under bounded controller-state mismatch, and extensive experiments demonstrate the effectiveness of FGMO across multiple multimodal benchmarks.

Zhongjing Gu, Fengqiang Wan, Yiming Cui +2

Modality Imbalance Multimodal Optimization

Ask PDF

Mar 20, 2026

BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates

Learning from multiple modalities often suffers from imbalance, where information-rich modalities dominate optimization while weaker or partially missing modalities contribute less. This imbalance becomes severe in realistic settings with imbalanced missing rates (IMR), where each modality is absent with different probabilities, distorting representation learning and gradient dynamics. We revisit this issue from a training-process perspective and propose BALM, a model-agnostic plug-in framework to achieve balanced multimodal learning under IMR. The framework comprises two complementary modules: the Feature Calibration Module (FCM), which recalibrates unimodal features using global context to establish a shared representation basis across heterogeneous missing patterns; the Gradient Rebalancing Module (GRM), which balances learning dynamics across modalities by modulating gradient magnitudes and directions from both distributional and spatial perspectives. BALM can be seamlessly integrated into diverse backbones, including multimodal emotion recognition (MER) models, without altering their architectures. Experimental results across multiple MER benchmarks confirm that BALM consistently enhances robustness and improves performance under diverse missing and imbalance settings. Code available at: https://github.com/np4s/BALM_CVPR2026.git

Phuong-Anh Nguyen, Tien Anh Pham, Duc-Trong Le +1

VNU University of Engineering and Technology, Hanoi, Vietnam

Modality Imbalance Multimodal Learning Missing-Modality Learning

Ask PDF

Abstract

Modality Imbalance

Explore similar work

Balancing Multimodal Domain Generalization via Gradient Modulation and Projection

Balancing Multimodal Learning via Functional Progress

BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Rates