Organizations: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Guangdong Laboratory of Artificial Intelligence and Digital Economy · Computer Science, University of Nottingham · University of Electronic Science and Technology of China
In multimodal learning, modality imbalance often lets one modality dominate optimization and keeps the model from fully exploiting cross-modal interactions. Balance-oriented methods such as the Multimodal Competition Regularizer (MCR) perform well by alleviating this imbalance during training. These dynamic balancing methods, however, mainly adjust modality contributions and do not consider how imbalance affects the formation of synergistic information during fusion. Synergy is a major source of multimodal advantage, because it is predictive gain that cannot be recovered from unimodal evidence alone. We propose the Multimodal Synergistic Competition Regularizer (MSCR), a fusion method that jointly promotes balanced modality utilization and the discovery of synergistic information. MSCR has two complementary objectives. An anti-dominance objective suppresses excessive unimodal influence and leaves room for synergistic information, and a synergy objective encourages the fused predictor to outperform a non-synergistic unimodal reference. An adaptive gate further concentrates the regularization on samples that need stronger collaborative modeling. Experiments on five benchmarks show that MSCR consistently improves performance over MCR and strengthens synergistic capability. It raises conditional recovery, the accuracy of the fused prediction on samples that the unimodal predictors misclassify, as well as the share of the model's output that InterSHAP attributes to cross-modal interactions. Its gain in conditional recovery persists when the fusion module is Transformer-based. Code is available at https://github.com/chenxinyu991-collab/MSCR.
Figures & tables
Figure 1: Multimodal Synergistic Competition Regularizer (MSCR) : MSCR suppresses modality dominance and encourages the discovery of synergistic information. Raw data ( X1,X2 ) are encoded into representations ( Z1,Z2 ). These and their permuted variants ( Z~1,Z~2 ) are fed into the fusion network to produce joint ( pjoint ) and shuffled ( pshuf1,pshuf2 ) predictions, while the unimodal heads provide an additive reference ( pref ). The MSCR loss is built from three mechanisms: Ldom penalizes single-modality dominance by applying a smoothmax over modality sensitivities; Lsyn requires the joint prediction to outperform the unimodal reference by a margin m , with Δ denoting the hinge argument; and a proficiency-aware gate ( g ) scales the regularizer according to the best unimodal loss ( ℓbest ).
CREMA-D
AVE
UCF
MOSI
MOSEI
ResNet
Conformer
ResNet
Conformer
ResNet
Transformer
Method
V-A
V-A
V-A
V-A
V-A
V-T
V-A-T
V-T
V-A-T
Unimodals
V: 55.4 ± 3.0 A: 60.6 ± 2.3
V: 69.4 ± 2.0 A: 76.0 ± 2.6
V: 45.7 ± 1.6 A: 62.6 ± 0.9
V: 75.5 ± 1.2 A: 76.5 ± 2.4
V: 38.5 ± 0.9 A: 30.3 ± 1.5
V: 54.1 ± 3.7 A: 53.7 ± 0.6 T: 72.1 ± 3.3
V: 64.8 ± 0.2 A: 64.4 ± 0.2 T: 78.9 ± 1.7
Ensemble
71.7 ± 2.2
84.6 ± 1.0
70.5 ± 0.2
88.4 ± 2.2
52.8 ± 0.5
70.5 ± 2.1
67.2 ± 1.5
78.4 ± 0.7
77.2 ± 0.6
Joint Training
62.6 ± 5.8
74.6 ± 2.2
66.7 ± 1.5
82.2 ± 0.8
47.7 ± 1.5
73.0 ± 1.3
73.6 ± 1.3
80.5 ± 0.2
80.8 ± 0.3
Multi-Loss Vielzeuf et al. (2018)
69.2 ± 1.8
82.6 ± 0.9
70.1 ± 0.9
86.3 ± 1.1
51.1 ± 1.8
72.1 ± 0.4
73.6 ± 2.9
80.0 ± 0.7
80.2 ± 0.5
Table 1: Comparison of MSCR with prior multimodal training methods on five real-world benchmarks, covering bimodal and trimodal settings. Following MCR, the regression-style tasks (CMU-MOSI and CMU-MOSEI) are evaluated by binary accuracy, and the other datasets are standard classification tasks. For a fair comparison, we follow the evaluation protocol of MCR. MCR results are reproduced by us, while the remaining baseline results are taken from that work.
CREMA-D
AVE
UCF
MOSI
MOSEI
ResNet
Conformer
ResNet
Conformer
ResNet
Transformer
Method
V-A
V-A
V-A
V-A
V-A
V-T
V-A-T
V-T
V-A-T
MCR
10.13%
5.92%
5.50%
6.93%
10.28%
4.85%
10.87%
7.12%
6.71%
MSCR
18.35%
9.98%
7.07%
12.95%
15.45%
34.35%
17.44%
12.50%
13.69%
Table 2: Conditional recovery rates (Eq. ( 12 )) of MCR and MSCR, i.e., the accuracy of the multimodal model on samples that the unimodal predictors misclassify. Higher values are better.
Figure 2: Error comparison on CREMA-D with ResNet and Conformer unimodal backbones. Each matrix shows multimodal prediction performance conditioned on unimodal prediction correctness, with cells giving the percentage of samples in each category. Columns indicate whether both unimodal predictors are wrong, only audio is correct, only video is correct, or both are correct; rows indicate whether the multimodal model is correct. Results on the other datasets are given in Appendix C .
Dataset / Backbone
MCR
MSCR
CREMA-D / Conformer
5.22±1.33
10.31±1.89
AVE / Conformer
4.74±0.93
9.16±1.28
UCF / ResNet
12.10±0.67
18.21±1.12
Table 3: Global InterSHAP scores (%, mean ± standard deviation) under identical architectures. Higher scores indicate greater relative interaction contributions.
CREMA-D
AVE
UCF
ResNet
Conformer
ResNet
ResNet
Method
V-A
V-A
V-A
V-A
MCR
15.66%
11.82%
11.11%
16.41%
MSCR
21.39%
25.66%
15.13%
22.63%
Table 4: Conditional recovery rates under Transformer-based fusion architectures. Higher values indicate a greater ability to recover correct joint predictions when both unimodal predictors fail.
Ldom
Lsyn
CREMA-D
AVE
UCF
MOSI
MOSI
MOSEI
MOSEI
Conformer
Conformer
ResNet
V-T
V-A-T
V-T
V-A-T
85.1 ± 0.5
87.4 ± 0.2
52.9 ± 2.4
75.3 ± 0.7
78.6 ± 0.2
80.8 ± 0.4
81.1 ± 0.4
✓
85.9 ± 0.6
88.8 ± 0.4
53.6 ± 2.2
75.9 ± 0.7
79.7 ± 0.5
81.1 ± 0.3
81.4 ± 0.4
✓
86.2 ± 0.5
88.3 ± 0.2
53.5 ± 2.3
76.1 ± 0.9
79.2 ± 0.6
80.9 ± 0.4
81.2 ± 0.2
✓
✓
86.4 ± 0.9
89.3 ± 0.3
54.4 ± 2.4
76.8 ± 0.8
80.1 ± 0.5
81.3 ± 0.5
81.8 ± 0.6
Table 5: Ablation study of the anti-dominance objective Ldom and the synergy objective Lsyn across multiple datasets, backbones, and modality settings.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
dataset
λdom
λsyn
m
τdom
κ
τ
CREMA-D ResNet V-A
0.02
0.03
0.05
10.0
5.0
1.5
CREMA-D Conformer V-A
0.05
0.03
0.05
10.0
5.0
1.2
AVE ResNet V-A
0.02
0.03
0.05
10.0
5.0
1.5
AVE Conformer V-A
0.05
0.03
0.05
10.0
5.0
1.2
UCF ResNet V-A
0.02
0.03
0.05
10.0
5.0
1.5
MOSI Transformer V-T
0.05
0.03
0.04
10.0
5.0
1.2
Appendix
Table 6: Hyperparameters for MSCR. λdom and λsyn denote the weights of the anti-dominance objective and the synergy objective. m is the margin in the synergy objective. τdom is the smoothmax temperature. κ and τ are the sharpness and threshold of the adaptive modulation gate.
Figure 3: Error comparison matrices for additional datasets and backbones. Each matrix groups the multimodal predictions by the correctness of the corresponding unimodal predictions. Compared with MCR, MSCR achieves higher conditional recovery in the “Both Wrong” cases (unimodal predictors fail but the multimodal model is correct), which indicates a stronger ability to extract synergistic information when unimodal evidence is insufficient.
Method
CREMA-D Conformer
AVE Conformer
UCF ResNet
OGM
82.4±1.0
79.7±1.4
51.8±1.9
OGM + MSCR
83.5±1.2
81.1±1.0
52.9±1.7
MCR
85.1±0.5
87.4±0.2
52.9±2.4
MSCR-only (w/o MCR)
85.3±0.7
86.8±0.4
52.7±2.1
MCR + MSCR (full)
86.4±0.9
89.3±0.3
54.4±2.4
Appendix
Table 7: Independence and compatibility of MSCR. MSCR-only removes all MCR-specific regularization terms. Results are accuracy (%), reported as mean ± standard deviation. OGM results are taken from [ 21 ] ; all other results are obtained from our own experiments.
A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.
Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4
Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a +5.98% gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.
Yavuz Yarici, Ghassan AlRegib
OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA
Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.
Qun Dai, Liangjian Wen, Jiang Duan +7
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5