MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information

Xinyu Chen; Liangjian Wen; Jiang Duan; Dongkai Wang; Yong Dai; Jiayu Bai; Jianzhuang Liu; Zhen Tian

cs.AIOct 17, 2026

MSCR: Jointly Balancing Utilization Across Modalities and Discovering Synergistic Information

Authors: Xinyu Chen, Liangjian Wen, Jiang Duan, Dongkai Wang, Yong Dai, Jiayu Bai, Jianzhuang Liu, Zhen Tian,

Organizations: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · X-Humanoid · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences ·

Abstract

In multimodal learning, modality imbalance often lets one modality dominate optimization and keeps the model from fully exploiting cross-modal interactions. Balance-oriented methods such as the Multimodal Competition Regularizer (MCR) perform well by alleviating this imbalance during training. These dynamic balancing methods, however, mainly adjust modality contributions and do not consider how imbalance affects the formation of synergistic information during fusion. Synergy is a major source of multimodal advantage, because it is predictive gain that cannot be recovered from unimodal evidence alone. We propose the Multimodal Synergistic Competition Regularizer (MSCR), a fusion method that jointly promotes balanced modality utilization and the discovery of synergistic information. MSCR has two complementary objectives. An anti-dominance objective suppresses excessive unimodal influence and leaves room for synergistic information, and a synergy objective encourages the fused predictor to outperform a non-synergistic unimodal reference. An adaptive gate further concentrates the regularization on samples that need stronger collaborative modeling. Experiments on five benchmarks show that MSCR consistently improves performance over MCR and strengthens synergistic capability. It raises conditional recovery, the accuracy of the fused prediction on samples that the unimodal predictors misclassify, as well as the share of the model's output that InterSHAP attributes to cross-modal interactions. Its gain in conditional recovery persists when the fusion module is Transformer-based. Code is available at https://github.com/chenxinyu991-collab/MSCR.

Figures & tables

Figure 1: Multimodal Synergistic Competition Regularizer (MSCR) : MSCR suppresses modality dominance and encourages the discovery of synergistic information. Raw data ( $X_{1},X_{2}$ ) are encoded into representations ( $Z_{1},Z_{2}$ ). These and their permuted variants ( $\tilde{Z}_{1},\tilde{Z}_{2}$ ) are fed into the fusion network to produce joint ( $p_{\text{joint}}$ ) and shuffled ( $p_{\text{shuf1}},p_{\text{shuf2}}$ ) predictions, while the unimodal heads provide an additive reference ( $p_{\text{ref}}$ ). The MSCR loss is built from three mechanisms: $\mathcal{L}_{\text{dom}}$ penalizes single-modality dominance by applying a smoothmax over modality sensitivities; $\mathcal{L}_{\text{syn}}$ requires the joint prediction to outperform the unimodal reference by a margin $m$ , with $\Delta$ denoting the hinge argument; and a proficiency-aware gate ( $g$ ) scales the regularizer according to the best unimodal loss ( $\ell_{\text{best}}$ ).

	CREMA-D		AVE		UCF	MOSI		MOSEI
	ResNet	Conformer	ResNet	Conformer	ResNet	Transformer
Method	V-A	V-A	V-A	V-A	V-A	V-T	V-A-T	V-T	V-A-T
Unimodals	V: 55.4 $\pm$ 3.0 A: 60.6 $\pm$ 2.3	V: 69.4 $\pm$ 2.0 A: 76.0 $\pm$ 2.6	V: 45.7 $\pm$ 1.6 A: 62.6 $\pm$ 0.9	V: 75.5 $\pm$ 1.2 A: 76.5 $\pm$ 2.4	V: 38.5 $\pm$ 0.9 A: 30.3 $\pm$ 1.5	V: 54.1 $\pm$ 3.7 A: 53.7 $\pm$ 0.6 T: 72.1 $\pm$ 3.3		V: 64.8 $\pm$ 0.2 A: 64.4 $\pm$ 0.2 T: 78.9 $\pm$ 1.7
Ensemble	71.7 $\pm$ 2.2	84.6 $\pm$ 1.0	70.5 $\pm$ 0.2	88.4 $\pm$ 2.2	52.8 $\pm$ 0.5	70.5 $\pm$ 2.1	67.2 $\pm$ 1.5	78.4 $\pm$ 0.7	77.2 $\pm$ 0.6
Joint Training	62.6 $\pm$ 5.8	74.6 $\pm$ 2.2	66.7 $\pm$ 1.5	82.2 $\pm$ 0.8	47.7 $\pm$ 1.5	73.0 $\pm$ 1.3	73.6 $\pm$ 1.3	80.5 $\pm$ 0.2	80.8 $\pm$ 0.3
Multi-Loss Vielzeuf et al. (2018)	69.2 $\pm$ 1.8	82.6 $\pm$ 0.9	70.1 $\pm$ 0.9	86.3 $\pm$ 1.1	51.1 $\pm$ 1.8	72.1 $\pm$ 0.4	73.6 $\pm$ 2.9	80.0 $\pm$ 0.7	80.2 $\pm$ 0.5

Table 1: Comparison of MSCR with prior multimodal training methods on five real-world benchmarks, covering bimodal and trimodal settings. Following MCR, the regression-style tasks (CMU-MOSI and CMU-MOSEI) are evaluated by binary accuracy, and the other datasets are standard classification tasks. For a fair comparison, we follow the evaluation protocol of MCR. MCR results are reproduced by us, while the remaining baseline results are taken from that work.

	CREMA-D		AVE		UCF	MOSI		MOSEI
	ResNet	Conformer	ResNet	Conformer	ResNet	Transformer
Method	V-A	V-A	V-A	V-A	V-A	V-T	V-A-T	V-T	V-A-T
MCR	10.13%	5.92%	5.50%	6.93%	10.28%	4.85%	10.87%	7.12%	6.71%
MSCR	18.35%	9.98%	7.07%	12.95%	15.45%	34.35%	17.44%	12.50%	13.69%

Table 2: Conditional recovery rates (Eq. ( 12 )) of MCR and MSCR, i.e., the accuracy of the multimodal model on samples that the unimodal predictors misclassify. Higher values are better.

Panel of Figure 2 — Figure 2: Error comparison on CREMA-D with ResNet and Conformer unimodal backbones. Each matrix shows multimodal prediction performance conditioned on unimodal prediction correctness, with cells giving the percentage of samples in each category. Columns indicate whether both unimodal predictors are wrong, only audio is correct, only video is correct, or both are correct; rows indicate whether the multimodal model is correct. Results on the other datasets are given in Appendix C .

Dataset / Backbone	MCR	MSCR
CREMA-D / Conformer	$5.22\pm 1.33$	$\mathbf{10.31\pm 1.89}$
AVE / Conformer	$4.74\pm 0.93$	$\mathbf{9.16\pm 1.28}$
UCF / ResNet	$12.10\pm 0.67$	$\mathbf{18.21\pm 1.12}$

Table 3: Global InterSHAP scores (%, mean

\pm

standard deviation) under identical architectures. Higher scores indicate greater relative interaction contributions.

	CREMA-D		AVE	UCF
	ResNet	Conformer	ResNet	ResNet
Method	V-A	V-A	V-A	V-A
MCR	15.66%	11.82%	11.11%	16.41%
MSCR	21.39%	25.66%	15.13%	22.63%

Table 4: Conditional recovery rates under Transformer-based fusion architectures. Higher values indicate a greater ability to recover correct joint predictions when both unimodal predictors fail.

$\mathcal{L}_{\mathrm{dom}}$	$\mathcal{L}_{\mathrm{syn}}$	CREMA-D	AVE	UCF	MOSI	MOSI	MOSEI	MOSEI
		Conformer	Conformer	ResNet	V-T	V-A-T	V-T	V-A-T
		85.1 $\pm$ 0.5	87.4 $\pm$ 0.2	52.9 $\pm$ 2.4	75.3 $\pm$ 0.7	78.6 $\pm$ 0.2	80.8 $\pm$ 0.4	81.1 $\pm$ 0.4
✓		85.9 $\pm$ 0.6	88.8 $\pm$ 0.4	53.6 $\pm$ 2.2	75.9 $\pm$ 0.7	79.7 $\pm$ 0.5	81.1 $\pm$ 0.3	81.4 $\pm$ 0.4
	✓	86.2 $\pm$ 0.5	88.3 $\pm$ 0.2	53.5 $\pm$ 2.3	76.1 $\pm$ 0.9	79.2 $\pm$ 0.6	80.9 $\pm$ 0.4	81.2 $\pm$ 0.2
✓	✓	86.4 $\pm$ 0.9	89.3 $\pm$ 0.3	54.4 $\pm$ 2.4	76.8 $\pm$ 0.8	80.1 $\pm$ 0.5	81.3 $\pm$ 0.5	81.8 $\pm$ 0.6

Table 5: Ablation study of the anti-dominance objective

\mathcal{L}_{\mathrm{dom}}

and the synergy objective

\mathcal{L}_{\mathrm{syn}}

across multiple datasets, backbones, and modality settings.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

dataset	$\lambda_{\mathrm{dom}}$	$\lambda_{\mathrm{syn}}$	$m$	$\tau_{\mathrm{dom}}$	$\kappa$	$\tau$
CREMA-D ResNet V-A	0.02	0.03	0.05	10.0	5.0	1.5
CREMA-D Conformer V-A	0.05	0.03	0.05	10.0	5.0	1.2
AVE ResNet V-A	0.02	0.03	0.05	10.0	5.0	1.5
AVE Conformer V-A	0.05	0.03	0.05	10.0	5.0	1.2
UCF ResNet V-A	0.02	0.03	0.05	10.0	5.0	1.5
MOSI Transformer V-T	0.05	0.03	0.04	10.0	5.0	1.2

Appendix

Table 6: Hyperparameters for MSCR.

\lambda_{\mathrm{dom}}

and

\lambda_{\mathrm{syn}}

denote the weights of the anti-dominance objective and the synergy objective.

m

is the margin in the synergy objective.

\tau_{\mathrm{dom}}

is the smoothmax temperature.

\kappa

and

\tau

are the sharpness and threshold of the adaptive modulation gate.

Panel of Figure 3 — Figure 3: Error comparison matrices for additional datasets and backbones. Each matrix groups the multimodal predictions by the correctness of the corresponding unimodal predictions. Compared with MCR, MSCR achieves higher conditional recovery in the “Both Wrong” cases (unimodal predictors fail but the multimodal model is correct), which indicates a stronger ability to extract synergistic information when unimodal evidence is insufficient.

Method	CREMA-D Conformer	AVE Conformer	UCF ResNet
OGM	$82.4\pm 1.0$	$79.7\pm 1.4$	$51.8\pm 1.9$
OGM + MSCR	$83.5\pm 1.2$	$81.1\pm 1.0$	$52.9\pm 1.7$
MCR	$85.1\pm 0.5$	$87.4\pm 0.2$	$52.9\pm 2.4$
MSCR-only (w/o MCR)	$85.3\pm 0.7$	$86.8\pm 0.4$	$52.7\pm 2.1$
MCR + MSCR (full)	$\mathbf{86.4\pm 0.9}$	$\mathbf{89.3\pm 0.3}$	$\mathbf{54.4\pm 2.4}$

Appendix

Table 7: Independence and compatibility of MSCR. MSCR-only removes all MCR-specific regularization terms. Results are accuracy (%), reported as mean

\pm

standard deviation. OGM results are taken from [ 21 ] ; all other results are obtained from our own experiments.

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 757 papers

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals
Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation Learning Multimodal Contrastive Learning
OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
Oct 8, 2026Qun Dai, Liangjian Wen, Jiang Duan +7Multimodal SSL Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5

May 12, 2026J/K move · Enter open · S save

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos+3

1KU Leuven · 2MIT

Cross-Modal Learning Multimodal Learning Multimodal Understanding Modality Reliance in Multimodal Models

Open paper AskPDF arXiv

A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.

Abstract

Multimodal Fusion

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling