SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Yavuz Yarici; Ghassan AlRegib

cs.CVSep 26, 2026

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Authors: Yavuz Yarici, Ghassan AlRegib

Organizations: OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA

Abstract

Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a $+5.98\%$ gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.

Figures & tables

Figure 1: Overview of SynCo. Two augmented views are encoded and processed through a multimodal fusion path producing $Z^{\prime}$ and $Z^{\prime\prime}$ , and a unimodal path producing $Z_{1},\ldots,Z_{n}$ . Three contrastive objectives target distinct PID components: $\hat{I}_{\mathrm{NCE}}(Z^{\prime},Z^{\prime\prime})$ captures $R+S+\sum_{i}U_{i}$ , $\hat{I}_{\mathrm{NCE}}(Z_{i},Z^{\prime})$ and $\hat{I}_{\mathrm{NCE}}(Z_{i},Z^{\prime\prime})$ capture $R+U_{i}$ , and $\hat{I}_{\mathrm{NCE}}(Z^{\prime}_{\mathrm{res}},Z^{\prime\prime}_{\mathrm{res}})$ provides synergy-focused supervision on the interaction residuals.

Table 2

Model	Regr.	Classification
Model	V&T $\downarrow$	MIMIC $\uparrow$	MOSI $\uparrow$	UR-FUNNY $\uparrow$	MUSTARD $\uparrow$	Avg. ∗ $\uparrow$
Cross †	$33.09_{\pm 3.67}$	$66.70_{\pm 0.10}$	$47.80_{\pm 1.80}$	$50.10_{\pm 1.90}$	$53.50_{\pm 2.90}$	$54.53$
Cross+Self †	$7.56_{\pm 0.31}$	$65.49_{\pm 0.00}$	$49.00_{\pm 1.10}$	$59.90_{\pm 0.90}$	$53.90_{\pm 4.00}$	$57.07$
FactorCL †	$10.82_{\pm 0.56}$	$67.30_{\pm 0.00}$	$51.20_{\pm 1.60}$	$60.50_{\pm 0.80}$	$55.80_{\pm 0.90}$	$58.70$
COrAL	$5.40_{\pm 0.28}$	$\mathbf{67.95}_{\pm 0.21}$	$66.40_{\pm 0.84}$	$64.80_{\pm 0.47}$	$66.10_{\pm 1.02}$	$66.31$
CoMM	$6.85_{\pm 2.13}$	$65.92_{\pm 0.35}$	$63.70_{\pm 2.50}$	$63.25_{\pm 0.81}$	$64.58_{\pm 1.50}$	$64.36$

Table 3: Linear probing on MultiBench: MSE (

\times 10^{-4}

) for regression and top-1 accuracy (%) for classification. Bold: best within each section.

\dagger

: results from Dufumier et al. (2024) . ∗ : classification average.

Model	w-F1 $\uparrow$	m-F1 $\uparrow$
CLIP † Radford et al. (2021)	$54.49_{\pm 0.19}$	$44.94_{\pm 0.30}$
SLIP † Mu et al. (2022)	$56.54_{\pm 0.19}$	$47.35_{\pm 0.27}$
CoMM Dufumier et al. (2024)	$61.29_{\pm 0.73}$	$53.79_{\pm 0.22}$
SynCo (ours)	$\mathbf{62.12}_{\pm 0.17}$	$\mathbf{55.41}_{\pm 0.16}$
InfMasking Wen et al. (2025)	$61.97_{\pm 0.28}$	$55.28_{\pm 0.15}$
SynCo + InfMask.	$\mathbf{62.23}_{\pm 0.27}$	$\mathbf{55.64}_{\pm 0.22}$

Table 4: Linear probing F1-scores (weighted and macro, in %) on MM-IMDb (vision and language). All self-supervised methods use a CLIP backbone.

\dagger

denotes results from Dufumier et al. (2024) .

Projector	$Z$ $\uparrow$	$\hat{Z}$ $\downarrow$	$Z_{\mathrm{res}}$ $\uparrow$
Linear (ours)	77.58	50.00	87.24
MLP-2	74.14	59.75	75.31
MLP-3	73.11	62.10	74.13
MLP-5	71.25	64.81	72.84
Gated Residual	70.56	68.19	69.41
Transformer-4	66.45	71.47	62.04

Table 5: Synergy probe accuracy (%) on Trifeature across projector architectures. Chance level is 50%.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	IMU+Insole $\uparrow$	IMU+EMG $\uparrow$	EMG+Insole $\uparrow$	Avg. $\uparrow$
FactorCL	$36.74_{\pm 1.15}$	$34.18_{\pm 1.05}$	$30.92_{\pm 1.18}$	$33.95$
CoMM	$42.12_{\pm 0.81}$	$41.85_{\pm 0.94}$	$32.21_{\pm 1.05}$	$38.73$
SynCo (ours)	$\mathbf{45.52}_{\pm 0.54}$	$\mathbf{43.78}_{\pm 0.42}$	$\mathbf{36.15}_{\pm 0.59}$	$\mathbf{41.82}$
InfMasking	$43.85_{\pm 1.13}$	$42.32_{\pm 0.84}$	$36.04_{\pm 1.12}$	$40.74$
SynCo (ours) + InfMasking	$\mathbf{45.21}_{\pm 0.98}$	$\mathbf{44.15}_{\pm 1.07}$	$\mathbf{36.38}_{\pm 1.06}$	$\mathbf{41.91}$

Appendix

Table 6: Linear probing top-1 accuracy (%) on DARai across three bimodal sensor configurations. Bold values indicate the best within each section.

Model	MOSI $\uparrow$	UR-FUNNY $\uparrow$	MUSTARD $\uparrow$	Average $\uparrow$
CoMM Dufumier et al. (2024)	$64.00_{\pm 2.33}$	$64.74_{\pm 0.71}$	$65.52_{\pm 1.42}$	$64.75$
SynCo (ours)	$\mathbf{68.11}_{\pm 0.92}$	$\mathbf{66.85}_{\pm 0.55}$	$\mathbf{68.12}_{\pm 1.24}$	$\mathbf{67.69}$
InfMasking Wen et al. (2025)	$67.15_{\pm 1.21}$	$64.95_{\pm 0.92}$	$66.92_{\pm 2.75}$	$66.34$
SynCo (ours) + InfMasking	$\mathbf{67.52}_{\pm 1.11}$	$\mathbf{66.18}_{\pm 0.57}$	$\mathbf{67.48}_{\pm 1.48}$	$\mathbf{67.06}$

Appendix

Table 7: Linear probing top-1 accuracy (in %) on trimodal classification benchmarks (text, audio, video).

Panel of Figure 2 — Figure 2: Sensitivity of SynCo’s loss weights on Trifeature. (a) $\lambda_{\mathrm{pred}}=0.1$ fixed. (b) $\lambda_{\mathrm{int}}=0.01$ fixed. Dotted lines mark the values used for all experiments.

Objective	redundancy $\uparrow$	uniqueness $\uparrow$	synergy $\uparrow$
$\mathcal{L}_{\text{MCL}}$ (CoMM)	$\mathbf{99.9}_{\pm 0.05}$	$86.6_{\pm 1.32}$	$71.6_{\pm 1.07}$
SynCo w/o $\hat{I}_{\text{NCE}}(Z^{\prime},Z^{\prime\prime})$	$99.7_{\pm 0.37}$	$88.1_{\pm 0.49}$	$75.5_{\pm 0.28}$
SynCo (full)	$\mathbf{99.9}_{\pm 0.21}$	$\mathbf{90.15}_{\pm 1.14}$	$\mathbf{77.58}_{\pm 0.74}$

Appendix

Table 8: Linear probing accuracy (%) of redundancy, uniqueness, and synergy on Trifeature under different combinations of loss terms.

Representation	synergy $\uparrow$
$Z_{1}$ (modality 1)	$50.00_{\pm 0.00}$
$Z_{2}$ (modality 2)	$50.00_{\pm 0.00}$
$[Z_{1};Z_{2}]$ (concatenation)	$50.00_{\pm 0.00}$
$Z$ (fused)	$\mathbf{77.58}_{\pm 0.74}$

Appendix

Table 9: Linear probing accuracy (%) on the Trifeature synergy task for the unimodal representations, their concatenation, and the fused representation. Chance level is 50%.

Figure 3: Sensitivity of SynCo to the warmup length on Trifeature. We sweep warmup_epochs $\in\{0,5,10,20,40,60\}$ with all other hyperparameters fixed. Linear probing accuracy (%) is reported for each PID component, averaged over five independent runs. The dotted line marks the value used for all main-text experiments.

Method	Time/step (ms)	Memory (GB)	Parameters	Overhead
CoMM	$21.91\pm 0.23$	$0.17$	$448{,}464$	baseline
SynCo (ours)	$21.94\pm 0.18$	$0.17$	$451{,}704$	$+0.15\%$
InfMasking	$57.82\pm 0.33$	$0.71$	$448{,}504$	$+163.92\%$

Appendix

Table 10: Computational requirements of SynCo, CoMM, and InfMasking. Overhead is reported relative to CoMM. All measurements use batch size 64 on a single GPU.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 205 papers

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
Oct 8, 2026Qun Dai, Liangjian Wen, Jiang Duan +7Multimodal SSL Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Feb 16, 2026Carolin Cissee, Raneen Younis, Zahra AhmadiMultimodal Embedding Multimodal Contrastive Learning
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany

May 12, 2026J/K move · Enter open · S save

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos+3

1KU Leuven · 2MIT

Cross-Modal Learning Multimodal Learning Multimodal Understanding Modality Reliance in Multimodal Models

Open paper AskPDF arXiv

A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.

Abstract

Cross-Modal Representation Learning

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations