Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning

Liangjian Wen; Jiangtao Zheng; Jiang Duan; Qun Dai; Yong Dai; Jianzhuang Liu; Jun Wang; Shu-Dong Huang

cs.CVNov 5, 2026

Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning

Authors: Liangjian Wen, Jiangtao Zheng, Jiang Duan, Qun Dai, Yong Dai, Jianzhuang Liu, Jun Wang, Shu-Dong Huang,

Organizations: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd. · X-Humanoid Research Institute, Beijing, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · School of Management Science and Engineering, Southwestern University of Finance and Economics, Chengdu, China · School of Artificial Intelligence, Sichuan University, Chengdu, China · School of Computer, National University of Defense Technology, Changsha, 410073 · Artificial Intelligence Innovation and Incubation Institute, Fudan University · Shanghai Academy of AI for Science, Shanghai, China · University of Electronic Science and Technology of China, Chengdu, China

Abstract

Multimodal self-supervised learning is challenged by heterogeneity, unequal informativeness, and complex information interactions across multimodal data sources such as images, text, audio, and tabular data. While contrastive methods effectively align shared semantics, they typically rely on strong multi-view redundancy assumptions, overlooking modality-specific and synergistic information, thus leading to suboptimal representations. We propose the Multimodal Information Interactions Bottleneck (MIIB), a principled framework grounded in Partial Information Decomposition that explicitly encodes redundant, unique, and synergistic information within multimodal interactions. MIIB employs a dual-path model architecture for modality-specific encoding and multimodal fusion, optimized using surrogate tractable entropy-based objectives that enforce minimal sufficient representations without requiring explicit mutual information estimation. By leveraging variational bounds and score-based entropy estimation, MIIB efficiently captures complex cross-modal interactions while suppressing superfluous information. Experiments across diverse benchmarks demonstrate consistent improvements over state-of-the-art multimodal self-supervised methods. Code is available at https://github.com/wanglingzaishi/MIIB.

Figures & tables

Fig. 1: The process framework for multimodal representation learning organizes modality-specific learning and fusion as a sequential yet interdependent process.

Fig. 2: The overall architecture of MIIB. Given $n$ input modalities $X=(X_{1},\dots,X_{n})$ , we first augment them to generate two views, $X^{\prime}$ and $X^{\prime\prime}$ . These views are encoded independently by modality-specific encoders and latent converters to extract latent features $e^{\prime}_{1}\dots e^{\prime}_{n}$ and $e^{\prime\prime}_{1}\dots e^{\prime\prime}_{n}$ , which provide unified embeddings for subsequent multimodal interaction. These features are then processed by two parallel Transformer streams. The Fusion Transformer processes concatenated multimodal features to produce fused representations $Z^{\prime},Z^{\prime\prime}$ while processing individual modality features to yield modality-specific representations $Z^{\prime}_{1}\dots Z^{\prime}_{n}$ and $Z^{\prime\prime}_{1}\dots Z^{\prime\prime}_{n}$ . The Synergy Transformer operates on the concatenated multimodal features to model cross-modal interactions, producing synergistic representations $Z^{\prime}_{s}$ and $Z^{\prime\prime}_{s}$ . Finally, these representations are used to compute the modality-specific loss ( $\mathcal{L}_{\text{MS}}$ ), modality fusion loss ( $\mathcal{L}_{\text{MF}}$ ), and synergistic loss ( $\mathcal{L}_{\text{Synergy}}$ ).

Model	Redundancy $\uparrow$	Uniqueness $\uparrow$	Synergy $\uparrow$
Cross † [ 5 ]	$\mathbf{100.0}$	$11.6$	$50.0$
Cross+Self † [ 54 ]	$99.7$	$86.9$	$50.0$
FactorCL † [ 15 ]	$99.8$	$62.5$	$46.5$
CoMM [ 16 ]	$99.9_{\pm 0.06}$	$86.8_{\pm 2.99}$	$71.4_{\pm 3.47}$
MIIB (ours)	$99.9_{\pm 0.12}$	$\mathbf{89.8}_{\pm 3.23}$	$\mathbf{79.1}_{\pm 2.97}$

TABLE I: Linear probing accuracy (in %) of redundancy (shape), uniqueness (texture) and synergy (color and texture) on Trifeature dataset. † denotes results are from [ 16 ] .

Model	Regression	Classification
Model	V&T EE $\downarrow$	MIMIC $\uparrow$	MOSI $\uparrow$	UR-FUNNY $\uparrow$	MUSTARD $\uparrow$	Average ∗ $\uparrow$
Cross † [ 5 ]	$33.09_{\pm 3.67}$	$66.7_{\pm 0.1}$	$47.8_{\pm 1.8}$	$50.1_{\pm 1.9}$	$53.5_{\pm 2.9}$	$54.52$
Cross+Self † [ 54 ]	$7.56_{\pm 0.31}$	$65.49_{\pm 0.0}$	$49.0_{\pm 1.1}$	$59.9_{\pm 0.9}$	$53.9_{\pm 4.0}$	$57.07$
FactorCL † [ 15 ]	$10.82_{\pm 0.56}$	$67.3_{\pm 0.0}$	$51.2_{\pm 1.6}$	$60.5_{\pm 0.8}$	$55.80_{\pm 0.9}$	$58.7$
CoMM [ 16 ]	$7.96_{\pm 2.13}$	$66.4_{\pm 0.4}$	$63.7_{\pm 2.5}$	$63.3_{\pm 0.5}$	$64.4_{\pm 1.1}$	$64.45$
MIIB (ours)	$\mathbf{5.74}_{\pm 0.69}$	$\mathbf{68.1}_{\pm 0.3}$	$\mathbf{67.4}_{\pm 1.8}$	$\mathbf{65.4}_{\pm 0.9}$	$\mathbf{66.0}_{\pm 2.0}$	$\mathbf{66.73}$

TABLE II: Linear probing MSE(

\times 10^{-4}

) for regression task and top-1 accuracy (in %) for classification tasks on Multibench. † denotes results are from [ 16 ] . ∗ denotes average is only selected from the results of classification tasks. Colored rows indicate fine-tuning results.

Model	#Mod.	V&T CP $\uparrow$	UR-FUNNY $\uparrow$
Cross [ 5 ]	2	$86.3_{\pm 0.25}$	$50.1^{\dagger}$
Cross+Self [ 54 ]	2	$87.6_{\pm 0.26}$	$59.9^{\dagger}$
CoMM [ 16 ]	2	$87.0_{\pm 1.58}$	$63.3_{\pm 0.51}$
MIIB (ours)	2	$\mathbf{88.4}_{\pm 0.32}$	$\mathbf{65.4}_{\pm 0.90}$
CMC † [ 61 ]	3	$94.1$	$59.2$
CoMM [ 16 ]	3	$94.1_{\pm 0.17}$	$64.8_{\pm 1.13}$

TABLE III: Linear probing top-1 accuracy (in %) for classification tasks on Vision&Touch and UR-FUNNY. † denotes results are from [ 16 ]

Model	Modalities	weighted-f1 $\uparrow$	macro-f1 $\uparrow$
SimCLR †△ [ 9 ]	V	$40.35_{\pm 0.23}$	$27.99_{\pm 0.33}$
CLIP † [ 5 ]	V	$51.5$	$40.8$
CLIP † [ 5 ]	L	$51.0$	$43.0$
CLIP † [ 5 ]	V+L	$58.9$	$50.9$
SLIP †△ [ 63 ]	V+L	$56.54_{\pm 0.19}$	$47.35_{\pm 0.27}$
CLIP †△ [ 5 ]	V+L	$54.49_{\pm 0.19}$	$44.94_{\pm 0.30}$

TABLE IV: Linear probing F1-score (weighted and macro) (in %) for MM-IMDb.

\triangle

indicates further training on unlabeled data.

\dagger

denotes results are from [ 16 ] . For fair comparison, both CoMM and MIIB are implemented using the same pre-trained CLIP backbone. Colored rows indicate fine-tuning results.

Setting	Model	weighted-f1 $\uparrow$	macro-f1 $\uparrow$
Linear	CoMM [ 16 ]	$61.29_{\pm 0.73}$	$53.79_{\pm 0.22}$
Linear	MIIB (ours)	$\mathbf{62.80}_{\pm 0.24}$	$\mathbf{56.20}_{\pm 0.16}$
1%	CoMM [ 16 ]	$60.68_{\pm 0.31}$	$53.66_{\pm 0.40}$
1%	MIIB (ours)	$\mathbf{63.22}_{\pm 0.31}$	$\mathbf{56.80}_{\pm 0.47}$
10%	CoMM [ 16 ]	$61.60_{\pm 0.41}$	$54.48_{\pm 0.72}$
10%	MIIB (ours)	$\mathbf{63.60}_{\pm 0.26}$	$\mathbf{57.00}_{\pm 0.47}$

TABLE V: Performance comparison on the MM-IMDb dataset under different evaluation protocols. Results are reported in Macro-F1 and Weighted-F1 along with standard deviations. “Linear” denotes the best self-supervised linear probing results.

Model Setting	UR-FUNNY	MIMIC
w/o Synergy	$5.78_{\pm 2.05}$	$1.34_{\pm 0.80}$
w/ Synergy	$\mathbf{7.32}_{\pm 1.98}$	$\mathbf{2.54}_{\pm 1.26}$
Improvement	$\mathbf{+1.54}$	$\mathbf{+1.20}$
t-test	$2.70$	$3.61$
$p$ -value	$0.018$	$0.003$

TABLE VI: InterSHAP analysis over 15 random seeds on two benchmark datasets. Results are reported as mean

\pm

standard deviation together with paired

t

-tests.

Loss Weights			UR-FUNNY $\uparrow$	MUSTARD $\uparrow$
$\lambda_{1}$	$\lambda_{2}$	$\lambda_{3}$	UR-FUNNY $\uparrow$	MUSTARD $\uparrow$
1	0	0	$63.5_{\pm 1.07}$	$62.4_{\pm 3.22}$
1	1	0	$63.8_{\pm 0.90}$	$65.2_{\pm 1.24}$
1	0	1	$63.9_{\pm 0.71}$	$62.8_{\pm 1.61}$
1	1	1	$\mathbf{65.4}_{\pm 0.89}$	$\mathbf{66.0}_{\pm 2.0}$

TABLE VII: Linear probing accuracy (in %) on UR-FUNNY and MUSTARD datasets under different combinations of loss functions.

\lambda_{1}

\lambda_{2}

, and

\lambda_{3}

denote the weights for

\mathcal{L}_{MF}

\mathcal{L}_{MS}

, and

\mathcal{L}_{synergy}

Fig. 3: Parameter sensitivity analysis on the UR-FUNNY dataset with respect to the fusion penalty (

\lambda_{\text{fusion}}

), unimodality penalty (

\lambda_{\text{single}}

), and the synergy penalty (

\lambda_{\text{synergy}}

Fig. 4: Parameter sensitivity analysis of the hyperparameter

\gamma

on the UR-FUNNY and MOSI datasets.

Dataset	Model	FLOPs	MACs	#Params	Total Training Time
UR-FUNNY	CoMM	$0.15\,\mathrm{G}$	$0.08\,\mathrm{G}$	$7.11\,\mathrm{M}$	$0.48\,\mathrm{h}$
UR-FUNNY	MIIB	$0.18\,\mathrm{G}$	$0.09\,\mathrm{G}$	$7.13\,\mathrm{M}$	$0.54\,\mathrm{h}$
MMIMDb-clip	CoMM	$36.27\,\mathrm{G}$	$18.14\,\mathrm{G}$	$236.5\,\mathrm{M}$	$4.11\,\mathrm{h}$
MMIMDb-clip	MIIB	$38.08\,\mathrm{G}$	$19.04\,\mathrm{G}$	$243.6\,\mathrm{M}$	$4.31\,\mathrm{h}$

TABLE VIII: Comparative analysis of computational complexity and training efficiency. We report FLOPs, MACs, model size (#Params), and total training time. All measurements are conducted on a single NVIDIA 4090 GPU

Fig. 5: t-SNE visualization of unimodal and synergy representations on the MOSI (left) and MM-IMDb (right) datasets. The figure illustrates the distribution of unimodal representations (Modality 0, Modality 1) and the corresponding synergy representations in the learned embedding space.

Multimodal SSL

See all 27 papers

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
Oct 8, 2026Qun Dai, Liangjian Wen, Jiang Duan +7Multimodal SSL Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Feb 16, 2026Carolin Cissee, Raneen Younis, Zahra AhmadiMultimodal Embedding Multimodal Contrastive Learning
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany

May 12, 2026J/K move · Enter open · S save

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos+3

1KU Leuven · 2MIT

Cross-Modal Learning Multimodal Learning Multimodal Understanding Modality Reliance in Multimodal Models

Open paper AskPDF arXiv

A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.

Abstract

Multimodal SSL

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations