InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

Liangjian Wen; Qun Dai; Jiang Duan; Yong Dai; Jianzhuang Liu; Dongkai Wang; Jun Wang; Fuyuan Cao

cs.CVNov 5, 2026

InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

Authors: Liangjian Wen, Qun Dai, Jiang Duan, Yong Dai, Jianzhuang Liu, Dongkai Wang, Jun Wang, Fuyuan Cao,

Organizations: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd. · Beijing Humanoid Robot Innovation Center (X-Humanoid), Beijing, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · School of Management Science and Engineering, Southwestern University of Finance and Economics, Chengdu, China · School of Computer and Information Technology, Shanxi University, Taiyuan, China · Artificial Intelligence Innovation and Incubation Institute, Fudan University · Shanghai Academy of AI for Science, Shanghai, China · University of Electronic Science and Technology of China, Chengdu, China

Abstract

Multimodal models are expected to exploit synergistic information, that is, information available only when multiple modalities are observed together. In practice, however, a fusion module can fit the training objective by relying on a dominant modality or on information that modalities share, and the synergy between modalities is often left unused. We propose InfMasking, a unified multimodal learning framework based on Infinite Masking. Before fusion, InfMasking randomly masks the features of each modality; each sampled mask produces a different partial combination of cross-modal evidence and a fused representation of the corresponding modality interactions. All masked fused representations are required to remain consistent with a reference target: the unmasked fused view when training is unsupervised, and the class proxy of the task label when labels are available. Maintaining this consistency across diverse partial combinations discourages reliance on a dominant modality or on shared information alone, and drives the fusion module to learn cross-modal interaction patterns that remain meaningful under partial observations. Since the objective is an expectation over infinitely many masks, we model the masked fused representations as Gaussian and optimize closed-form variational bounds, whose population-moment gaps are controlled by the masking-induced covariance under this model. On the evaluated benchmarks, InfMasking improves predictive performance and yields higher mean PID estimates of task-label synergy in learned modality-specific representations, alongside higher InterSHAP interaction attribution in the trained predictors. Code is available at https://github.com/brightest66/InfMasking.

Figures & tables

Fig. 1: Overview of the proposed InfMasking framework. Given $n$ multimodal inputs $X=\{X_{1},X_{2},\ldots,X_{n}\}$ , InfMasking uses label-proxy supervision in the supervised setting (upper) and augmented-view alignment in the unsupervised setting (lower). Both settings share the same backbone architecture. Modality-specific encoders $f_{m}$ , $m=1,\ldots,n$ , extract latent features, and the infinite masking operator samples subsets of their tokens before fusion. The fusion module $g_{\phi}$ then produces the complete multimodal representation $Z$ from unmasked modality features and masked fused representations $Z_{\mathrm{mask}}^{1},Z_{\mathrm{mask}}^{2},\ldots,Z_{\mathrm{mask}}^{k}$ from sampled combinations of retained tokens across modalities. In the unsupervised setting, the fusion module $g_{\phi}$ additionally produces unimodal representations $Z_{1},Z_{2},\ldots,Z_{n}$ from individual modalities, whereas in the supervised setting the corresponding unimodal branches are obtained by projecting modality-specific latent features.

Model	Redundancy $\uparrow$	Uniqueness $\uparrow$	Synergy $\uparrow$
Cross ♣ [ 19 ]	100.0	$11.6$	$50.0$
Cross+Self ♣ [ 64 ]	$99.7$	$86.9$	$50.0$
FactorCL ♣ [ 51 ]	$99.8$	$62.5$	$46.5$
MAE [ 46 ]	$99.8_{\pm 0.11}$	$82.4_{\pm 3.09}$	$50.1_{\pm 0.24}$
CoMM [ 8 ]	$99.9_{\pm 0.06}$	$86.8_{\pm 2.99}$	$71.4_{\pm 3.47}$
InfMasking (ours)	$99.9_{\pm 0.09}$	$\textbf{90.6}_{\pm 2.31}$	$\textbf{77.0}_{\pm 4.22}$

TABLE I: Linear-probing accuracy (%) on the synthetic Trifeature dataset. Redundancy measures shared shape recognition, uniqueness measures texture recognition, and synergy measures texture–color composition. ♣ denotes that results are from [ 8 ] .

Model	Regression	Classification
Model	V&T EE $\downarrow$	MIMIC $\uparrow$	MOSI $\uparrow$	UR-FUNNY $\uparrow$	MUStARD $\uparrow$	Average ${}^{*}\uparrow$
$\text{Cross}^{\clubsuit}$ [ 19 ]	$33.09_{\pm 3.67}$	$66.7_{\pm 0.1}$	$47.8_{\pm 1.8}$	$50.1_{\pm 1.9}$	$53.5_{\pm 2.9}$	$54.53$
$\text{Cross+Self}^{\clubsuit}$ [ 64 ]	$7.56_{\pm 0.31}$	$65.49_{\pm 0.0}$	$49.0_{\pm 1.1}$	$59.9_{\pm 0.9}$	$53.9_{\pm 4.0}$	$57.07$
$\text{FactorCL}^{\clubsuit}$ [ 51 ]	$10.82_{\pm 0.56}$	$67.3_{\pm 0.0}$	$51.2_{\pm 1.6}$	$60.5_{\pm 0.8}$	$55.8_{\pm 0.9}$	$58.7$
MultiMAE-inspired [ 47 ]	$25.19_{\pm 12.65}$	$67.8_{\pm 0.3}$	$64.2_{\pm 3.0}$	$63.5_{\pm 0.9}$	$64.9_{\pm 4.3}$	$65.10$
DCP-inspired [ 48 ]	$8.07_{\pm 1.86}$	$67.7_{\pm 0.4}$	$66.1_{\pm 2.6}$	$64.1_{\pm 1.0}$	$63.6_{\pm 2.9}$	$65.38$

TABLE II: Linear-probing performance on bimodal MultiBench datasets in the unsupervised setting. Vision&Touch end-effector prediction reports MSE (

\times 10^{-4}

; lower is better), whereas MIMIC, MOSI, UR-FUNNY, and MUStARD report top-1 accuracy (%). ♣ denotes results reported in [ 51 ] . ∗ denotes the average over classification tasks only.

Model	#Mod.	V&T CP $\uparrow$	UR-FUNNY $\uparrow$
Cross	2	$86.3_{\pm 0.25}$	$50.1^{\clubsuit}$
Cross+Self	2	$87.6_{\pm 0.26}$	$59.9^{\clubsuit}$
CoMM	2	$87.0_{\pm 1.77}$	$63.3_{\pm 0.51}$
InfMasking (ours)	2	$88.5_{\pm 0.33}$	$64.3_{\pm 0.9}$
CMC ♣ [ 25 ]	3	$94.1$	$59.2$
CoMM	3	$\textbf{94.1}_{\pm 0.17}$	$64.8_{\pm 1.13}$

TABLE III: Linear-probing top-1 accuracy (%) on the Vision&Touch contact prediction and UR-FUNNY humor detection tasks in the unsupervised setting. #Mod. denotes the number of input modalities. ♣ denotes results are from [ 8 ] .

Model	Modalities	Weighted F1 $\uparrow$	Macro F1 $\uparrow$
SimCLR ♣△ [ 39 ]	V	$40.35_{\pm 0.23}$	$27.99_{\pm 0.33}$
CLIP ♣ [ 19 ]	V	$51.5$	$40.8$
	L	$51.0$	$43.0$
	V+L	$58.9$	$50.9$
SLIP ♣△ [ 65 ]	V+L	$56.54_{\pm 0.19}$	$47.35_{\pm 0.27}$
CLIP ♣△ [ 19 ]	V+L	$54.49_{\pm 0.19}$	$44.94_{\pm 0.30}$

TABLE IV: Linear-probing F1-scores (%) on the MM-IMDb multi-label genre classification task in the unsupervised setting. V and L denote visual and language modalities, respectively. △ indicates additional training on unlabeled data, and ♣ denotes results are from [ 8 ] .

$\lambda_{1}$	$\lambda_{2}$	$\lambda_{3}$	R	$U_{1}$	$U_{2}$	S	Average
0	0	1	$95.8_{\pm 1.91}$	$85.9_{\pm 2.11}$	$83.8_{\pm 2.97}$	$58.7_{\pm 7.11}$	$81.1$
0	1	1	$\textbf{99.9}_{\pm 0.06}$	$87.1_{\pm 3.31}$	$86.5_{\pm 2.60}$	$71.4_{\pm 3.47}$	$86.2$
1	1	0	$\textbf{99.9}_{\pm 0.08}$	$\textbf{90.7}_{\pm 2.10}$	$\textbf{91.4}_{\pm 3.03}$	$69.2_{\pm 6.20}$	$87.8$
1	1	1	$\textbf{99.9}_{\pm 0.09}$	$90.3_{\pm 1.52}$	$90.8_{\pm 2.88}$	$\textbf{77.0}_{\pm 4.22}$	89.5

TABLE V: Ablation of the unsupervised objective (Eq. ( 12 )): linear-probing accuracy (%) on Trifeature.

\lambda_{1}

\lambda_{2}

, and

\lambda_{3}

indicate whether the masked fusion, unimodal alignment, and fused contrastive terms are included (1) or excluded (0).

	Variant	MIMIC	MOSI	UR-FUNNY	MUStARD	Average
(a)	CoMM objective, no masking	$66.4_{\pm 0.41}$	$63.7_{\pm 2.5}$	$63.3_{\pm 0.51}$	$64.4_{\pm 1.1}$	$64.45$
(b)	RandomMask ( $k$ masks, no variational bound)	$67.7_{\pm 0.4}$	$67.8_{\pm 1.4}$	$63.5_{\pm 0.3}$	$63.26_{\pm 4.10}$	$65.57$
(c)	InfMasking (variational bound, $k$ masks)	$\textbf{68.1}_{\pm 0.42}$	$\textbf{69.0}_{\pm 1.2}$	$\textbf{64.3}_{\pm 0.9}$	$\textbf{66.8}_{\pm 2.5}$	67.05

TABLE VI: RandomMask and InfMasking in the unsupervised setting (linear-probing accuracy, %). Average is taken over the four datasets.

Model	MOSI	MOSEI	MIMIC	MAMI	MUTE	Average
Unimodal	T: $72.8_{\pm 0.83}$ V: $57.6_{\pm 2.12}$ A: $51.7_{\pm 2.31}$	T: $72.0_{\pm 0.18}$ V: $59.0_{\pm 0.46}$ A: $57.3_{\pm 0.27}$	TS: $61.2_{\pm 0.92}$ Tab: $55.2_{\pm 1.15}$	T: $58.5_{\pm 6.39}$ V: $61.8_{\pm 0.40}$	T: $63.1_{\pm 1.23}$ V: $64.3_{\pm 1.47}$	–
Fusion only	$71.1_{\pm 2.78}$	$72.8_{\pm 0.4}$	$68.0_{\pm 0.41}$	$64.7_{\pm 4.61}$	$64.3_{\pm 0.97}$	$68.2$
Vanilla MT	$72.9_{\pm 1.26}$	$72.2_{\pm 1.17}$	$68.1_{\pm 0.34}$	$63.4_{\pm 3.44}$	$64.1_{\pm 2.34}$	$68.1$
MulT [ 4 ]	$73.1_{\pm 1.54}$	$72.4_{\pm 0.21}$	$67.6_{\pm 0.40}$	$65.6_{\pm 0.99}$	$64.2_{\pm 1.50}$	$68.6$
MBT [ 6 ]	$\underline{73.8}_{\pm 0.34}$	$\underline{73.4}_{\pm 0.22}$	$\underline{68.9}_{\pm 0.19}$	$62.8_{\pm 1.91}$	$\underline{67.5}_{\pm 1.41}$	$69.3$
JMT [ 71 ]	$73.7_{\pm 1.07}$	$72.3_{\pm 0.48}$	$68.7_{\pm 0.24}$	$60.3_{\pm 1.40}$	$65.4_{\pm 1.05}$	$68.1$

TABLE VII: Supervised performance (%) on five multimodal benchmarks; T/V/A/TS/Tab denote unimodal text, visual, audio, time-series, and tabular baselines.

-

denotes a result not reported and

\mathtt{\_}

denotes the best performance among baseline methods.

$\lambda_{\mathrm{fuse}}$	$\lambda_{\mathrm{uni}}$	$\lambda_{\mathrm{mask}}$	MOSEI	MAMI	MUTE	Average
1	0	0	$72.8_{\pm 0.4}$	$64.7_{\pm 4.61}$	$64.3_{\pm 0.97}$	$67.3$
1	1	0	$72.2_{\pm 1.17}$	$63.4_{\pm 3.44}$	$64.1_{\pm 2.34}$	$66.6$
1	0	1	$72.4_{\pm 0.2}$	$68.7_{\pm 1.1}$	$67.6_{\pm 3.2}$	$69.6$
0	1	1	$72.4_{\pm 0.9}$	$66.9_{\pm 1.2}$	$68.6_{\pm 2.0}$	$69.3$
1	1	1	$73.8_{\pm 0.25}$	$69.3_{\pm 1.60}$	$70.0_{\pm 2.87}$	$71.0$

TABLE VIII: Ablation of the supervised objective in Eq. ( 17 ): classification accuracy (%).

\lambda_{\mathrm{fuse}}

\lambda_{\mathrm{uni}}

, and

\lambda_{\mathrm{mask}}

indicate whether the fused cross-entropy loss, the auxiliary unimodal cross-entropy losses, and the label-proxy InfMasking loss are included (1) or excluded (0).

	Variant	MOSI	MOSEI	MIMIC	MAMI	MUTE	Average
(a)	Vanilla MT (no masking)	$72.9_{\pm 1.26}$	$72.2_{\pm 1.17}$	$68.1_{\pm 0.34}$	$63.4_{\pm 3.44}$	$64.1_{\pm 2.34}$	$68.1$
(b)	RandomMask ( $k$ masks, no variational bound)	$75.3_{\pm 0.8}$	$\textbf{73.8}_{\pm 0.3}$	$68.7_{\pm 0.5}$	$67.4_{\pm 2.1}$	$68.7_{\pm 0.7}$	$70.8$
(c)	InfMasking (variational bound, $k$ masks)	$\textbf{75.9}_{\pm 0.89}$	$\textbf{73.8}_{\pm 0.25}$	$\textbf{69.4}_{\pm 0.06}$	$\textbf{69.3}_{\pm 1.60}$	$\textbf{70.0}_{\pm 2.87}$	71.7

TABLE IX: RandomMask versus the variational surrogate in the supervised setting (classification accuracy, %). Average is taken over the five datasets.

Setting	Dataset	Method	Synergy	Growth (%)
Unsupervised	UR-FUNNY	CoMM	$0.0362\pm 0.0037$	–
	UR-FUNNY	InfMasking	$\mathbf{0.0469}\pm 0.0049$	$+29.7\%$
	MOSI	CoMM	$0.0497\pm 0.0108$	–
	MOSI	InfMasking	$\mathbf{0.0635}\pm 0.0123$	$+27.9\%$
Supervised	MAMI	Vanilla MT	$0.0734\pm 0.0159$	–
	MAMI	InfMasking	$\mathbf{0.0868}\pm 0.0077$	$+18.2\%$

TABLE X: Estimated synergistic information (in nats) under the unsupervised and supervised settings. Growth rates are computed from unrounded means.

	Unsupervised		Supervised
	UR-FUNNY	MIMIC	MUTE	MAMI
w/o InfMasking	$5.30_{\pm 1.56}$	$1.19_{\pm 0.66}$	$0.25_{\pm 0.20}$	$3.81_{\pm 2.68}$
w/ InfMasking	$\mathbf{9.25}_{\pm 2.05}$	$\mathbf{1.67}_{\pm 0.68}$	$\mathbf{8.91}_{\pm 6.10}$	$\mathbf{6.73}_{\pm 2.30}$
Improvement	$+3.95$	$+0.48$	$+8.66$	$+2.92$
$t$ / $p$	$4.36$ / $0.003$	$2.61$ / $0.035$	$3.99$ / $0.005$	$4.55$ / $0.003$

TABLE XI: Cross-modal interaction analysis with InterSHAP (%) in the unsupervised (UR-FUNNY, MIMIC) and supervised (MUTE, MAMI) settings. Results are reported as the mean

\pm

standard deviation over eight independent runs together with paired

t

-test results.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 347 papers

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
Structured Latent Modeling for Supervised Multimodal Information Decomposition
Sep 28, 2026Wanting Huang, Sanvesh Srivastava, Weiran WangMultimodal Disentangled Representation Learning Multimodal Learning
Department of Computer Science University of Iowa · Department of Statistics University of Iowa
SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals
Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation Learning Multimodal Contrastive Learning
OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA

May 12, 2026J/K move · Enter open · S save

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos+3

1KU Leuven · 2MIT

Cross-Modal Learning Multimodal Learning Multimodal Understanding Modality Reliance in Multimodal Models

Open paper AskPDF arXiv

A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.

Abstract

Multimodal Learning

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Structured Latent Modeling for Supervised Multimodal Information Decomposition

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals