InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

Liangjian Wen; Qun Dai; Jiang Duan; Yong Dai; Jianzhuang Liu; Dongkai Wang; Jun Wang; Fuyuan Cao

cs.CVNov 5, 2026

InfMasking: A Unified Variational Framework for Synergistic Multimodal Learning

Authors: Liangjian Wen, Qun Dai, Jiang Duan, Yong Dai, Jianzhuang Liu, Dongkai Wang, Jun Wang, Fuyuan Cao,

Organizations: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd. · Beijing Humanoid Robot Innovation Center (X-Humanoid), Beijing, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · School of Management Science and Engineering, Southwestern University of Finance and Economics, Chengdu, China · School of Computer and Information Technology, Shanxi University, Taiyuan, China · Artificial Intelligence Innovation and Incubation Institute, Fudan University · Shanghai Academy of AI for Science, Shanghai, China · University of Electronic Science and Technology of China, Chengdu, China

Abstract

Multimodal models are expected to exploit synergistic information, that is, information available only when multiple modalities are observed together. In practice, however, a fusion module can fit the training objective by relying on a dominant modality or on information that modalities share, and the synergy between modalities is often left unused. We propose InfMasking, a unified multimodal learning framework based on Infinite Masking. Before fusion, InfMasking randomly masks the features of each modality; each sampled mask produces a different partial combination of cross-modal evidence and a fused representation of the corresponding modality interactions. All masked fused representations are required to remain consistent with a reference target: the unmasked fused view when training is unsupervised, and the class proxy of the task label when labels are available. Maintaining this consistency across diverse partial combinations discourages reliance on a dominant modality or on shared information alone, and drives the fusion module to learn cross-modal interaction patterns that remain meaningful under partial observations. Since the objective is an expectation over infinitely many masks, we model the masked fused representations as Gaussian and optimize closed-form variational bounds, whose population-moment gaps are controlled by the masking-induced covariance under this model. On the evaluated benchmarks, InfMasking improves predictive performance and yields higher mean PID estimates of task-label synergy in learned modality-specific representations, alongside higher InterSHAP interaction attribution in the trained predictors. Code is available at https://github.com/brightest66/InfMasking.

Figures & tables

Fig. 1: Overview of the proposed InfMasking framework. Given $n$ multimodal inputs $X=\{X_{1},X_{2},\ldots,X_{n}\}$ , InfMasking uses label-proxy supervision in the supervised setting (upper) and augmented-view alignment in the unsupervised setting (lower). Both settings share the same backbone architecture. Modality-specific encoders $f_{m}$ , $m=1,\ldots,n$ , extract latent features, and the infinite masking operator samples subsets of their tokens before fusion. The fusion module $g_{\phi}$ then produces the complete multimodal representation $Z$ from unmasked modality features and masked fused representations $Z_{\mathrm{mask}}^{1},Z_{\mathrm{mask}}^{2},\ldots,Z_{\mathrm{mask}}^{k}$ from sampled combinations of retained tokens across modalities. In the unsupervised setting, the fusion module $g_{\phi}$ additionally produces unimodal representations $Z_{1},Z_{2},\ldots,Z_{n}$ from individual modalities, whereas in the supervised setting the corresponding unimodal branches are obtained by projecting modality-specific latent features.

Model	Redundancy $\uparrow$	Uniqueness $\uparrow$	Synergy $\uparrow$
Cross ♣ [ 19 ]	100.0	$11.6$	$50.0$
Cross+Self ♣ [ 64 ]	$99.7$	$86.9$	$50.0$
FactorCL ♣ [ 51 ]	$99.8$	$62.5$	$46.5$
MAE [ 46 ]	$99.8_{\pm 0.11}$	$82.4_{\pm 3.09}$	$50.1_{\pm 0.24}$
CoMM [ 8 ]	$99.9_{\pm 0.06}$	$86.8_{\pm 2.99}$	$71.4_{\pm 3.47}$
InfMasking (ours)	$99.9_{\pm 0.09}$	$\textbf{90.6}_{\pm 2.31}$	$\textbf{77.0}_{\pm 4.22}$

TABLE I: Linear-probing accuracy (%) on the synthetic Trifeature dataset. Redundancy measures shared shape recognition, uniqueness measures texture recognition, and synergy measures texture–color composition. ♣ denotes that results are from [ 8 ] .

Model	Regression	Classification
Model	V&T EE $\downarrow$	MIMIC $\uparrow$	MOSI $\uparrow$	UR-FUNNY $\uparrow$	MUStARD $\uparrow$	Average ${}^{*}\uparrow$
$\text{Cross}^{\clubsuit}$ [ 19 ]	$33.09_{\pm 3.67}$	$66.7_{\pm 0.1}$	$47.8_{\pm 1.8}$	$50.1_{\pm 1.9}$	$53.5_{\pm 2.9}$	$54.53$
$\text{Cross+Self}^{\clubsuit}$ [ 64 ]	$7.56_{\pm 0.31}$	$65.49_{\pm 0.0}$	$49.0_{\pm 1.1}$	$59.9_{\pm 0.9}$	$53.9_{\pm 4.0}$	$57.07$
$\text{FactorCL}^{\clubsuit}$ [ 51 ]	$10.82_{\pm 0.56}$	$67.3_{\pm 0.0}$	$51.2_{\pm 1.6}$	$60.5_{\pm 0.8}$	$55.8_{\pm 0.9}$	$58.7$
MultiMAE-inspired [ 47 ]	$25.19_{\pm 12.65}$	$67.8_{\pm 0.3}$	$64.2_{\pm 3.0}$	$63.5_{\pm 0.9}$	$64.9_{\pm 4.3}$	$65.10$
DCP-inspired [ 48 ]	$8.07_{\pm 1.86}$	$67.7_{\pm 0.4}$	$66.1_{\pm 2.6}$	$64.1_{\pm 1.0}$	$63.6_{\pm 2.9}$	$65.38$

TABLE II: Linear-probing performance on bimodal MultiBench datasets in the unsupervised setting. Vision&Touch end-effector prediction reports MSE (

\times 10^{-4}

; lower is better), whereas MIMIC, MOSI, UR-FUNNY, and MUStARD report top-1 accuracy (%). ♣ denotes results reported in [ 51 ] . ∗ denotes the average over classification tasks only.

Model	#Mod.	V&T CP $\uparrow$	UR-FUNNY $\uparrow$
Cross	2	$86.3_{\pm 0.25}$	$50.1^{\clubsuit}$
Cross+Self	2	$87.6_{\pm 0.26}$	$59.9^{\clubsuit}$
CoMM	2	$87.0_{\pm 1.77}$	$63.3_{\pm 0.51}$
InfMasking (ours)	2	$88.5_{\pm 0.33}$	$64.3_{\pm 0.9}$
CMC ♣ [ 25 ]	3	$94.1$	$59.2$
CoMM	3	$\textbf{94.1}_{\pm 0.17}$	$64.8_{\pm 1.13}$

TABLE III: Linear-probing top-1 accuracy (%) on the Vision&Touch contact prediction and UR-FUNNY humor detection tasks in the unsupervised setting. #Mod. denotes the number of input modalities. ♣ denotes results are from [ 8 ] .

Model	Modalities	Weighted F1 $\uparrow$	Macro F1 $\uparrow$
SimCLR ♣△ [ 39 ]	V	$40.35_{\pm 0.23}$	$27.99_{\pm 0.33}$
CLIP ♣ [ 19 ]	V	$51.5$	$40.8$
	L	$51.0$	$43.0$
	V+L	$58.9$	$50.9$
SLIP ♣△ [ 65 ]	V+L	$56.54_{\pm 0.19}$	$47.35_{\pm 0.27}$
CLIP ♣△ [ 19 ]	V+L	$54.49_{\pm 0.19}$	$44.94_{\pm 0.30}$

TABLE IV: Linear-probing F1-scores (%) on the MM-IMDb multi-label genre classification task in the unsupervised setting. V and L denote visual and language modalities, respectively. △ indicates additional training on unlabeled data, and ♣ denotes results are from [ 8 ] .

$\lambda_{1}$	$\lambda_{2}$	$\lambda_{3}$	R	$U_{1}$	$U_{2}$	S	Average
0	0	1	$95.8_{\pm 1.91}$	$85.9_{\pm 2.11}$	$83.8_{\pm 2.97}$	$58.7_{\pm 7.11}$	$81.1$
0	1	1	$\textbf{99.9}_{\pm 0.06}$	$87.1_{\pm 3.31}$	$86.5_{\pm 2.60}$	$71.4_{\pm 3.47}$	$86.2$
1	1	0	$\textbf{99.9}_{\pm 0.08}$	$\textbf{90.7}_{\pm 2.10}$	$\textbf{91.4}_{\pm 3.03}$	$69.2_{\pm 6.20}$	$87.8$
1	1	1	$\textbf{99.9}_{\pm 0.09}$	$90.3_{\pm 1.52}$	$90.8_{\pm 2.88}$	$\textbf{77.0}_{\pm 4.22}$	89.5

TABLE V: Ablation of the unsupervised objective (Eq. ( 12 )): linear-probing accuracy (%) on Trifeature.

\lambda_{1}

\lambda_{2}

, and

\lambda_{3}

indicate whether the masked fusion, unimodal alignment, and fused contrastive terms are included (1) or excluded (0).

	Variant	MIMIC	MOSI	UR-FUNNY	MUStARD	Average
(a)	CoMM objective, no masking	$66.4_{\pm 0.41}$	$63.7_{\pm 2.5}$	$63.3_{\pm 0.51}$	$64.4_{\pm 1.1}$	$64.45$
(b)	RandomMask ( $k$ masks, no variational bound)	$67.7_{\pm 0.4}$	$67.8_{\pm 1.4}$	$63.5_{\pm 0.3}$	$63.26_{\pm 4.10}$	$65.57$
(c)	InfMasking (variational bound, $k$ masks)	$\textbf{68.1}_{\pm 0.42}$	$\textbf{69.0}_{\pm 1.2}$	$\textbf{64.3}_{\pm 0.9}$	$\textbf{66.8}_{\pm 2.5}$	67.05

TABLE VI: RandomMask and InfMasking in the unsupervised setting (linear-probing accuracy, %). Average is taken over the four datasets.

Model	MOSI	MOSEI	MIMIC	MAMI	MUTE	Average
Unimodal	T: $72.8_{\pm 0.83}$ V: $57.6_{\pm 2.12}$ A: $51.7_{\pm 2.31}$	T: $72.0_{\pm 0.18}$ V: $59.0_{\pm 0.46}$ A: $57.3_{\pm 0.27}$	TS: $61.2_{\pm 0.92}$ Tab: $55.2_{\pm 1.15}$	T: $58.5_{\pm 6.39}$ V: $61.8_{\pm 0.40}$	T: $63.1_{\pm 1.23}$ V: $64.3_{\pm 1.47}$	–
Fusion only	$71.1_{\pm 2.78}$	$72.8_{\pm 0.4}$	$68.0_{\pm 0.41}$	$64.7_{\pm 4.61}$	$64.3_{\pm 0.97}$	$68.2$
Vanilla MT	$72.9_{\pm 1.26}$	$72.2_{\pm 1.17}$	$68.1_{\pm 0.34}$	$63.4_{\pm 3.44}$	$64.1_{\pm 2.34}$	$68.1$
MulT [ 4 ]	$73.1_{\pm 1.54}$	$72.4_{\pm 0.21}$	$67.6_{\pm 0.40}$	$65.6_{\pm 0.99}$	$64.2_{\pm 1.50}$	$68.6$
MBT [ 6 ]	$\underline{73.8}_{\pm 0.34}$	$\underline{73.4}_{\pm 0.22}$	$\underline{68.9}_{\pm 0.19}$	$62.8_{\pm 1.91}$	$\underline{67.5}_{\pm 1.41}$	$69.3$
JMT [ 71 ]	$73.7_{\pm 1.07}$	$72.3_{\pm 0.48}$	$68.7_{\pm 0.24}$	$60.3_{\pm 1.40}$	$65.4_{\pm 1.05}$	$68.1$

TABLE VII: Supervised performance (%) on five multimodal benchmarks; T/V/A/TS/Tab denote unimodal text, visual, audio, time-series, and tabular baselines.

-

denotes a result not reported and

\mathtt{\_}

denotes the best performance among baseline methods.

$\lambda_{\mathrm{fuse}}$	$\lambda_{\mathrm{uni}}$	$\lambda_{\mathrm{mask}}$	MOSEI	MAMI	MUTE	Average
1	0	0	$72.8_{\pm 0.4}$	$64.7_{\pm 4.61}$	$64.3_{\pm 0.97}$	$67.3$
1	1	0	$72.2_{\pm 1.17}$	$63.4_{\pm 3.44}$	$64.1_{\pm 2.34}$	$66.6$
1	0	1	$72.4_{\pm 0.2}$	$68.7_{\pm 1.1}$	$67.6_{\pm 3.2}$	$69.6$
0	1	1	$72.4_{\pm 0.9}$	$66.9_{\pm 1.2}$	$68.6_{\pm 2.0}$	$69.3$
1	1	1	$73.8_{\pm 0.25}$	$69.3_{\pm 1.60}$	$70.0_{\pm 2.87}$	$71.0$

TABLE VIII: Ablation of the supervised objective in Eq. ( 17 ): classification accuracy (%).

\lambda_{\mathrm{fuse}}

\lambda_{\mathrm{uni}}

, and

\lambda_{\mathrm{mask}}

indicate whether the fused cross-entropy loss, the auxiliary unimodal cross-entropy losses, and the label-proxy InfMasking loss are included (1) or excluded (0).

	Variant	MOSI	MOSEI	MIMIC	MAMI	MUTE	Average
(a)	Vanilla MT (no masking)	$72.9_{\pm 1.26}$	$72.2_{\pm 1.17}$	$68.1_{\pm 0.34}$	$63.4_{\pm 3.44}$	$64.1_{\pm 2.34}$	$68.1$
(b)	RandomMask ( $k$ masks, no variational bound)	$75.3_{\pm 0.8}$	$\textbf{73.8}_{\pm 0.3}$	$68.7_{\pm 0.5}$	$67.4_{\pm 2.1}$	$68.7_{\pm 0.7}$	$70.8$
(c)	InfMasking (variational bound, $k$ masks)	$\textbf{75.9}_{\pm 0.89}$	$\textbf{73.8}_{\pm 0.25}$	$\textbf{69.4}_{\pm 0.06}$	$\textbf{69.3}_{\pm 1.60}$	$\textbf{70.0}_{\pm 2.87}$	71.7

TABLE IX: RandomMask versus the variational surrogate in the supervised setting (classification accuracy, %). Average is taken over the five datasets.

Setting	Dataset	Method	Synergy	Growth (%)
Unsupervised	UR-FUNNY	CoMM	$0.0362\pm 0.0037$	–
	UR-FUNNY	InfMasking	$\mathbf{0.0469}\pm 0.0049$	$+29.7\%$
	MOSI	CoMM	$0.0497\pm 0.0108$	–
	MOSI	InfMasking	$\mathbf{0.0635}\pm 0.0123$	$+27.9\%$
Supervised	MAMI	Vanilla MT	$0.0734\pm 0.0159$	–
	MAMI	InfMasking	$\mathbf{0.0868}\pm 0.0077$	$+18.2\%$

TABLE X: Estimated synergistic information (in nats) under the unsupervised and supervised settings. Growth rates are computed from unrounded means.

	Unsupervised		Supervised
	UR-FUNNY	MIMIC	MUTE	MAMI
w/o InfMasking	$5.30_{\pm 1.56}$	$1.19_{\pm 0.66}$	$0.25_{\pm 0.20}$	$3.81_{\pm 2.68}$
w/ InfMasking	$\mathbf{9.25}_{\pm 2.05}$	$\mathbf{1.67}_{\pm 0.68}$	$\mathbf{8.91}_{\pm 6.10}$	$\mathbf{6.73}_{\pm 2.30}$
Improvement	$+3.95$	$+0.48$	$+8.66$	$+2.92$
$t$ / $p$	$4.36$ / $0.003$	$2.61$ / $0.035$	$3.99$ / $0.005$	$4.55$ / $0.003$

TABLE XI: Cross-modal interaction analysis with InterSHAP (%) in the unsupervised (UR-FUNNY, MIMIC) and supervised (MUTE, MAMI) settings. Results are reported as the mean

\pm

standard deviation over eight independent runs together with paired

t

-test results.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 347 papers

May 12, 2026

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4

1KU Leuven · 2MIT

Cross-Modal Learning Multimodal Learning Multimodal Understanding Modality Reliance in Multimodal Models

Ask PDF

Sep 28, 2026

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

Wanting Huang, Sanvesh Srivastava, Weiran Wang

Department of Computer Science University of Iowa · Department of Statistics University of Iowa

Multimodal Disentangled Representation Learning Multimodal Learning

Ask PDF

Sep 26, 2026

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes into three components: redundancy shared between modalities, uniqueness specific to each modality, and synergy available only from their joint observation. Recent frameworks extend contrastive learning to capture all three components, yet synergy remains undertrained in practice. We propose SynCo (Synergy Contrastive Learning), a method that directly addresses synergy undertraining through dedicated supervision on an interaction residual. SynCo fits a linear projector to predict the fused representation from independently computed unimodal features, and the resulting interaction residual, which removes the linearly unimodal-predictable component, receives dedicated contrastive supervision at negligible computational cost. On the controlled Trifeature benchmark, SynCo achieves state-of-the-art synergy capture with a

+5.98\%

gain over the baseline, and on real-world benchmarks from MultiBench, DARai, and MM-IMDb, SynCo consistently outperforms or matches prior methods across diverse modality combinations and task types. The method operates as a plug-in to existing contrastive multimodal frameworks without modifying the underlying fusion architecture and can further improve synergy capture when combined with other methods.

Yavuz Yarici, Ghassan AlRegib

OLIVES, CSIP, School of ECE Georgia Institute of Technology Atlanta, GA, USA

Cross-Modal Representation Learning Multimodal Contrastive Learning Multimodal SSL

Ask PDF

Abstract

Multimodal Learning

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Structured Latent Modeling for Supervised Multimodal Information Decomposition

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals