Structured Latent Modeling for Supervised Multimodal Information Decomposition

Wanting Huang; Sanvesh Srivastava; Weiran Wang

cs.LGSep 28, 2026

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Authors: Wanting Huang, Sanvesh Srivastava, Weiran Wang

Organizations: Department of Computer Science University of Iowa · Department of Statistics University of Iowa

Abstract

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

Figures & tables

Setting		Canonical correlation (CC) $\uparrow$
$k=4$ , $\sigma=0.4$	$dim=12$	$0.99479\pm 0.00315$
	$dim=24$	$0.98816\pm 0.00396$
	$dim=48$	$0.94867\pm 0.03073$
	$dim=96$	$0.93104\pm 0.03477$
$dim=24$ , $k=4$	$\sigma=0.2$	$0.98624\pm 0.00523$
	$\sigma=0.4$	$0.98816\pm 0.00396$

Table 1: Recovery of S2MLVM subspaces. Here

dim=d_{1}=d_{2}

and

k=k_{12}=k_{1}=k_{2}=k_{c}

Method	$R$ -ACC $\uparrow$	$U_{1}$ -ACC $\uparrow$	$U_{2}$ -ACC $\uparrow$	$U$ -ACC $\uparrow$	$S$ -ACC $\uparrow$
FactorCL	$99.8^{*}$			$62.5^{*}$	$46.5^{*}$
CoMM	$99.9\pm 0.1^{*}$	$84.4\pm 2.4^{*}$	$91.2\pm 1.0^{*}$	$86.8\pm 3.0^{*}$	$71.4\pm 3.5^{*}$
InfMasking	$99.9\pm 0.1^{*}$	$90.7\pm 2.1^{*}$	$91.4\pm 3.0^{*}$	$90.6\pm 2.3^{*}$	$77.0\pm 4.2^{*}$
MCR	$99.9\pm 0.1$	$92.1\pm 1.4$	$93.2\pm 0.9$	$92.7\pm 1.2$	$90.2\pm 3.9$
S2MLVM-Mask	$93.4\pm 1.1$	$91.3\pm 2.2$	$84.9\pm 3.5$	$88.1\pm 2.7$	$85.4\pm 4.3$
S2MLVM-Contrast	$\mathbf{99.9}\pm\mathbf{0.1}$	$\mathbf{95.1}\pm\mathbf{0.7}$	$\mathbf{95.3}\pm\mathbf{0.8}$	$\mathbf{95.2}\pm\mathbf{0.5}$	$\mathbf{92.3}\pm\mathbf{0.9}$

Table 2: Classification accuracy (%) on TriFeature, for different task labels

R

U_{1}

U_{2}

S

Method	V&T EE $\downarrow$	MIMIC $\uparrow$	CREMA-D $\uparrow$	UCF101 $\uparrow$	MOSEI $\uparrow$
FactorCL	$10.8\pm 0.6^{*}$	$67.3\pm 0.0^{*}$	$61.1\pm 1.3$	$50.3\pm 2.2$	$78.5\pm 0.1$
CoMM	$8.0\pm 2.1^{*}$	$66.4\pm 0.4^{*}$	$59.9\pm 2.5$	$51.0\pm 2.1$	$79.7\pm 0.3$
InfMasking	$4.2\pm 0.4^{*}$	$68.1\pm 0.4^{*}$	$63.9\pm 2.7$	$51.3\pm 1.9$	$80.4\pm 0.4$
Unimodals	$\mathrm{I}:1.9\pm 0.0$	$\mathrm{S}:56.3\pm 0.5$	$\mathrm{V}:55.6\pm 3.4$	$\mathrm{V}:42.7\pm 1.2$	$\mathrm{V}:65.2\pm 0.2$
Unimodals	$\mathrm{F}:87.7\pm 0.5$	$\mathrm{T}:68.0\pm 0.4$	$\mathrm{A}:56.5\pm 3.7$	$\mathrm{A}:28.8\pm 1.7$	$\mathrm{T}:78.4\pm 1.3$
Joint Training	$1.9\pm 0.1$	$66.8\pm 1.4$	$62.6\pm 5.8^{*}$	$47.7\pm 1.5^{*}$	$80.5\pm 0.2^{*}$

Table 3: Test results on two modality benchmarks, by unsupervised, supervised, and our methods.

Method	MO $\uparrow$	UR $\uparrow$	MU $\uparrow$
FactorCL	$51.2\pm 1.6^{*}$	$60.5\pm 0.8^{*}$	$55.8\pm 0.9^{*}$
CoMM	$63.7\pm 2.5^{*}$	$63.3\pm 0.5^{*}$	$64.4\pm 1.1^{*}$
InfMasking	$69.0\pm 1.2^{*}$	$64.3\pm 0.9^{*}$	$66.8\pm 2.5^{*}$
Unimodals	$\mathrm{V}:55.1\pm 1.3$	$52.4\pm 0.5$	$57.7\pm 2.2$
Unimodals	$\mathrm{T}:72.0\pm 1.4$	$61.7\pm 0.7$	$63.4\pm 2.0$
Joint Training	$72.9\pm 1.6$	$61.7\pm 1.0$	$60.6\pm 2.0$

Table 4: Test results on two modality benchmarks MOSI (MO), UR-FUNNY (UR), MUSTARD (MU), by unsupervised, supervised, and our methods.

Method	UR-FUNNY $\uparrow$	V&T Contact $\uparrow$	MOSEI $\uparrow$
CoMM	$64.8\pm 1.1^{*}$	$94.1\pm 0.2^{*}$	$79.0\pm 0.3$
InfMasking	$65.6\pm 1.2^{*}$	$94.1\pm 0.1^{*}$	$78.9\pm 0.8$
MCR	$66.6\pm 1.5$	$94.1\pm 0.3$	$81.1\pm 0.4^{*}$
S2MLVM-Mask	$66.6\pm 1.8$	$94.0\pm 0.3$	$80.7\pm 0.7$
S2MLVM-Contrast	$66.0\pm 0.9$	$93.6\pm 0.9$	$81.4\pm 0.5$

Table 6: Test accuracies (%) on three-modality benchmarks.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	S2MLVM-Mask		S2MLVM-Contrast		dim $k$	loss
	Mask ratio	Batch size	Temperature	Batch size		$(\lambda_{\mathrm{density}},\lambda_{\mathrm{rep}})$
MOSI	$0.55$	$96$	$0.10$	$32$	$4$	(1,1)
MIMIC	$0.60$	$96$	$0.20$	$48$	$12$	(1,1)
UR-FUNNY	$0.30$	$64$	$0.10$	$128$	$4$	(1,1)
MUSTARD	$0.45$	$64$	$0.10$	$128$	$4$	(1,1)
CREMA-D	$0.55$	$32$	$0.15$	$8$	$16$	(0.2,0.8)

Appendix

Table 7: Selected hyperparameter for each dataset.

Study	Candidate	MOSI $\uparrow$	MUSTARD $\uparrow$
Mask ratio ( S2MLVM-Mask )	$m=0.40$	77.6	65.2
	$m=0.45$	79.1	70.3
	$m=0.50$	77.8	66.7
	$m=0.55$	80.0	70.0
	$m=0.60$	75.6	62.7
	$m=0.65$	77.9	65.2

Appendix

Table 8: Hyperparameter sensitivity on validation sets.

Variant	MOSI $\uparrow$	MUSTARD $\uparrow$
S2MLVM-Mask $(\lambda_{\mathrm{density}}=1,\lambda_{\mathrm{rep}}=1)$	79.0	68.6
w/o $L_{\mathrm{density}}$	64.6	63.2
w/o $L_{\mathrm{rep}}$	73.3	66.5
S2MLVM-Contrast $(\lambda_{\mathrm{density}}=1,\lambda_{\mathrm{rep}}=1)$	79.9	66.4
w/o $L_{\mathrm{density}}$	75.4	64.2
w/o $L_{\mathrm{rep}}$	77.5	62.8

Appendix

Table 9: Loss-coefficient ablations on validation datasets.

Explore similar work

Information-Theoretic Decomposition for Multimodal Interaction Learning
Jun 10, 2026Zequn Yang, Yake Wei, Haotian Ni +2Multimodal Learning Information-Theoretic Limits
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2
MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning
Jul 18, 2026Sana Tonekaboni, Viktoria Schuster, Caroline UhlerMultimodal Learning Modalities
Massachusetts Institute of Technology, Cambridge, MA, USA · Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard, Cambridge, MA, USA · Vector Institute, Toronto, Canada +1
Unsupervised Learning for Missing Modalities in Multimodal Learning
Jun 14, 2026Hassan Ismkhan, Hamid BouchahciaMissing Modalities Multimodal Learning
Bournemouth University Bournemouth, UK

Jun 10, 2026, cs.LGJ/K move · Enter open · S save

Information-Theoretic Decomposition for Multimodal Interaction Learning

Zequn Yang, Yake Wei, Haotian Ni, Zhihao Xu+1

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE+2

Multimodal Learning Information-Theoretic Limits

Open paper AskPDF arXiv

Multimodal learning hinges on capturing redundant, unique, and synergistic information across modalities, which collectively constitute multimodal interactions. A critical yet underexplored challenge is that these implicit interactions vary dynamically across samples. In this work, we present the first systematic, information-theoretic analysis highlighting why learning these dynamic, sample-specific interactions is critical for effective multimodal learning. Our analysis further reveals deficits in conventional paradigms at learning these distinct interaction types: modality ensemble approaches struggle to capture synergy, while joint learning paradigms often under-utilize redundant information. This highlights the need for an approach that can adaptively learn from different interaction types on a per-sample basis. To this end, we propose Decomposition-based Multimodal Interaction Learning (DMIL), a novel paradigm that explicitly models and learns from sample-specific interactions. First, we design a variational decomposition architecture to isolate the constituent interaction components. Second, we employ a new learning strategy that leverages these explicit interaction components in a fine-tuning process to achieve comprehensive interaction learning. Extensive experiments across diverse tasks and architectures demonstrate that DMIL consistently achieves superior performance by adapting to holistic sample-specific interactions. Our framework is flexible and broadly applicable, establishing an interaction-centric paradigm for multimodal learning. The code is available at https://github.com/GeWu-Lab/DMIL.

Abstract

Explore similar work

Information-Theoretic Decomposition for Multimodal Interaction Learning

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

Unsupervised Learning for Missing Modalities in Multimodal Learning