Hierarchical Contrastive Learning for Multimodal Data

Huichao Li; Junhan Yu; Doudou Zhou

stat.MLApr 7, 2026

Hierarchical Contrastive Learning for Multimodal Data

Authors: Huichao Li, Junhan Yu, Doudou Zhou

Organizations: Department of Mathematical Sciences, University of Chinese Academy of Sciences · Department of Statistics and Data Science, National University of Singapore

Abstract

Multimodal representation learning is commonly built on a shared-private decomposition, treating latent information as either common to all modalities or specific to one. This binary view is often inadequate: many factors are shared by only subsets of modalities, and ignoring such partial sharing can over-align unrelated signals and obscure complementary information. We propose Hierarchical Contrastive Learning (HCL), a framework that learns globally shared, partially shared, and modality-specific representations within a unified model. HCL combines a hierarchical latent-variable formulation with structural sparsity and a structure-aware contrastive objective that aligns only modalities that genuinely share a latent factor. Under uncorrelated latent variables, we prove identifiability of the hierarchical decomposition, establish recovery guarantees for the loading matrices, and derive parameter estimation and excess-risk bounds for downstream prediction. Simulations show accurate recovery of hierarchical structure and effective selection of task-relevant components. On multimodal electronic health records, HCL yields more informative representations and consistently improves predictive performance.

Figures & tables

Figure 1: Overall framework of Hierarchical Contrastive Learning. In Hierarchical contrastive learning module, each encoder function $h$ is designed to recover a specific latent structure. The illustration uses electronic health record data in Section 5 as a concrete example, but the framework extends directly to general multimodal data given appropriate encoders.

Panel of Figure 2 — Figure 2: Hierarchical decomposition for multimodal data.

Figure 3: Block matrices error of the hierarchical contrastive learning (HCL) framework as a function of the unlabeled sample size for modality dimensions

(300,500,800)

. The twelve subplots are organized by modality (rows) and latent structure (columns).

Figure 4: Global matrix error of the hierarchical contrastive learning (HCL) framework versus unlabeled sample size. The three panels correspond, from left to right, to modality dimensions (500, 500, 500), (300, 500, 800), and (800, 800, 800)

Figure 5: Downstream prediction performance of different methods under varying sample size

n

with noise level

c=0.1

\sigma_{\xi}=0.1

Figure 6: Comparison of methods for 30-day readmission prediction under joint training and pretrain–finetune. Results are averaged over 50 random splits.

Figure 7: Comparison of methods for next-visit length-of-stay prediction under joint training and pretrain–finetune. Results are averaged over 50 random splits.

Figure 8: Comparison of methods for one-year mortality prediction under joint training and pretrain–finetune. Results are averaged over 50 random splits.

Figure 9: Normalized importance weights of latent structures for the three downstream tasks under HCL. Results are averaged over 5 independent runs.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 799 papers

Structured Latent Modeling for Supervised Multimodal Information Decomposition
Sep 28, 2026Wanting Huang, Sanvesh Srivastava, Weiran WangMultimodal Disentangled Representation Learning Multimodal Learning
Department of Computer Science University of Iowa · Department of Statistics University of Iowa
Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations
Feb 16, 2026Carolin Cissee, Raneen Younis, Zahra AhmadiMultimodal Embedding Multimodal Contrastive Learning
Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Hannover, Germany · Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed), Hannover, Germany
CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning
Feb 23, 2026Chunlei Meng, Guanhong Huang, Rong Fu +3Cross-Modal Representation Learning Multimodal Learning
Fudan University · Shantou University · University of Macau +1

Sep 28, 2026J/K move · Enter open · S save

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Wanting Huang, Sanvesh Srivastava, Weiran Wang

Department of Computer Science University of Iowa · Department of Statistics University of Iowa

Multimodal Disentangled Representation Learning Multimodal Learning

Open paper AskPDF arXiv

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

Abstract

Representation Learning

Explore similar work

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations

CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning