HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Qun Dai; Liangjian Wen; Jiang Duan; Yong Dai; Dongkai Wang; Maolin Wang; Mingjie Wang; Jianzhuang Liu

cs.AIOct 8, 2026

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Authors: Qun Dai, Liangjian Wen, Jiang Duan, Yong Dai, Dongkai Wang, Maolin Wang, Mingjie Wang, Jianzhuang Liu, +2 more

Organizations: School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd · X-Humanoid · Hong Kong Institute of AI for Science, City University of Hong Kong · Zhejiang Sci-Tech University · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Electronic Science and Technology of China

Abstract

Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.

Figures & tables

Figure 1: PID decomposes joint mutual information

I(X_{1},X_{2};Y)

into shared

R

, unique

U_{1}/U_{2}

, and synergistic

S

Figure 2: The overall pipeline of HRIL. Given $M$ modalities, we apply data augmentation to construct two views $X^{\prime}$ and $X^{\prime\prime}$ , and encode them into unimodal representations $\{{\mathrm{\mathbf{Z}}}_{m}^{\prime}\}_{m=1}^{M}$ and $\{{\mathrm{\mathbf{Z}}}_{m}^{\prime\prime}\}_{m=1}^{M}$ , as well as fused representations ${\mathrm{\mathbf{Z}}}_{f}^{\prime}$ and ${\mathrm{\mathbf{Z}}}_{f}^{\prime\prime}$ . The contrastive objective $\mathcal{L}_{\text{InfoNCE}}$ performs cross-view alignment between each unimodal representation and the fused one, while $\mathcal{L}_{\text{syn}}$ and $\mathcal{L}_{\text{align}}$ are computed from unimodal representations in both views.

Model	Redundancy $\uparrow$	Uniqueness $\uparrow$	Synergy $\uparrow$
Cross ♣ [ 36 ]	100.0	$11.6$	$50.0$
Cross+Self ♣ [ 52 ]	$99.7$	$86.9$	$50.0$
FactorCL ♣ [ 26 ]	$99.8$	$62.5$	$46.5$
CoMM [ 11 ]	$99.9_{\pm 0.06}$	$86.8_{\pm 2.99}$	$71.4_{\pm 3.47}$
HRIL (ours)	$99.7_{\pm 0.14}$	$\bm{92.6_{\pm 3.04}}$	$\bm{82.3_{\pm 1.61}}$

Table 1: Linear probing accuracy (in %) for redundancy (shape), uniqueness (texture), and synergy (color and texture) on the Trifeature dataset. ♣ denotes results from [ 11 ] .

Model	UR-FUNNY $\uparrow$	MOSI $\uparrow$	MOSEI $\uparrow$	Average
Tri_CLIP	$60.6_{\pm 0.79}$	$60.1_{\pm 3.43}$	$64.1_{\pm 0.42}$	$61.6$
CMC [ 39 ]	$60.1_{\pm 0.78}$	$62.4_{\pm 1.67}$	$62.7_{\pm 0.72}$	$61.73$
ConFu [ 23 ]	$61.3_{\pm 0.96}$	$62.5_{\pm 2.66}$	$64.8_{\pm 0.61}$	$62.87$
CoMM [ 11 ]	$64.8_{\pm 1.13}$	$66.0_{\pm 2.27}$	$69.9_{\pm 0.34}$	$66.9$
HRIL (ours)	$\bm{65.5_{\pm 0.56}}$	$\bm{67.2_{\pm 1.28}}$	$\bm{70.5_{\pm 0.20}}$	$\bm{67.73}$

Table 2: Linear probing top-1 accuracy (in %) for classification tasks on trimodal Multibench.

Model	Classification
Model	MIMIC $\uparrow$	MOSI $\uparrow$	UR-FUNNY $\uparrow$	MUSTARD $\uparrow$	Average ${}^{*}\uparrow$
$\text{Cross}^{\clubsuit}$ [ 36 ]	$66.7_{\pm 0.1}$	$47.8_{\pm 1.8}$	$50.1_{\pm 1.9}$	$53.5_{\pm 2.9}$	$54.52$
$\text{Cross+Self}^{\clubsuit}$ [ 52 ]	$65.49_{\pm 0.0}$	$49.0_{\pm 1.1}$	$59.9_{\pm 0.9}$	$53.9_{\pm 4.0}$	$57.07$
$\text{FactorCL}^{\clubsuit}$ [ 26 ]	$67.3_{\pm 0.0}$	$51.2_{\pm 1.6}$	$60.5_{\pm 0.8}$	$55.80_{\pm 0.9}$	$58.7$
CoMM [ 11 ]	$66.4_{\pm 0.4}$	$63.7_{\pm 2.5}$	$63.3_{\pm 0.5}$	$64.4_{\pm 1.1}$	$64.45$
HRIL (ours)	$\bm{68.0_{\pm 0.7}}$	$\bm{65.8_{\pm 1.2}}$	$\bm{63.9_{\pm 0.9}}$	$\bm{66.6_{\pm 2.7}}$	$\bm{66.08}$

Table 3: Linear probing top-1 accuracy (in %) for classification tasks on bimodal Multibench. ♣ denotes results are from [ 26 ] .

Loss components			Redundancy $\uparrow$	Uniqueness $\uparrow$	Synergy probe $\uparrow$
$\mathcal{L}_{\mathrm{InfoNCE}}$	$\mathcal{L}_{\mathrm{align}}$	$\mathcal{L}_{\mathrm{syn}}$	Redundancy $\uparrow$	Uniqueness $\uparrow$	Synergy probe $\uparrow$
$\times$	$\checkmark$	$\checkmark$	$28.5_{\pm 2.74}$	$12.5_{\pm 3.15}$	$50.0_{\pm 0.00}$
$\checkmark$	$\times$	$\times$	$99.9_{\pm 0.10}$	$87.2_{\pm 2.13}$	$50.0_{\pm 0.00}$
$\checkmark$	$\times$	$\checkmark$	$99.9_{\pm 0.05}$	$86.6_{\pm 2.79}$	$72.4_{\pm 3.16}$
$\checkmark$	$\checkmark$	$\times$	$99.9_{\pm 0.09}$	$85.1_{\pm 6.86}$	$78.0_{\pm 0.44}$
$\checkmark$	$\checkmark$	$\checkmark$	$99.7_{\pm 0.14}$	$\mathbf{92.6_{\pm 3.04}}$	$\mathbf{82.3_{\pm 1.61}}$

Table 4: Linear probing accuracy (in %) of multimodal interactions on the Trifeature dataset under different combinations of loss terms in Eq. 8 .

$\mathcal{L}_{\text{syn}}$	InterSHAP Value (%)			Average
$\mathcal{L}_{\text{syn}}$	UR-FUNNY	MOSI	MOSEI	Average
$\times$	$7.78_{\pm 4.56}$	$6.82_{\pm 3.73}$	$1.35_{\pm 0.53}$	5.32
✓	$9.25_{\pm 5.56}$	$9.90_{\pm 4.81}$	$1.47_{\pm 0.40}$	6.87
Improvement	$\bm{+1.47}$	$\bm{+3.08}$	$\bm{+0.12}$	$\bm{+1.55}$

Table 5: Quantitative evaluation of InterSHAP on Multibench [ 27 ] datasets.

\times

denotes the model without

\mathcal{L}_{\text{syn}}

, and

\checkmark

denotes the full objective of HRIL.

Fusion	$R$	$U$	$S$	Average
Concat+Linear	$99.9_{\pm 0.1}$	$65.4_{\pm 10.73}$	$50.0_{\pm 0.00}$	$71.77$
HRIL	$99.7_{\pm 0.14}$	$\bm{92.6_{\pm 3.04}}$	$\bm{82.3_{\pm 1.61}}$	$\bm{91.83}$

Table 6: Linear probing accuracy (in %) of multimodal interactions on the Trifeature dataset. Concat+Linear: concatenate the modality embeddings and pass through a linear layer.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	UR-FUNNY	MOSI
CoMM	$7.77_{\pm 3.05}$	$3.82_{\pm 1.90}$
HRIL without $\mathcal{L}_{\mathrm{syn}}$	$7.78_{\pm 4.56}$	$6.82_{\pm 3.73}$
Multilinear-rank control	$8.17_{\pm 6.33}$	$5.87_{\pm 4.26}$
HRIL	$9.25_{\pm 5.56}$	$9.90_{\pm 4.81}$

Appendix

Table 7: Further comparison with InterSHAP on the UR-FUNNY and MOSI datasets.

Hyperparameter	Value	UR-FUNNY	MOSEI
Tucker rank $r_{m}$	8	$65.0_{\pm 0.54}$	$69.9_{\pm 0.31}$
	16	$64.9_{\pm 1.34}$	$70.0_{\pm 0.47}$
	24	$65.5_{\pm 0.56}$	$70.5_{\pm 0.20}$
Projection dimension $r_{p}$	8	$64.8_{\pm 0.59}$	$70.2_{\pm 0.26}$
	16	$64.9_{\pm 0.66}$	$70.2_{\pm 0.20}$
	24	$65.5_{\pm 0.56}$	$70.5_{\pm 0.20}$

Appendix

Table 8: Sensitivity analysis: accuracy (%)

Figure 3: Performance sensitivity to mini-batch size on MOSEI, UR-FUNNY, and MOSI.

Model	Modalities	weighted-F1 $\uparrow$	macro-F1 $\uparrow$
SimCLR ♣△ [ 7 ]	V	$40.35_{\pm 0.23}$	$27.99_{\pm 0.33}$
CLIP ♣ [ 36 ]	V	$51.5$	$40.8$
	L	$51.0$	$43.0$
	V+L	$58.9$	$50.9$
SLIP ♣△ [ 32 ]	V+L	$56.54_{\pm 0.19}$	$47.35_{\pm 0.27}$
CLIP ♣△ [ 36 ]	V+L	$54.49_{\pm 0.19}$	$44.94_{\pm 0.30}$

Appendix

Table 9: Linear probing F1 scores (%) on MM-IMDb. △ indicates further training on unlabeled data. ♣ denotes results from [ 11 ] .

Model	V&T CP $\uparrow$
Cross	$86.3_{\pm 0.25}$
Cross+Self	$87.6_{\pm 0.26}$
CoMM	$87.0_{\pm 1.77}$
HRIL (ours)	$\mathbf{88.6}_{\pm 0.14}$

Appendix

Table 10: Linear probing accuracy (%) on Vision&Touch contact prediction (V&T CP).

Dataset	Model	Training Efficiency Metrics
Dataset	Model	Step Time (ms)	Optimizer Time (ms)	Backward / Step Ratio (%)
UR-FUNNY	CoMM	90.73	3.86	53.82
UR-FUNNY	HRIL	155.65	3.47	46.23
MOSI	CoMM	90.00	3.90	54.07
MOSI	HRIL	156.95	3.47	46.96
MOSEI	CoMM	90.57	4.06	53.99

Appendix

Table 11: Training efficiency comparison between CoMM [ 11 ] and HRIL on trimodal Multibench [ 27 ] datasets. Both methods use the same multimodal backbone. We report the average wall-clock step time, optimizer update time, and the percentage of each step spent in backward computation.

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning
May 12, 2026Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen +4Cross-Modal Learning Multimodal Learning
1KU Leuven · 2MIT
Structured Latent Modeling for Supervised Multimodal Information Decomposition
Sep 28, 2026Wanting Huang, Sanvesh Srivastava, Weiran WangMultimodal Disentangled Representation Learning Multimodal Learning
Department of Computer Science University of Iowa · Department of Statistics University of Iowa
Information-Theoretic Decomposition for Multimodal Interaction Learning
Jun 10, 2026Zequn Yang, Yake Wei, Haotian Ni +2Multimodal Fusion Multimodal Learning
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2

May 12, 2026, cs.LGJ/K move · Enter open · S save

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos+3

1KU Leuven · 2MIT

Cross-Modal Learning Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

A central objective in multimodal learning is to capture synergy: task-relevant information that arises only from the joint use of multiple modalities, and is not available from any single modality alone. While most approaches operate at the architectural level through larger or more complex fusion models, we propose a complementary axis: shaping the training objective itself. Standard training often emphasizes unimodal or redundant information, falling short on examples that require cross-modal reasoning. We formalize multimodal synergy through information theory and introduce the Synergistic Information Bottleneck (SynIB), a scalable objective that targets synergy directly. To prioritize learning synergy, SynIB motivates the model to predict accurately from all modalities while penalizing confidence when information from any modality is withheld. Alongside the standard task loss, the model runs forward passes with one modality masked at a time and is penalized for remaining confident, which would indicate reliance on unimodal cues rather than cross-modal interactions. We validate SynIB in two regimes. On synthetic XOR tasks where the ground-truth synergy is known by construction, standard training fails to recover it while SynIB does. On five real-world benchmarks, including three MultiBench affective tasks, Hateful Memes with CLIP-ViT and DeBERTa backbones, and a controllable irony extension of CREMA-D we introduce, SynIB improves accuracy on synergy-dependent examples by up to 7.8% and overall accuracy by up to 3.8%.

Abstract

Explore similar work

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Information-Theoretic Decomposition for Multimodal Interaction Learning