Cross-Dimension and Cross-Modality Relational Tensor Representation Learning for Multimodal Health Data Modeling

Jiawei Guo; Meng Zhao

cs.LGOct 17, 2026

Cross-Dimension and Cross-Modality Relational Tensor Representation Learning for Multimodal Health Data Modeling

Authors: Jiawei Guo, Meng Zhao

Organizations: Department of Industrial and Systems Engineering, Lehigh University

Abstract

Multimodal health data integrate information from heterogeneous clinical sources, creating substantial opportunities for data-driven decision support. Realizing these opportunities, however, requires effectively modeling complex dependencies that arise both within individual modalities and across modalities. This paper proposes Relational Tensor Representation (RTR), a unified framework that jointly learns dimension-specific tensor representations and cross-dimensional and cross-modality relationships. RTR introduces learnable relational bridge matrices that maintain relational information while preserving the structural identities of tensor dimensions and modalities. A structured fusion mechanism further integrates factor-level and relationship-level information into compact representations for downstream prediction. Experiments on simulated data and two multimodal neurophysiological datasets, covering emotion recognition and brain activity classification, demonstrate that RTR consistently outperforms representative tensor learning, multimodal fusion, and Transformer-based baselines. These results highlight the value of explicitly modeling cross-dimensional and cross-modality relationships for learning informative representations from heterogeneous multimodal tensor data. The source code is planned for public release to support reproducibility and future research.

Figures & tables

Figure 1: Overview of the proposed RTR framework.

Scenarios	Cross-dimensional dependency $\alpha$	Cross-modality dependency $\beta$
Scenario 1: Weak – Weak	0.2	0.2
Scenario 2: Strong – Weak	0.8	0.2
Scenario 3: Weak – Strong	0.2	0.8
Scenario 4: Strong – Strong	0.8	0.8

Table 1: Simulation scenarios under different levels of within-modality and cross-modality dependency.

Figure 2: Illustrative dependency patterns under the four simulation scenarios.

Panel A: Weak cross-modality dependency ( $\beta=0.2$ )
Method	Scenario 1: Weak dimension ( $\alpha=0.2$ )				Scenario 2: Strong dimension ( $\alpha=0.8$ )
Method	Acc.	Precision	Recall	Macro-F1	Acc.	Precision	Recall	Macro-F1
SVM	$0.8653\pm 0.0252$	$0.8655\pm 0.0252$	$0.8653\pm 0.0252$	$0.8653\pm 0.0252$	$0.8670\pm 0.0406$	$0.8671\pm 0.0406$	$0.8670\pm 0.0406$	$0.8669\pm 0.0406$
Modality1 MLP	$0.9069\pm 0.0171$	$0.9073\pm 0.0176$	$0.9069\pm 0.0171$	$0.9069\pm 0.0171$	$0.8980\pm 0.0248$	$0.8981\pm 0.0247$	$0.8980\pm 0.0248$	$0.8980\pm 0.0248$
Modality2 MLP	$0.9080\pm 0.0143$	$0.9084\pm 0.0146$	$0.9080\pm 0.0143$	$0.9080\pm 0.0143$	$0.8952\pm 0.0289$	$0.8955\pm 0.0287$	$0.8952\pm 0.0289$	$0.8952\pm 0.0289$
Tensor Fusion	$0.9030\pm 0.0158$	$0.9032\pm 0.0160$	$0.9030\pm 0.0158$	$0.9030\pm 0.0158$	$0.8958\pm 0.0311$	$0.8960\pm 0.0310$	$0.8958\pm 0.0311$	$0.8958\pm 0.0311$

Table 2: Simulation performance under four scenarios. Results are reported as mean

\pm

standard deviation over four independent repeats. Bold values indicate the best performance in each setting.

Method	SEED-IV				EEG-fMRI
Method	Acc.	Precision	Recall	Macro-F1	Acc.	Precision	Recall	Macro-F1
SVM	$84.37\pm 0.79$	$84.95\pm 0.73$	$84.18\pm 0.72$	$84.36\pm 0.73$	$85.42\pm 5.38$	$87.57\pm 3.91$	$85.42\pm 5.38$	$85.13\pm 5.73$
MLP	$95.35\pm 1.73$	$95.55\pm 1.40$	$95.18\pm 1.88$	$95.30\pm 1.74$	$88.54\pm 2.08$	$88.74\pm 1.95$	$88.54\pm 2.08$	$88.53\pm 2.09$
Tensor Fusion	$95.50\pm 0.29$	$95.50\pm 0.27$	$95.45\pm 0.34$	$95.48\pm 0.31$	$88.54\pm 7.12$	$90.03\pm 6.46$	$88.54\pm 7.12$	$88.39\pm 7.26$
CMTF	$96.49\pm 1.15$	$96.47\pm 1.17$	$96.39\pm 1.16$	$96.42\pm 1.18$	$92.71\pm 2.08$	$93.38\pm 1.93$	$92.71\pm 2.08$	$92.68\pm 2.10$
Transformer	$96.27\pm 1.04$	$96.22\pm 1.07$	$96.23\pm 0.96$	$96.20\pm 1.03$	$85.42\pm 7.98$	$86.01\pm 8.61$	$85.42\pm 7.98$	$85.39\pm 7.95$

Table 3: Classification performance on SEED-IV and EEG-fMRI. Results are reported as mean

\pm

standard deviation. Bold values indicate the best performance for each metric.

Method	SEED-IV				EEG-fMRI
Method	Acc.	Precision	Recall	Macro-F1	Acc.	Precision	Recall	Macro-F1
Full RTR	$\mathbf{98.78\pm 0.12}$	$\mathbf{98.86\pm 0.16}$	$\mathbf{98.78\pm 0.14}$	$\mathbf{98.61\pm 0.10}$	$\mathbf{94.44\pm 1.96}$	$\mathbf{95.05\pm 1.55}$	$\mathbf{94.44\pm 1.96}$	$\mathbf{94.42\pm 1.99}$
w/o Dimension Bridge Matrix	$97.21\pm 0.54$	$97.21\pm 0.60$	$97.20\pm 0.48$	$97.20\pm 0.54$	$89.58\pm 4.66$	$89.73\pm 4.74$	$89.58\pm 4.66$	$89.58\pm 4.66$
w/o Modality Bridge Matrix	$96.66\pm 0.15$	$96.65\pm 0.13$	$96.61\pm 0.16$	$96.63\pm 0.14$	$83.33\pm 5.89$	$83.43\pm 5.90$	$83.33\pm 5.89$	$83.32\pm 5.89$
w/o Feature Fusion Module	$93.43\pm 0.88$	$93.50\pm 0.82$	$93.28\pm 0.84$	$93.36\pm 0.84$	$72.92\pm 7.12$	$72.99\pm 7.12$	$72.92\pm 7.12$	$72.89\pm 7.13$

Table 4: Ablation study of RTR on SEED-IV and EEG-fMRI. Results are reported as mean

\pm

standard deviation. Bold values indicate the best performance for each metric.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 335 papers

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
Oct 8, 2026Qun Dai, Liangjian Wen, Jiang Duan +7Multimodal SSL Multimodal Learning
School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd +5
Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling
Jun 13, 2026Zhemin Zhang, Weijie Chen, David Le +7Event Time Prediction Multimodal Robustness
Arizona State University · Mayo Clinic
Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention
Jul 21, 2026Joy Dhar, Manish Kumar Pandey, Nayyar Zaidi +4Cross-Modal Representation Learning Cross-Modal Attention
Indian Institute of Technology Ropar Rupnagar, Punjab, India · RoentGen Health New Delhi, Delhi, India · Deakin University Melbourne, Victoria, Australia +3

Oct 8, 2026J/K move · Enter open · S save

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Qun Dai, Liangjian Wen, Jiang Duan, Yong Dai+6

School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics · Artificial Intelligence and Digital Finance Key Laboratory of Sichuan Province · Chengdu Everimaging Science and Technology Co., Ltd+5

Multimodal SSL Multimodal Learning

Open paper AskPDF arXiv

Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.

Abstract

Multimodal Learning

Explore similar work

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling

Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention