Cross Modal Prediction

Cross-modal prediction focuses on leveraging information from multiple data sources (modalities) to improve prediction accuracy in a target modality. Current research emphasizes developing robust methods for integrating diverse data types, often employing deep learning architectures like variational autoencoders and contrastive learning frameworks to disentangle shared and private information across modalities. This work is significant for diverse applications, including improved weather forecasting from sparse data, enhanced recommendation systems through visual feature integration, and more accurate multimodal sentiment analysis, ultimately leading to more powerful and reliable predictive models across various scientific and engineering domains.

Papers

March 10, 2024

Disentangling shared and private latent factors in multimodal Variational Autoencoders
Kaspar Märtens, Christopher Yau
Multimodal Data Modality Specific Latent Factor Multimodal Variational AutoEncoders Cross Modal Prediction

January 30, 2024

Multi-modal Representation Learning for Cross-modal Prediction of Continuous Weather Patterns from Discrete Low-Dimensional Data
Alif Bin Abdul Qayyum, Xihaier Luo, Nathan M. Urban, Xiaoning Qian, Byung-Jun Yoon
Dimension Reduction Multi Modal Representation Learning Climate Pattern Wind Data Low Dimensional Data Resolution Wind Cross Modal Prediction

June 8, 2023

COURIER: Contrastive User Intention Reconstruction for Large-Scale Visual Recommendation
Jia-Qi Yang, Chenglei Dai, Dan OU, Dongshuai Li, Ju Huang, De-Chuan Zhan, Xiaoyi Zeng, Yang Yang
Click Through Rate Prediction Visual Feature Cross Modal Prediction

October 26, 2022

Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis
Ronghao Lin, Haifeng Hu
Contrastive Learning Multimodal Representation Multimodal Sentiment Analysis Multimodal Representation Learning Multimodal Joint Cross Modal Prediction

July 22, 2022

Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction
Hu Wang, Jianpeng Zhang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, Gustavo Carneiro
Multi Modal Data Multi Modal Learning Input Modality Cross Modal Prediction

Cross Modal Prediction

Papers

Disentangling shared and private latent factors in multimodal Variational Autoencoders

Multi-modal Representation Learning for Cross-modal Prediction of Continuous Weather Patterns from Discrete Low-Dimensional Data

COURIER: Contrastive User Intention Reconstruction for Large-Scale Visual Recommendation

Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis

Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Prediction