Multimodal Learning

Multimodal learning aims to improve machine learning performance by integrating data from multiple sources, such as text, images, and audio, to create richer, more robust representations. Current research focuses on addressing challenges like missing modalities (developing models resilient to incomplete data), modality imbalance (ensuring fair contribution from all modalities), and efficient fusion techniques (e.g., dynamic anchor methods, single-branch networks, and various attention mechanisms). This field is significant because it enables more accurate and contextually aware systems across diverse applications, including healthcare diagnostics, recommendation systems, and video understanding.

Papers

May 23, 2023

Counterfactual Augmentation for Multimodal Learning Under Presentation Bias
Victoria Lin, Louis-Philippe Morency, Dimitrios Dimitriadis, Srinagesh Sharma
Multimodal Learning High Quality Counterfactuals Bias Correction Representation Bias Counterfactual Augmentation

May 12, 2023

ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter
Zhengqing Yuan, Yunhong He, Kun Wang, Yanfang Ye, Lichao Sun
Large Vision Language Model Multimodal Learning Image Text Pair Long Short Range Adapter

May 11, 2023

OneCAD: One Classifier for All image Datasets using multimodal learning
Shakti N. Wadekar, Eugenio Culurciello
Convolutional Neural Network Multimodal Learning Simple Classifier Image Datasets Model Agnostic DNN Architecture Model Architecture

May 4, 2023

Multimodal Understanding Through Correlation Maximization and Minimization
Yifeng Shi, Marc Niethammer
Latent Representation Multimodal Data Multimodal Learning Multimodal Understanding Correlation Function

April 17, 2023

MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning
Shicai Wei, Yang Luo, Chunbo Luo
Multimodal Data Multimodal Learning Multimodal Classification Prior Distillation Modality Combination

April 13, 2023

April 10, 2023

On Robustness in Multimodal Learning
Brandon McKinzie, Joseph Cheng, Vaishaal Shankar, Yinfei Yang, Jonathon Shlens, Alexander Toshev
Native Robustness Multimodal Learning Multimodal Representation Learning Input Modality Robustness Issue MUltimodal RObustness

April 7, 2023

April 6, 2023

Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification
Thomas Z. Li, John M. Still, Kaiwen Xu, Ho Hin Lee, Leon Y. Cai, Aravind R. Krishnan, Riqiang Gao, Mirza S. Khan, Sanja Antic, Michael Kammer, Kim L. Sandler, Fabien Maldonado, Bennett A. Landman, Thomas A. Lasko
Electronic Health Record Multimodal Learning Nodule Classification Clinical Signature

March 31, 2023

Self-Supervised Multimodal Learning: A Survey
Yongshuo Zong, Oisin Mac Aodha, Timothy Hospedales
Timely Survey Multimodal Data Multimodal Learning Multimodal Self Supervised Learning

March 27, 2023

Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation
Rongyu Zhang, Xiaowei Chi, Guiliang Liu, Wenyi Zhang, Yuan Du, Fangxin Wang
Federated Learning Multimodal Model Multimodal Learning Unimodal Model Multimodal Prediction Level Aggregation

March 22, 2023

BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency
Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, Min Xu
Cross Modal Multimodal Learning Multi Modal Data Cross Modal Matching

March 16, 2023

March 11, 2023

Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review
Asim Waqas, Aakash Tripathi, Ravi P. Ramachandran, Paul Stewart, Ghulam Rasool
Deep Neural Network Narrative Review Multimodal Learning New Era Cancer Patient Multimodal Data Fusion Multimodal Neural Network Multimodal Data Integration

February 24, 2023

Revisiting Modality Imbalance In Multimodal Pedestrian Detection
Arindam Das, Sudip Das, Ganesh Sistu, Jonathan Horgan, Ujjwal Bhattacharya, Edward Jones, Martin Glavin, Ciarán Eising
Multimodal Learning Pedestrian Detection Input Modality Multimodal Distribution Multimodal Architecture Modality Imbalance

February 4, 2023

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu
Real World Audio Visual Multimodal Learning Neural Field NeRF SLAM Spatial Audio Audio Driven Visual Synthesis

Multimodal Learning

Papers

Counterfactual Augmentation for Multimodal Learning Under Presentation Bias

ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter

OneCAD: One Classifier for All image Datasets using multimodal learning

Multimodal Understanding Through Correlation Maximization and Minimization

MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning

Efficient Multimodal Fusion via Interactive Prompting

Automated Cardiovascular Record Retrieval by Multimodal Learning between Electrocardiogram and Clinical Report

Noisy Correspondence Learning with Meta Similarity Correction

On Robustness in Multimodal Learning

On the Importance of Contrastive Loss in Multimodal Learning

Linking Representations with Multimodal Contrastive Learning

Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification

Self-Supervised Multimodal Learning: A Survey

Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation

BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency

Identifiability Results for Multimodal Contrastive Learning

Preoperative Prognosis Assessment of Lumbar Spinal Surgery for Low Back Pain and Sciatica Patients based on Multimodalities and Multimodal Learning

Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review

Revisiting Modality Imbalance In Multimodal Pedestrian Detection

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis