Multimodal Deep Learning

Multimodal deep learning integrates data from diverse sources (e.g., images, text, audio) to build more robust and accurate predictive models than those using single data types. Current research emphasizes efficient fusion strategies (intermediate fusion being a prominent example), exploring various neural network architectures like CNNs, RNNs, and transformers, often incorporating attention mechanisms to weigh the importance of different modalities. This approach is significantly impacting various fields, including healthcare (improving diagnostics and prognostics), autonomous driving (sensor fusion), and scientific discovery (analyzing complex datasets), by enabling more comprehensive and insightful analyses.

Papers

December 18, 2022

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis
Firas Khader, Gustav Mueller-Franzes, Tianci Wang, Tianyu Han, Soroosh Tayebi Arasteh, Christoph Haarburger, Johannes Stegmaier, Keno Bressem, Christiane Kuhl, Sven Nebelung, Jakob Nikolas Kather, Daniel Truhn
Deep Learning Multimodal Phenomenon Medical Diagnosis Multimodal Transformer Diverse Datasets Multimodal Input Multimodal Deep Learning Multimodal Datasets

November 4, 2022

Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions
Gaurav Verma, Vishwa Vinay, Ryan A. Rossi, Srijan Kumar
Native Robustness Multimodal Phenomenon Multimodal Learning Multimodal Deep Learning Multimodal Classification

September 12, 2022

August 23, 2022

Multimodal Across Domains Gaze Target Detection
Francesco Tonini, Cigdem Beyan, Elisa Ricci
Multimodal Phenomenon Multimodal Deep Learning Multimodal Network Gaze Target Detection

May 14, 2022

Classification of Astronomical Bodies by Efficient Layer Fine-Tuning of Deep Neural Networks
Sabeesh Ethiraj, Bharath Kumar Bolla
Deep Learning Deep Neural Network Fine Tuning Classification Code Astronomical Data Multimodal Deep Learning Layer Selection

April 7, 2022

Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes
Valerio Guarrasi, Paolo Soda
Application Proficiency Covid 19 Deep Network Multi Objective Optimization Unimodal Model Multimodal Deep Learning Modality Effect

February 18, 2022

A Review on Methods and Applications in Multimodal Deep Learning
Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Jabbar Abdul
Financial Application NCD Method Multimodal Learning Multimodal Deep Learning Unimodal Learning

November 26, 2021

Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network
Maysam Behmanesh, Peyman Adibi, Mohammad Saeed Ehsani, Jocelyn Chanussot
Multimodal Data Cross Modal Representation Multimodal Approach Multimodal Deep Learning Graph Wavelet Multimodal Learning Method

Multimodal Deep Learning

Papers

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis

Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions

Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning

TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival Prediction

Multimodal Across Domains Gaze Target Detection

Classification of Astronomical Bodies by Efficient Layer Fine-Tuning of Deep Neural Networks

Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes

A Review on Methods and Applications in Multimodal Deep Learning

Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network