Multimodal Deep Learning

Multimodal deep learning integrates data from diverse sources (e.g., images, text, audio) to build more robust and accurate predictive models than those using single data types. Current research emphasizes efficient fusion strategies (intermediate fusion being a prominent example), exploring various neural network architectures like CNNs, RNNs, and transformers, often incorporating attention mechanisms to weigh the importance of different modalities. This approach is significantly impacting various fields, including healthcare (improving diagnostics and prognostics), autonomous driving (sensor fusion), and scientific discovery (analyzing complex datasets), by enabling more comprehensive and insightful analyses.

48papers

Papers - Page 3

May 2, 2023

Multimodal Neural Databases
Giovanni Trappolini, Andrea Santilli, Emanuele Rodolà, Alon Halevy, Fabrizio Silvestri
Multimedia Retrieval Multimodal Neural Multimodal Deep Learning

March 29, 2023

A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials
Shun Muroga, Yasuaki Miki, Kenji Hata
Inverse Material Design Many Property Multimodal Deep Learning Novel Material

March 18, 2023

Just Noticeable Visual Redundancy Forecasting: A Deep Multimodal-driven Approach
Wuyuan Xie, Shukang Wang, Sukun Tian, Lirong Huang, Ye Liu, Miaohui Wang
Human Saliency Multimodal Deep Learning Multimodal Information Visual Adaptation

February 27, 2023

Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma
Tonmoy Hossain, Zoraiz Qureshi, Nivetha Jayakumar, Thomas Eluvathingal Muttikkal, Sohil Patel, David Schiff, Miaomiao Zhang, Bijoy Kundu
Multimodal Deep Learning Brain Tumor Glioblastoma Multiforme Treatment Effect FDG Pet Positron Emission Tomography

February 1, 2023

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, Shangsong Liang
Timely Survey Multimodal Representation Learning Multimodal Deep Learning Financial Application Specie Evolution Multi Modal Task Multimodal Learning Multimodal Chart

January 12, 2023

Multimodal Deep Learning
Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan+9
Deep Learning Different Modality Representation Learning Multimodal Deep Learning Multi Modal Model

December 29, 2022

Learning Multimodal Data Augmentation in Feature Space
Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, Andrew Gordon Wilson
Multimodal Data Augmentation Multimodal Data Feature Space Multimodal Deep Learning

December 18, 2022

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis
Firas Khader, Gustav Mueller-Franzes, Tianci Wang, Tianyu Han, Soroosh Tayebi Arasteh, Christoph Haarburger, Johannes Stegmaier, Keno Bressem+4
Diverse Datasets Medical Diagnosis Multimodal Input Multimodal Phenomenon Multimodal Datasets Deep Learning Multimodal Transformer Multimodal Deep Learning

November 4, 2022

Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions
Gaurav Verma, Vishwa Vinay, Ryan A. Rossi, Srijan Kumar
Multimodal Phenomenon Multimodal Deep Learning Native Robustness Multimodal Classification Multimodal Learning

September 12, 2022

August 23, 2022

Multimodal Across Domains Gaze Target Detection
Francesco Tonini, Cigdem Beyan, Elisa Ricci
Gaze Target Detection Multimodal Network Multimodal Phenomenon Multimodal Deep Learning

May 14, 2022

Classification of Astronomical Bodies by Efficient Layer Fine-Tuning of Deep Neural Networks
Sabeesh Ethiraj, Bharath Kumar Bolla
Deep Learning Fine Tuning Deep Neural Network Layer Selection Astronomical Data Multimodal Deep Learning Classification Code

April 7, 2022

Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes
Valerio Guarrasi, Paolo Soda
Covid 19 Multimodal Deep Learning Deep Network Multi Objective Optimization Modality Effect Application Proficiency Unimodal Model

February 18, 2022

A Review on Methods and Applications in Multimodal Deep Learning
Jabeen Summaira, Xi Li, Amin Muhammad Shoib, Jabbar Abdul
Multimodal Deep Learning NCD Method Financial Application Unimodal Learning Multimodal Learning

November 26, 2021

Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network
Maysam Behmanesh, Peyman Adibi, Mohammad Saeed Ehsani, Jocelyn Chanussot
Cross Modal Representation Multimodal Data Multimodal Deep Learning Multimodal Learning Method Multimodal Approach Graph Wavelet

Multimodal Deep Learning

Papers - Page 3

Multimodal Neural Databases

A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials

Just Noticeable Visual Redundancy Forecasting: A Deep Multimodal-driven Approach

Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Multimodal Deep Learning

Learning Multimodal Data Augmentation in Feature Space

Medical Diagnosis with Large Scale Multimodal Transformers: Leveraging Diverse Data for More Accurate Diagnosis

Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions

Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning

TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival Prediction

Multimodal Across Domains Gaze Target Detection

Classification of Astronomical Bodies by Efficient Layer Fine-Tuning of Deep Neural Networks

Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes

A Review on Methods and Applications in Multimodal Deep Learning

Geometric Multimodal Deep Learning with Multi-Scaled Graph Wavelet Convolutional Network