Multimodal Deep Learning

Multimodal deep learning integrates data from diverse sources (e.g., images, text, audio) to build more robust and accurate predictive models than those using single data types. Current research emphasizes efficient fusion strategies (intermediate fusion being a prominent example), exploring various neural network architectures like CNNs, RNNs, and transformers, often incorporating attention mechanisms to weigh the importance of different modalities. This approach is significantly impacting various fields, including healthcare (improving diagnostics and prognostics), autonomous driving (sensor fusion), and scientific discovery (analyzing complex datasets), by enabling more comprehensive and insightful analyses.

Papers

January 15, 2024

Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learning
Rahul Vishwakarma, Amin Rezaei
Generative Adversarial Network Multimodal Deep Learning Hardware Trojan Hardware Trojan Detection Hardware Security

December 22, 2023

Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment
Zhiyuan Li, Hailong Li, Anca L. Ralescu, Jonathan R. Dillman, Mekibib Altaye, Kim M. Cecil, Nehal A. Parikh, Lili He
Multimodal Data Functional Magnetic Resonance Imaging Supervised Contrastive Learning Multimodal Feature Multimodal Deep Learning Neurodevelopmental Disorder

December 4, 2023

TextAug: Test time Text Augmentation for Multimodal Person Re-identification
Mulham Fawakherji, Eduard Vazquez, Pasquale Giampa, Binod Bhattarai
Person Re Identification Text Augmentation Multimodal Deep Learning Test Time Augmentation ID Benchmark Multimodal Re Identification

November 11, 2023

Asymmetric Contrastive Multimodal Learning for Advancing Chemical Understanding
Hao Xu, Yifei Wang, Yunrui Li, Lin Liu, Pengyu Hong
Contrastive Learning Multimodal Deep Learning Chemical Knowledge Chemical Named Entity Recognition Asymmetric Contrastive Learning

October 27, 2023

MalFake: A Multimodal Fake News Identification for Malayalam using Recurrent Neural Networks and VGG-16
Adhish S. Sujan, Ajitha. V, Aleena Benny, Amiya M. P., V. S. Anoop
Recurrent Neural Network Fake News Fake News Detection Multi Modality Misinformation Detection Multimodal Approach Multimodal Deep Learning

October 2, 2023

HyMNet: a Multimodal Deep Learning System for Hypertension Classification using Fundus Photographs and Cardiometabolic Risk Factors
Mohammed Baharoon, Hessa Almatar, Reema Alduhayan, Tariq Aldebasi, Badr Alahmadi, Yahya Bokhari, Mohammed Alawad, Ahmed Almazroa, Abdulrhman Aljouie
Fundus Image Multimodal Deep Learning Blood Pressure Cardiovascular Disease Risk Hypertension Subtypes

September 21, 2023

Multimodal Deep Learning for Scientific Imaging Interpretation
Abdulelah S. Alshehri, Franklin L. Lee, Shihu Wang
Visual Data Multimodal Deep Learning SEM Image Scientific Image

September 6, 2023

3D Object Positioning Using Differentiable Multimodal Learning
Sean Zanyk-McLean, Krishna Kumar, Paul Navratil
Multi Modal Mid Range LiDAR 3D Object LiDAR Data Multimodal Deep Learning

July 14, 2023

A scoping review on multimodal deep learning in biomedical images and texts
Zhaoyi Sun, Mingquan Lin, Qingqing Zhu, Qianqian Xie, Fei Wang, Zhiyong Lu, Yifan Peng
Narrative Review Multimodal Data Medical Imaging Text Based Multimodal Deep Learning Biomedical Image

July 10, 2023

Rapid Deforestation and Burned Area Detection using Deep Multimodal Learning on Satellite Imagery
Gabor Fodor, Marcos V. Conde
Satellite Imagery Multimodal Deep Learning Deforestation Area Deforestation Estimation

July 7, 2023

Multimodal Deep Learning for Personalized Renal Cell Carcinoma Prognosis: Integrating CT Imaging and Clinical Data
Maryamalsadat Mahootiha, Hemin Ali Qadir, Jacob Bergsland, Ilangko Balasingham
Computed Tomography Survival Prediction Multimodal Deep Learning Clinical Data Renal Cell Carcinoma

July 6, 2023

A Novel Site-Agnostic Multimodal Deep Learning Model to Identify Pro-Eating Disorder Content on Social Media
Jonathan Feldman
Deep Learning Deep Learning Model Social Medium Multimodal Deep Learning

June 28, 2023

MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Arav Agarwal, Yun Cheng, Louis-Philippe Morency, Ruslan Salakhutdinov
Multimodal Model Multimodal Representation Multimodal Deep Learning Modality Robustness

May 2, 2023

Multimodal Neural Databases
Giovanni Trappolini, Andrea Santilli, Emanuele Rodolà, Alon Halevy, Fabrizio Silvestri
Multimodal Deep Learning Multimedia Retrieval Multimodal Neural

March 29, 2023

A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials
Shun Muroga, Yasuaki Miki, Kenji Hata
Many Property Multimodal Deep Learning Novel Material Inverse Material Design

March 18, 2023

Just Noticeable Visual Redundancy Forecasting: A Deep Multimodal-driven Approach
Wuyuan Xie, Shukang Wang, Sukun Tian, Lirong Huang, Ye Liu, Miaohui Wang
Multimodal Information Human Saliency Multimodal Deep Learning Visual Adaptation

February 27, 2023

Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma
Tonmoy Hossain, Zoraiz Qureshi, Nivetha Jayakumar, Thomas Eluvathingal Muttikkal, Sohil Patel, David Schiff, Miaomiao Zhang, Bijoy Kundu
Treatment Effect Brain Tumor Positron Emission Tomography Multimodal Deep Learning Glioblastoma Multiforme FDG Pet

February 1, 2023

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, Shangsong Liang
Timely Survey Financial Application Multimodal Learning Specie Evolution Multimodal Deep Learning Multimodal Representation Learning Multimodal Chart Multi Modal Task

January 12, 2023

Multimodal Deep Learning
Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schneider, Rickmer Schulte, Karol Urbanczyk, Jann Goschenhofer, Christian Heumann, Rasmus Hvingelby, Daniel Schalk, Matthias Aßenmacher
Deep Learning Representation Learning Different Modality Multi Modal Model Multimodal Deep Learning

December 29, 2022

Learning Multimodal Data Augmentation in Feature Space
Zichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, Andrew Gordon Wilson
Multimodal Data Feature Space Multimodal Deep Learning Multimodal Data Augmentation

Multimodal Deep Learning

Papers

Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learning

Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment

TextAug: Test time Text Augmentation for Multimodal Person Re-identification

Asymmetric Contrastive Multimodal Learning for Advancing Chemical Understanding

MalFake: A Multimodal Fake News Identification for Malayalam using Recurrent Neural Networks and VGG-16

HyMNet: a Multimodal Deep Learning System for Hypertension Classification using Fundus Photographs and Cardiometabolic Risk Factors

Multimodal Deep Learning for Scientific Imaging Interpretation

3D Object Positioning Using Differentiable Multimodal Learning

A scoping review on multimodal deep learning in biomedical images and texts

Rapid Deforestation and Burned Area Detection using Deep Multimodal Learning on Satellite Imagery

Multimodal Deep Learning for Personalized Renal Cell Carcinoma Prognosis: Integrating CT Imaging and Clinical Data

A Novel Site-Agnostic Multimodal Deep Learning Model to Identify Pro-Eating Disorder Content on Social Media

MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning

Multimodal Neural Databases

A Comprehensive and Versatile Multimodal Deep Learning Approach for Predicting Diverse Properties of Advanced Materials

Just Noticeable Visual Redundancy Forecasting: A Deep Multimodal-driven Approach

Multimodal Deep Learning to Differentiate Tumor Recurrence from Treatment Effect in Human Glioblastoma

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Multimodal Deep Learning

Learning Multimodal Data Augmentation in Feature Space