Multimodal Deep Learning

Multimodal deep learning integrates data from diverse sources (e.g., images, text, audio) to build more robust and accurate predictive models than those using single data types. Current research emphasizes efficient fusion strategies (intermediate fusion being a prominent example), exploring various neural network architectures like CNNs, RNNs, and transformers, often incorporating attention mechanisms to weigh the importance of different modalities. This approach is significantly impacting various fields, including healthcare (improving diagnostics and prognostics), autonomous driving (sensor fusion), and scientific discovery (analyzing complex datasets), by enabling more comprehensive and insightful analyses.

Papers

September 27, 2024

Reducing Overtreatment of Indeterminate Thyroid Nodules Using a Multimodal Deep Learning Model
Shreeram Athreya, Andrew Melehy, Sujit Silas Armstrong Suthahar, Vedrana Ivezić, Ashwath Radhachandran, Vivek Sant, Chace Moleta, Henry Zheng, Maitraya Patel, Rinat Masamed, Corey W. Arnold, William Speier
Multimodal Deep Learning Thyroid Cancer

September 20, 2024

Validation & Exploration of Multimodal Deep-Learning Camera-Lidar Calibration models
Venkat Karramreddy, Liam Mitchell
Environment Exploration Deep Learning Architecture External Validation Sensor Fusion Multimodal Deep Learning 3D LiDAR LiDAR Calibration LiDAR Camera Extrinsic Calibration

September 12, 2024

Deep Multimodal Learning with Missing Modality: A Survey
Renjie Wu, Hu Wang, Hsiang-Ting Chen, Gustavo Carneiro
Deep Learning Comprehensive Survey Multimodal Learning Missing Modality Multimodal Deep Learning

August 2, 2024

A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications
Valerio Guarrasi, Fatih Aksu, Camillo Maria Caruso, Francesco Di Feola, Aurora Rofena, Filippo Ruffini, Paolo Soda
Multimodal Model Systematic Review Modality Specific Multimodal Deep Learning Biomedical Application Intermediate Fusion

June 14, 2024

LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Data
Grigor Bezirganyan, Sana Sellami, Laure Berti-Équille, Sébastien Fournier
Multimodal Data MAESTRO Dataset Benchmark Dataset Deep Ensemble Multimodal Deep Learning Uncertain Reasoning

June 13, 2024

June 2, 2024

Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications
David Restrepo, Chenwei Wu, Sebastián Andrés Cajas, Luis Filipe Nakayama, Leo Anthony Celi, Diego M López
Low Resource Multimodal Learning X Vector Multimodal Deep Learning Alignment Approach Healthcare Application Multi Modal Deep Learning Vector Embeddings Embedding Based

May 23, 2024

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis
Ziyan Yao, Fei Lin, Sheng Chai, Weijie He, Lu Dai, Xinghui Fei
Deep Learning Model Medical Image Multimodal Phenomenon Multimodal Deep Learning Imaging Task

May 21, 2024

The Role of Emotions in Informational Support Question-Response Pairs in Online Health Communities: A Multimodal Deep Learning Approach
Mohsen Jozani, Jason A. Williams, Ahmed Aleroud, Sarbottam Bhagat
Integral Role Experienced Emotion Question Answer Pair Multimodal Deep Learning Information Seeking Social Cognition Online Health

May 6, 2024

Research on Image Recognition Technology Based on Multimodal Deep Learning
Jinyin Wang, Xingchen Li, Yixuan Jin, Yihao Zhong, Keke Zhang, Chang Zhou
Deep Learning Model DH Research Image Recognition Multimodal Deep Learning Pedestrian Behavior

May 2, 2024

A Toolchain for Comprehensive Audio/Video Analysis Using Deep Learning Based Multimodal Approach (A use case of riot or violent context detection)
Lam Pham, Phat Lam, Tin Nguyen, Hieu Tang, Alexander Schindler
Multimodal Approach Acoustic Scene Classification Multimodal Deep Learning Video Analysis Violence Detection

April 29, 2024

Feature importance to explain multimodal prediction models. A clinical use case
Jorn-Jan van de Beld, Shreyasi Pathak, Jeroen Geerdink, Johannes H. Hegeman, Christin Seifert
Feature Importance Multimodal Deep Learning Explainable Model Surgical Datasets Clinical Application Multimodal Prediction Postoperative Complication Hip Fracture

March 16, 2024

Integrating Wearable Sensor Data and Self-reported Diaries for Personalized Affect Forecasting
Zhongqi Yang, Yuning Wang, Ken S. Yamashita, Maryam Sabah, Elahe Khatibi, Iman Azimi, Nikil Dutt, Jessica L. Borelli, Amir M. Rahmani
Wearable Sensor Data Multimodal Deep Learning Positive Emotional Language Emotional State Emotion Estimation Self Reported Diary

March 12, 2024

A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection
Morteza Bodaghi, Majid Hosseini, Raju Gottumukkala
Manifold Learning Dimension Reduction Stress Detection Multimodal Deep Learning Multimodal Network Multimodal Fusion Network

March 11, 2024

Restoring Ancient Ideograph: A Multimodal Multitask Neural Network Approach
Siyu Duan, Jun Wang, Qi Su
Multimodal Information Multimodal Deep Learning Ancient Text Text Restoration Multi Task Image Restoration Logographic Writing System

February 28, 2024

Multimodal Learning To Improve Cardiac Late Mechanical Activation Detection From Cine MR Images
Jiarui Xing, Nian Wu, Kenneth Bilchick, Frederick Epstein, Miaomiao Zhang
Multimodal Learning Image Registration Multimodal Deep Learning Cine Magnetic Resonance Cardiac Motion Latent Motion Myocardial Strain

January 22, 2024

Multimodal Deep Learning of Word-of-Mouth Text and Demographics to Predict Customer Rating: Handling Consumer Heterogeneity in Marketing
Junichiro Niimi
Text Modality Multimodal Phenomenon Multimodal Learning Multimodal Deep Learning Heterogeneous User

January 18, 2024

Bootstrapping OTS-Funcimg Pre-training Model (Botfip) -- A Comprehensive Symbolic Regression Framework
Tianhao Chen, Pengbo Xu, Haibiao Zheng
Symbolic Regression Multimodal Deep Learning Multimodal Framework Scientific Computing

January 15, 2024

Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learning
Rahul Vishwakarma, Amin Rezaei
Generative Adversarial Network Multimodal Deep Learning Hardware Trojan Hardware Trojan Detection Hardware Security

Multimodal Deep Learning

Papers

Reducing Overtreatment of Indeterminate Thyroid Nodules Using a Multimodal Deep Learning Model

Validation & Exploration of Multimodal Deep-Learning Camera-Lidar Calibration models

Deep Multimodal Learning with Missing Modality: A Survey

A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications

LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Data

Advanced Multimodal Deep Learning Architecture for Image-Text Matching

Research on Optimization of Natural Language Processing Model Based on Multimodal Deep Learning

Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

The Role of Emotions in Informational Support Question-Response Pairs in Online Health Communities: A Multimodal Deep Learning Approach

Research on Image Recognition Technology Based on Multimodal Deep Learning

A Toolchain for Comprehensive Audio/Video Analysis Using Deep Learning Based Multimodal Approach (A use case of riot or violent context detection)

Feature importance to explain multimodal prediction models. A clinical use case

Integrating Wearable Sensor Data and Self-reported Diaries for Personalized Affect Forecasting

A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection

Restoring Ancient Ideograph: A Multimodal Multitask Neural Network Approach

Multimodal Learning To Improve Cardiac Late Mechanical Activation Detection From Cine MR Images

Multimodal Deep Learning of Word-of-Mouth Text and Demographics to Predict Customer Rating: Handling Consumer Heterogeneity in Marketing

Bootstrapping OTS-Funcimg Pre-training Model (Botfip) -- A Comprehensive Symbolic Regression Framework

Uncertainty-Aware Hardware Trojan Detection Using Multimodal Deep Learning