Robust Multimodal

Robust multimodal learning aims to build systems that effectively combine information from multiple data sources (e.g., images, text, audio) while maintaining performance even when some data is missing or corrupted. Current research focuses on developing model architectures and training strategies that address this robustness challenge, including methods that leverage masked modality projection, conditional hypernetworks, and representation decoupling to handle varying input sizes and missing modalities. This field is significant because it enables the development of more reliable and practical AI systems for applications like healthcare diagnostics, sentiment analysis, and autonomous driving, where incomplete or noisy data is common.

Papers

October 3, 2024

MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection
Niki Nezakati, Md Kaykobad Reza, Ameya Patil, Mashhour Solh, M. Salman Asif
Multimodal Learning Multimodal Input Missing Modality Robust Multimodal Masked Multimodal Modality Robustness

July 30, 2024

HyperMM : Robust Multimodal Learning with Varying-sized Inputs
Hava Chaptoukaev, Vincenzo Marcianó, Francesco Galati, Maria A. Zuluaga
Multimodal Model Multimodal Learning Multi Modality Missing Modality Robust Multimodal Modality Imputation Resized Diverse Input

July 23, 2024

Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities
Muhammad Irzam Liaqat, Shah Nawaz, Muhammad Zaigham Zaheer, Muhammad Saad Saeed, Hassan Sajjad, Tom De Schepper, Karthik Nandakumar, Muhammad Haris Khan Markus Schedl
Text Modality Multimodal Learning Different Modality Pace Adaptive Robust Multimodal Modality Agnostic Multimodal Learning Method

July 5, 2024

Robust Multimodal Learning via Representation Decoupling
Shicai Wei, Yang Luo, Yuji Wang, Chunbo Luo
Multimodal Representation Learning Multimodal Classification Robust Multimodal Modality Specific Information Modality Combination

June 20, 2024

Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning
Yupei Zhang, Xiaofei Wang, Fangliangzi Meng, Jin Tang, Chao Li
Multi Modal Learning Deep Subspace Tumor Microenvironment Robust Multimodal

April 2, 2024

CATP: Cross-Attention Token Pruning for Accuracy Preserved Multimodal Model Inference
Ruqi Liao, Chuqing Zhao, Jin Li, Weiqi Feng
Token Pruning Robust Multimodal Token Importance

February 9, 2024

Quantifying and Enhancing Multi-modal Robustness with Modality Preference
Zequn Yang, Yake Wei, Ce Liang, Di Hu
Multi Modal Model Multimodal Joint Robust Multimodal MUltimodal RObustness Modality Competition

January 20, 2024

Toward Robust Multimodal Learning using Multimodal Foundational Models
Xianbing Zhao, Soujanya Poria, Xuejiao Li, Yixin Chen, Buzhou Tang
Multimodal Data Multimodal Sentiment Robust Multimodal Modality Representation Multimodal Foundation

October 19, 2023

Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts
Jonathan Crabbé, Pau Rodríguez, Vaishaal Shankar, Luca Zappella, Arno Blaas
Native Robustness Large Multimodal Model Representation Space Robust Model Robust Multimodal Outlier Explanation

October 6, 2023

Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Md Kaykobad Reza, Ashley Prater-Bennette, M. Salman Asif
Multimodal Learning Missing Modality Multimodal Network Parameter Efficient Adaptation Robust Multimodal Modality Robustness Multimodal Semantic Segmentation

July 7, 2023

Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection
Yinghan Guan, Haoran Dai, Zekuan Yu, Shouyu Wang, Yuanjie Gu
Multi Modal Wildfire Detection Robust Multimodal Smoke Detection Fusion Detection

June 2, 2023

Calibrating Multimodal Learning
Huan Ma. Qingyang Zhang, Changqing Zhang, Bingzhe Wu, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu
Multimodal Learning Confidence Calibration Multimodal Machine Learning Multimodal Classification Robust Multimodal

May 30, 2023

Trustworthy Sensor Fusion against Inaudible Command Attacks in Advanced Driver-Assistance System
Jiwei Guan, Lei Pan, Chen Wang, Shui Yu, Longxiang Gao, Xi Zheng
Audio Visual Multimodal Fusion Sensor Fusion Driver Assistance System Robust Multimodal Inaudible Voice

December 15, 2022

Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift
Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, Mu Li
Native Robustness Full Model Distribution Shift Multimodal Phenomenon Multimodal Model Image Text Model Robust Multimodal MUltimodal RObustness

May 30, 2022

Analyzing Modality Robustness in Multimodal Sentiment Analysis
Devamanyu Hazarika, Yingting Li, Bo Cheng, Shuai Zhao, Roger Zimmermann, Soujanya Poria
Multimodal Model Multimodal Sentiment Analysis Robust Multimodal Modality Robustness

April 5, 2022

Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization
Zhengqi Gao, Sucheng Ren, Zihui Xue, Siting Li, Hang Zhao
Native Robustness Multimodal Fusion Fusion Method Late Fusion Robust Multimodal Jacobian Regularization

Robust Multimodal

Papers

MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection

HyperMM : Robust Multimodal Learning with Varying-sized Inputs

Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities

Robust Multimodal Learning via Representation Decoupling

Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning

CATP: Cross-Attention Token Pruning for Accuracy Preserved Multimodal Model Inference

Quantifying and Enhancing Multi-modal Robustness with Modality Preference

Toward Robust Multimodal Learning using Multimodal Foundational Models

Interpreting CLIP: Insights on the Robustness to ImageNet Distribution Shifts

Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation

Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection

Calibrating Multimodal Learning

Trustworthy Sensor Fusion against Inaudible Command Attacks in Advanced Driver-Assistance System

Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift

Analyzing Modality Robustness in Multimodal Sentiment Analysis

Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization