Multimodal Decision

Multimodal decision-making focuses on developing systems that integrate information from multiple sources (e.g., vision, language, audio) to make informed decisions, mirroring human cognitive processes. Current research emphasizes improving the robustness and accuracy of these systems, exploring techniques like adversarial perturbation analysis to identify vulnerabilities and developing novel hybrid training frameworks to enhance performance on complex tasks. This field is crucial for advancing AI in various applications, from improving healthcare diagnostics and emergency response to creating more sophisticated and reliable autonomous systems. The development of robust benchmarks and evaluation protocols is also a key area of focus, enabling more rigorous comparison and progress in the field.

Papers

August 6, 2024

Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models
Haonan Zheng, Wen Jiang, Xinyang Deng, Wenrui Li
Adversarial Attack Full Model Adversarial Sample Adversarial Patch Visual Language Task Multimodal Decision

July 25, 2024

MDS-ED: Multimodal Decision Support in the Emergency Department -- a Benchmark Dataset for Diagnoses and Deterioration Prediction in Emergency Medicine
Juan Miguel Lopez Alcaraz, Hjalmar Bouma, Nils Strodthoff
New Benchmark Benchmark Dataset Emergency Department Multimodal Diagnosis Multimodal Decision Medical Decision Deterioration Prediction

July 3, 2024

VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values
Zhe Hu, Yixiao Ren, Jing Li, Yu Yin
New Benchmark Large Vision Language Model Human Value Multimodal Capability Multimodal Decision

February 21, 2024

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Xiangdi Meng, Tianyu Liu, Baobao Chang
Multimodal Large Language Model PCa Detection Embodied Instruction Multimodal Decision Action Perception

July 16, 2023

Breaking Down the Task: A Unit-Grained Hybrid Training Framework for Vision and Language Decision Making
Ruipu Luo, Jiwen Zhang, Zhongyu Wei
Vision Language Vision Paper Related Task Unknown Language Multimodal Problem Multimodal Decision

May 20, 2023

Efficient Multimodal Neural Networks for Trigger-less Voice Assistants
Sai Srujana Buddi, Utkarsh Oggy Sarawgi, Tashweena Heeramun, Karan Sawnhey, Ed Yanosik, Saravana Rathinam, Saurabh Adya
Voice Assistant Multimodal Interaction Gesture Data Audio Gesture Multimodal Decision Multimodal Neural

January 18, 2022

Combining Fast and Slow Thinking for Human-like and Efficient Navigation in Constrained Environments
Marianna B. Ganapini, Murray Campbell, Francesco Fabiano, Lior Horesh, Jon Lenchner, Andrea Loreggia, Nicholas Mattei, Taher Rahgooy, Francesca Rossi, Biplav Srivastava, Brent Venable
AI System Human Like Human Decision Efficient Navigation Select and Combine Constrained Environment Metacognitive Difficulty Multimodal Decision

December 17, 2021

A Multimodal Approach for Automatic Mania Assessment in Bipolar Disorder
Pınar Baki
Multimodal Approach Multimodal System Multimodal Decision

Multimodal Decision

Papers

Sample-agnostic Adversarial Perturbation for Vision-Language Pre-training Models

MDS-ED: Multimodal Decision Support in the Emergency Department -- a Benchmark Dataset for Diagnoses and Deterioration Prediction in Emergency Medicine

VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

Breaking Down the Task: A Unit-Grained Hybrid Training Framework for Vision and Language Decision Making

Efficient Multimodal Neural Networks for Trigger-less Voice Assistants

Combining Fast and Slow Thinking for Human-like and Efficient Navigation in Constrained Environments

A Multimodal Approach for Automatic Mania Assessment in Bipolar Disorder