Multimodal Capability

Multimodal capability refers to the ability of artificial intelligence systems to process and integrate information from multiple sources, such as text, images, audio, and sensor data. Current research focuses on developing and evaluating large multimodal models, often based on transformer architectures, that can effectively fuse these diverse data types for tasks like visual question answering, image captioning, and report generation. This field is significant because it pushes the boundaries of AI towards more human-like understanding and reasoning, with applications ranging from improved recommendation systems to advanced medical image analysis and more robust driver monitoring systems. The development of effective evaluation benchmarks is also a key area of ongoing research, aiming to ensure fair and accurate comparisons between different multimodal models.

Papers

June 15, 2023

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, Ping Luo
Zero Shot Large Vision Language Model Large Multimodal Model Evaluation Benchmark Multimodal Capability Multimodal Vision Language Model

May 24, 2023

Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, Rongrong Ji
Multimodal Large Language Model Multimodal Dialogue Low Cost Multimodal Capability Vision Language Instruction Tuning Large Vision Language

April 13, 2023

Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention
Yiming Ma, Victor Sanchez, Soodeh Nikan, Devesh Upadhyay, Bhushan Atote, Tanaya Guha
Feature Fusion Multi Head Self Attention Multimodal Capability Decision Level Fusion Driver Monitoring System

December 31, 2022

Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching
Jiaqing Zhang, Jie Lei, Weiying Xie, Yunsong Li, Xiuping Jia
Knowledge Distillation Visual Imagery Multimodal Capability Mixed Quantization Quantization Model Quantization Aware Knowledge Distillation One to One

October 12, 2022

That's the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Data
Denis Jered McInerney, Geoffrey Young, Jan-Willem van de Meent, Byron C. Wallace
Inherent Interpretability Multimodal Model Human Annotation Medical Data Multimodal Capability Multimodal Encoder Soft Alignment

April 1, 2022

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence
Large Language Model Zero Shot Human Language Visual Language Model Multimodal Reasoning Multimodal Capability Multimodal Prompting Socratic Model

March 22, 2022

Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers
Xiang Zhang, Lijun Yin
Multi Modality Multi Modal Learning Multimodal Capability AU Detection

Multimodal Capability

Papers

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models

Robust Multiview Multimodal Driver Monitoring System Using Masked Multi-Head Self-Attention

Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching

That's the Wrong Lung! Evaluating and Improving the Interpretability of Unsupervised Multimodal Encoders for Medical Data

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers