Multimodal Task

Multimodal tasks involve integrating information from multiple sources like text, images, and audio to perform complex reasoning and generation. Current research focuses on developing and evaluating large multimodal models (LLMs) using techniques like next-token prediction, prompt tuning, and mixture-of-experts architectures to improve efficiency and performance across diverse tasks, including visual question answering and image captioning. These advancements are significant for improving the capabilities of AI systems in various fields, particularly those requiring the interpretation and generation of multimodal data, such as healthcare and insurance. Addressing challenges like hallucination and improving the explainability of these models remains a key focus.

Papers

June 8, 2023

Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis-Philippe Morency, Ruslan Salakhutdinov
Contrastive Learning Multimodal Task Multimodal Self Supervised Learning

May 26, 2023

BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks
Kai Zhang, Jun Yu, Eashan Adhikarla, Rong Zhou, Zhiling Yan, Yixin Liu, Zhengliang Liu, Lifang He, Brian Davison, Xiang Li, Hui Ren, Sunyang Fu, James Zou, Wei Liu, Jing Huang, Chen Chen, Yuyin Zhou, Tianming Liu, Xun Chen, Yong Chen, Quanzheng Li, Hongfang Liu, Lichao Sun
Vision Language Vision Paper Generative Pre Trained Transformer Unified Alignment Multimodal Task Modality Specific Medical AI Biomedical Task

May 25, 2023

ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, Jing Liu
Large Language Model Zero Shot Different Modality Multimodal Input Multimodal Task Multimodal Instruction

May 23, 2023

i-Code Studio: A Configurable and Composable Framework for Integrative AI
Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang
Zero Shot Artificial General Intelligence Multimodal Task Multimodal Agent Composable Framework Jim Code

May 17, 2023

Evaluating Object Hallucination in Large Vision-Language Models
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, Ji-Rong Wen
Large Vision Language Model Multimodal Task Hallucination Evaluation Object Hallucination

April 11, 2023

MoMo: A shared encoder Model for text, image and multi-Modal representations
Rakesh Chada, Zhaoheng Zheng, Pradeep Natarajan
Text Modality Image Text Pair Multimodal Task Multimodal Benchmark Multi Modal Representation Encoder Model Encoder Layer

March 29, 2023

MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks
Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, Claire Cui, Anelia Angelova
Vision Language Task Joint Learning Multimodal Task Contrastive Method Video Language Task

March 10, 2023

Single-branch Network for Multimodal Training
Muhammad Saad Saeed, Shah Nawaz, Muhammad Haris Khan, Muhammad Zaigham Zaheer, Karthik Nandakumar, Muhammad Haroon Yousaf, Arif Mahmood
Cross Modal Retrieval Multimodal Task Multi Modal Training Multimodal Problem Face Voice Branch Network Multimodal Application

January 23, 2023

Zorro: the masked multimodal transformer
Adrià Recasens, Jason Lin, Joāo Carreira, Drew Jaegle, Luyu Wang, Jean-baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, Andrew Zisserman
Audio Visual Multimodal Transformer Multimodal Task Contrastive Pre Training Multimodal Processing Audio Visual Model

December 21, 2022

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
Zhiyang Xu, Ying Shen, Lifu Huang
Zero Shot Instruction Tuning Multimodal Task Instruction Dataset Multimodal Instruction

December 15, 2022

December 1, 2022

Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis
Odysseas S. Chlapanis, Georgios Paraskevopoulos, Alexandros Potamianos
Sentiment Analysis Multimodal Phenomenon Multimodal Task BERT Embeddings BERT Architecture Layer Fusion Multimodal BERT

October 23, 2022

MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences
Wei Han, Hui Chen, Min-Yen Kan, Soujanya Poria
Optimal Transport Multimodal Task LD Align Missing Modality Exact Inference Input Modality Optimal Alignment

October 21, 2022

Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?
Mitja Nikolaus, Emmanuelle Salin, Stephane Ayache, Abdellah Fourtassi, Benoit Favre
Multimodal Task Multimodal Reasoning Multimodal Knowledge Vision and Language Model Question Based Dependency Vision and Language

October 5, 2022

Vision+X: A Survey on Multimodal Learning in the Light of Data
Ye Zhu, Yu Wu, Nicu Sebe, Yan Yan
Timely Survey Raw Data Multimodal Learning Multimodal Task Light Work Data Modality Multimodal Machine Learning Multimodal Machine

October 4, 2022

Uncertainty-Aware Meta-Learning for Multimodal Task Distributions
Cesar Almecija, Apoorva Sharma, Navid Azizan
Meta Learning Uncertainty Aware Multimodal Task

August 2, 2022

Silo NLP's Participation at WAT2022
Shantipriya Parida, Subhadarshi Panda, Stig-Arne Grönroos, Mark Granroth-Wilding, Mika Koistinen
NLP Field Multimodal Task Specific Audience

June 28, 2022

MACSA: A Multimodal Aspect-Category Sentiment Analysis Dataset with Multimodal Fine-grained Aligned Annotations
Hao Yang, Yanyan Zhao, Jianwei Liu, Yang Wu, Bing Qin
Fine Grained Multimodal Task Fine Grained Sentiment Multimodal Aspect Category Sentiment Analysis

June 18, 2022

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
Tejas Srinivasan, Ting-Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, Jesse Thomason
Multi Task Vision Language Task Multimodal Task Task Adaptation Continual Learning Benchmark Speed Climbing Training