Multimodal Few Shot

Multimodal few-shot learning aims to enable models to learn from limited examples across different data types (e.g., images, text, audio). Current research focuses on developing efficient model architectures, such as masked autoencoders and meta-learning approaches, to effectively fuse multimodal information and adapt to new tasks with minimal training data. This field is crucial for addressing data scarcity issues in various applications, including medical image analysis, visual question answering, and low-resource language processing, by improving the efficiency and robustness of AI systems. The development of new benchmark datasets and the exploration of task-specific personalization are also active areas of investigation.

Papers

October 18, 2024

Few-Shot Joint Multimodal Entity-Relation Extraction via Knowledge-Enhanced Cross-modal Prompt Model
Li Yuan, Yi Cai, Junsheng Huang
Fine Grained Prompt Learning Multimodal Few Shot

August 8, 2024

MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning
Rex Liu, Xin Liu
LeArning Abstract Activity Recognition Multimodal Phenomenon Multimodal Fusion Wearable Sensor Data Sce Mae Multimodal Sensor Multimodal Few Shot

March 17, 2024

Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches
Igor Sterner, Weizhe Lin, Jinghong Chen, Bill Byrne
Large Language Model Constructive Approach Visual Question Answering Image Caption Cautionary TALE Visual Embeddings Frozen Large Language Model Multimodal Few Shot

November 1, 2023

On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval
Jiayi Chen, Hanjun Dai, Bo Dai, Aidong Zhang, Wei Wei
Unseen Entity Multimodal Few Shot

September 30, 2023

Self-Supervised Open-Ended Classification with Small Visual Language Models
Mohammad Mahdi Derakhshani, Ivona Najdenkoska, Cees G. M. Snoek, Marcel Worring, Yuki M. Asano
Self Supervised Learning Self Supervised Approach Contextual Adapter Multimodal Few Shot Pseudo Caption

July 27, 2023

Med-Flamingo: a Multimodal Medical Few-shot Learner
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec
Vision Language Model Shot Learning Medical Visual Question Answering Multimodal Few Shot

June 20, 2023

Visually grounded few-shot word learning in low-resource settings
Leanne Nortje, Dan Oneata, Herman Kamper
Low Resource Image Text Pair Shot Scenario Multimodal Few Shot

June 1, 2023

MEWL: Few-shot multimodal word learning with referential uncertainty
Guangyuan Jiang, Manjie Xu, Shiji Xin, Wei Liang, Yujia Peng, Chi Zhang, Yixin Zhu
Multimodal Learning Multimodal Understanding Pragmatic Reasoning Multimodal Few Shot Referential Ambiguity

May 26, 2023

Multi-Objective Genetic Algorithm for Multi-View Feature Selection
Vandad Imani, Carlos Sevilla-Salcedo, Elaheh Moradi, Vittorio Fortino, Jussi Tohka
Multi View Feature Selection Genetic Algorithm Multi Objective Feature Selection Multimodal Few Shot

February 28, 2023

Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning
Ivona Najdenkoska, Xiantong Zhen, Marcel Worring
Language Model LeArning Abstract Meta Learning Multimodal Phenomenon Bridging Vision Multimodal Few Shot

February 19, 2023

Few-shot Multimodal Multitask Multilingual Learning
Aman Chadha, Vinija Jain
Shot Learning Multitask Learning Multi Modal Multi Multimodal Few Shot