Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

September 21, 2024

VLM-Vac: Enhancing Smart Vacuums through VLM Knowledge Distillation and Language-Guided Experience Replay
Reihaneh Mirjalili, Michael Krawez, Florian Walter, Wolfram Burgard
Zero Shot Knowledge Distillation Vision Language Model Robot Person Cleaning Robot
KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data
Grace Tang, Swetha Rajkumar, Yifei Zhou, Homer Rich Walke, Sergey Levine, Kuan Fang
Vision Language Model Affordance Learning Open World Robot Data

September 20, 2024

OLiVia-Nav: An Online Lifelong Vision Language Approach for Mobile Robot Social Navigation
Siddarth Narasimhan, Aaron Hao Tan, Daniel Choi, Goldie Nejat
Vision Language Model Social Navigation Social Robot Navigation Large Vision Language
FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs
Bowen Yan, Zhengsong Zhang, Liqiang Jing, Eftekhar Hossain, Xinya Du
Vision Language Model Large Vision Language Model Scene Graph Hallucination Evaluation Language Model Hallucination
YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models
Abhilash Nandy, Yash Agarwal, Ashish Patwa, Millon Madhur Das, Aman Bansal, Ankit Raj, Pawan Goyal, Niloy Ganguly
Vision Language Model Multimodal Dataset Fake Image Yes Yes

September 19, 2024

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case
Peng Chen, Pi Bu, Jun Song, Yuan Gao, Bo Zheng
Vision Language Model Case Study Multimodal Agent Game Environment Multi Action
Vision Language Models Can Parse Floor Plan Maps
David DeFazio, Hrudayangam Mehta, Jeremy Blackburn, Shiqi Zhang
Vision Language Model Image Captioning Navigation Task Floor Plan Indoor Navigation
LARE: Latent Augmentation using Regional Embedding with Vision-Language Model
Kosuke Sakurai, Tatsuya Ishii, Ryotaro Shimizu, Linxin Song, Masayuki Goto
Vision Language Model Contrastive Language Image Image Embeddings Space Augmentation

September 18, 2024

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, Junyang Lin
Vision Language Model Large Vision Language Model Large Multimodal Model World Event Perception Aware Multimodal Benchmark Conjunct ReSolution Visual Processing Visual Description Qwen Audio
Mixture of Prompt Learning for Vision Language Models
Yu Du, Tong Niu, Rong Zhao
Vision Language Model Mixture Component Prompt Learning Pre Trained Vision Language Model Text Supervision Prompt Template
GauTOAO: Gaussian-based Task-Oriented Affordance of Objects
Jiawen Wang, Dingsheng Luo
Vision Language Model Arbitrary Object Affordance Learning Deep Object
LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
Amaia Cardiel, Eloi Zablocki, Elias Ramzi, Oriane Siméoni, Matthieu Cord
Large Language Model Vision Language Model Black Box Open Vocabulary Referring Expression Comprehension Lightweight LLM Semantic Reasoning

September 17, 2024

Constructive Apraxia: An Unexpected Limit of Instructible Vision-Language Models and Analog for Human Cognitive Disorders
David Noever, Samantha E. Miller Noever
Vision Language Model Continuum Limit Visual Perception Spatial Reasoning Cognitive Impairment Analog Bit Spatial Cognition
Mind the Uncertainty in Human Disagreement: Evaluating Discrepancies between Model Predictions and Human Responses in VQA
Jian Lan, Diego Frassinelli, Barbara Plank
Vision Language Model Large Vision Language Model Visual Question Answering High Uncertainty Anticipation Human Evaluation User Response Model Prediction Multi Annotator Human Disagreement
CAST: Cross-modal Alignment Similarity Test for Vision Language Models
Gautier Dagan, Olga Loginova, Anil Batra
Vision Language Model Visual Question Answering Cross Modal Alignment
KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph
Yanbei Jiang, Krista A. Ehinger, Jey Han Lau
Vision Language Model Image Captioning Heterogeneous Graph

September 16, 2024

Do Pre-trained Vision-Language Models Encode Object States?
Kaleb Newman, Shijie Wang, Yuan Zang, David Heffren, Chen Sun
Vision Language Model Object Recognition Object State
HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models
Vineet Bhat, Prashanth Krishnamurthy, Ramesh Karri, Farshad Khorrami
Vision Language Model Visual Grounding Robotic Grasping

September 15, 2024

Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, David Acuna
Vision Language Model Large Vision Language Model Reasoning Path Spatial Reasoning Higher Quality Reference Zero Shot Prompting Spatial Reasoning Task Spatial Analysis
Finetuning CLIP to Reason about Pairwise Differences
Dylan Sam, Devin Willmott, Joao D. Semedo, J. Zico Kolter
Contrastive Learning Vision Language Model Jina Embeddings Text to Image Generation Single CLIP Text Embeddings Reason Giving Pairwise Distance

Vision Language Model

Papers

VLM-Vac: Enhancing Smart Vacuums through VLM Knowledge Distillation and Language-Guided Experience Replay

KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data

OLiVia-Nav: An Online Lifelong Vision Language Approach for Mobile Robot Social Navigation

FIHA: Autonomous Hallucination Evaluation in Vision-Language Models with Davidson Scene Graphs

YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

Vision Language Models Can Parse Floor Plan Maps

LARE: Latent Augmentation using Regional Embedding with Vision-Language Model

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

Mixture of Prompt Learning for Vision Language Models

GauTOAO: Gaussian-based Task-Oriented Affordance of Objects

LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension

Constructive Apraxia: An Unexpected Limit of Instructible Vision-Language Models and Analog for Human Cognitive Disorders

Mind the Uncertainty in Human Disagreement: Evaluating Discrepancies between Model Predictions and Human Responses in VQA

CAST: Cross-modal Alignment Similarity Test for Vision Language Models

KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph

Do Pre-trained Vision-Language Models Encode Object States?

HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models

Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models

Finetuning CLIP to Reason about Pairwise Differences