Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

October 4, 2024

ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities
Ying Su, Zhan Ling, Haochen Shi, Jiayang Cheng, Yauwai Yim, Yangqiu Song
Vision Language Model Task Planning Planning Benchmark Planning Ability Counterfactual Plan
The Wallpaper is Ugly: Indoor Localization using Vision and Language
Seth Pate, Lawson L.S. Wong
Vision Language Model Human Language Vision Paper Indoor Localization Natural Language Query Text Description Similarity Score
An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation
Ahmed Abdulaal, Hugo Fry, Nina Montaña-Brown, Ayodeji Ijishakin, Jack Gao, Stephanie Hyland, Daniel C. Alexander, Daniel C. Castro
Vision Language Model Latent Representation Feature Wise X Ray Radiology Report Generation Sparse Autoencoders
Generalizable Prompt Tuning for Vision-Language Models
Qian Zhang
Vision Language Model Text Modality Visual Modality Prompt Template
CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization
Shigemichi Matsuzaki, Kazuhito Tanaka, Kazuhiro Shintani
Vision Language Model Object Localization Global Localization Object Landmark Soft Landmark Correspondence Matching Robust Correspondence Landmark Matching

October 3, 2024

Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations
Nick Jiang, Anish Kachinthaya, Suzie Petryk, Yossi Gandelsman
Vision Language Model Latent Representation Human Editing Internal Representation Feature Representation COCO Dataset Vision Language Representation Machine Translated
Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
Shuoyuan Wang, Yixuan Li, Hongxin Wei
Vision Language Model Fine Tuning Human Understanding Prompt Tuning Confidence Calibration Calibration Error
LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model
Duy M. H. Nguyen, Nghiem T. Diep, Trung Q. Nguyen, Hoang-Bao Le, Tai Nguyen, Tien Nguyen, TrungTin Nguyen, Nhat Ho, Pengtao Xie, Roger Wattenhofer, James Zhou, Daniel Sonntag, Mathias Niepert
Vision Language Model Cross Modal Long Context Image Text Pair LLaVA HD Medical Large Language Model Multi Graph Matching
Guiding Long-Horizon Task and Motion Planning with Vision Language Models
Zhutian Yang, Caelan Garrett, Dieter Fox, Tomás Lozano-Pérez, Leslie Pack Kaelbling
Vision Language Model Motion Planning Robotic Task Long Horizon Task Vision Language Action Motion Planner

October 2, 2024

ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning
Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, Zhou Yu
Vision Language Model Monte Carlo Tree Search AI Agent Tree Search Autonomous Software Agent Minimax McTs
Enhancing Screen Time Identification in Children with a Multi-View Vision Language Model and Screen Time Tracker
Xinlong Hou, Sen Shen, Xueshen Li, Xinran Gao, Ziyi Huang, Steven J. Holiday, Matthew R. Cribbet, Susan W. White, Edward Sazonov, Yu Gan
Vision Language Model Nine Year Old Child Wearable Sensor New Tracker Screen Content Exposure Data
Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities
Kenza Amara, Lukas Klein, Carsten Lüth, Paul Jäger, Hendrik Strobelt, Mennatallah El-Assady
Vision Language Model Text Modality Complex Reasoning Visual Question Answering Context Information Visual Language Model Different Modality Image Modality Modal Integration
Toward a Holistic Evaluation of Robustness in CLIP Models
Weijie Tu, Weijian Deng, Tom Gedeon
Native Robustness Vision Language Model Contrastive Loss Large Multimodal Model Contrastive Language Image Holistic Evaluation
Information-Theoretical Principled Trade-off between Jailbreakability and Stealthiness on Vision Language Models
Ching-Chia Kao, Chia-Mu Yu, Chun-Shien Lu, Chu-Song Chen
Vision Language Model Jailbreak Attack Digital STEALTH Metric Optimal Trade
Backdooring Vision-Language Models with Out-Of-Distribution Data
Weimin Lyu, Jiachen Yao, Saumya Gupta, Lu Pang, Tao Sun, Lingjie Yi, Lijie Hu, Haibin Ling, Chao Chen
Vision Language Model Backdoor Attack Multimodal Model Distribution Data Image to Text Task
UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark
Hasnat Md Abdullah, Tian Liu, Kangda Wei, Shu Kong, Ruihong Huang
Vision Language Model Video Dataset Video Language Model Video Understanding Model

October 1, 2024

ScVLM: a Vision-Language Model for Driving Safety Critical Event Understanding
Liang Shi, Boyu Jiang, Feng Guo
Vision Language Model Video Understanding Driver Assistance System Crash Data Driving Datasets Safety Critical Driving Scenario
Find Everything: A General Vision Language Model Approach to Multi-Object Search
Daniel Choi, Angus Fung, Haitong Wang, Aaron Hao Tan
Deep Reinforcement Learning Vision Language Model Semantic Correlation Multi Object Search
AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, Yijie Guo
Vision Language Model Data Detection Complex Reasoning Robotic Manipulation Robotic System Leg Failure Failure Analysis Robot Failure
Vision Language Models Know Law of Conservation without Understanding More-or-Less
Dezhi Luo, Haiyun Lyu, Qingying Gao, Haoran Sun, Yijiang Li, Hokin Deng
Vision Language Model Average Approximation Legal Text Environmental Conservation Cognitively Inspired Benchmark Cognitive Development

Vision Language Model

Papers

ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities

The Wallpaper is Ugly: Indoor Localization using Vision and Language

An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

Generalizable Prompt Tuning for Vision-Language Models

CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization

Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations

Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models

LoGra-Med: Long Context Multi-Graph Alignment for Medical Vision-Language Model

Guiding Long-Horizon Task and Motion Planning with Vision Language Models

ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning

Enhancing Screen Time Identification in Children with a Multi-View Vision Language Model and Screen Time Tracker

Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities

Toward a Holistic Evaluation of Robustness in CLIP Models

Information-Theoretical Principled Trade-off between Jailbreakability and Stealthiness on Vision Language Models

Backdooring Vision-Language Models with Out-Of-Distribution Data

UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark

ScVLM: a Vision-Language Model for Driving Safety Critical Event Understanding

Find Everything: A General Vision Language Model Approach to Multi-Object Search

AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation

Vision Language Models Know Law of Conservation without Understanding More-or-Less