Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

June 20, 2024

Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang, Dahua Lin, Kai Chen
Vision Language Model New Framework Vision Language Task Capability Evolution Visual Question Multi Agent Decoupling Coefficient Prisma Satellite Mission
From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment
Yusuke Hirota, Ryo Hachiuma, Chao-Han Huck Yang, Yuta Nakashima
Vision Language Model Absolute Stance Bias Generated Caption

June 19, 2024

WATT: Weight Average Test-Time Adaptation of CLIP
David Osowiechi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ben Ayed, Christian Desrosiers
Zero Shot Vision Language Model Test Time Adaptation Single CLIP Weight Update
AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding
Alessandro Suglia, Claudio Greco, Katie Baker, Jose L. Part, Ioannis Papaioannou, Arash Eshghi, Ioannis Konstas, Oliver Lemon
Vision Language Model Egocentric Video Embodied AI Egocentric Vision Egocentric Video Understanding
LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application
Zhe Huang, John Pohovey, Ananya Yammanuru, Katherine Driggs-Campbell
Vision Language Model Human Robot Collaboration Intent Prediction Robot Action ProActive Behavior Intention Estimation
SpatialBot: Precise Spatial Understanding with Vision Language Models
Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, Bo Zhao
Vision Language Model Spatial Understanding
Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens
Xikang Yang, Xuehai Tang, Fuqing Zhu, Jizhong Han, Songlin Hu
Vision Language Model Adversarial Image Adversarial Defense Context Injection Prompt Transferability

June 18, 2024

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model
Jiang-Xin Shi, Chi Zhang, Tong Wei, Yu-Feng Li
Vision Language Model High Efficiency Shot Learning Pre Trained Vision Language Model CLIP Model Zero Shot Generalization
VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
Xiang Li, Jian Ding, Mohamed Elhoseiny
Vision Language Model Vision Language Remote Sensing Image Vision Language Benchmark
VoCo-LLaMA: Towards Vision Compression with Large Language Models
Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, Yansong Tang
Vision Language Model Visual Token Token Compression Vision Compression
The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge
Hongpeng Pan, Shifeng Yi, Shouwei Yang, Lei Qi, Bing Hu, Yi Xu, Yang Yang
Vision Language Model Object Detection Multimodal Large Language Model Pseudo Label Solution Path Iterative Pseudo Labeling

June 17, 2024

SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model
Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, Jing Shao
Vision Language Model Multimodal Information Visual Semantics
Unveiling Encoder-Free Vision-Language Models
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, Xinlong Wang
Vision Language Model Vision Language Representation Vision Language Benchmark
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
Geewook Kim, Minjoon Seo
Vision Language Model Read V Code Efficiency Vision Module Visually Grounded Vision Assistant
See It from My Perspective: Diagnosing the Western Cultural Bias of Large Vision-Language Models in Image Understanding
Amith Ananthram, Elias Stengel-Eskin, Carl Vondrick, Mohit Bansal, Kathleen McKeown
Vision Language Model Large Vision Language Model Visual Perspective Diverse Image Image Understanding Visual Task Cultural Bias
They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias
Salma Abdel Magid, Jui-Hsien Wang, Kushal Kafle, Hanspeter Pfister
Vision Language Model Text to Image Model CLIP Model Counterfactual Image Internal Medicine DOCTOR Visual Appearance Biased Association Diverse Counterfactuals
GUICourse: From General Vision Language Models to Versatile GUI Agents
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, Maosong Sun
Vision Language Model Course Specific Context User Interface Agent
BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models
Xuefeng Hu, Ke Zhang, Min Sun, Albert Chen, Cheng-Hao Kuo, Ram Nevatia
Vision Language Model Test Time Adaptation Visual Embeddings Actor Loss Class Embeddings

June 16, 2024

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, Bill Yuchen Lin
Vision Language Model Wild Challenge Human Preference Multimodal Interaction Biological Vision Contextual Cue Real World Interaction Visual Imagination
Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models
Yikai Zhang, Qianyu He, Xintao Wang, Siyu Yuan, Jiaqing Liang, Yanghua Xiao
Vision Language Model Image Text Pair Quantum Shadow Light Work Multi Modal Knowledge Graph Tail Entity

Vision Language Model

Papers

Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment

WATT: Weight Average Test-Time Adaptation of CLIP

AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding

LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

SpatialBot: Precise Spatial Understanding with Vision Language Models

Enhancing Cross-Prompt Transferability in Vision-Language Models through Contextual Injection of Target Tokens

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model

VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding

VoCo-LLaMA: Towards Vision Compression with Large Language Models

The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge

SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model

Unveiling Encoder-Free Vision-Language Models

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning

See It from My Perspective: Diagnosing the Western Cultural Bias of Large Vision-Language Models in Image Understanding

They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias

GUICourse: From General Vision Language Models to Versatile GUI Agents

BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models