Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

June 18, 2024

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model
Jiang-Xin Shi, Chi Zhang, Tong Wei, Yu-Feng Li
Vision Language Model High Efficiency Shot Learning Pre Trained Vision Language Model CLIP Model Zero Shot Generalization
VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
Xiang Li, Jian Ding, Mohamed Elhoseiny
Vision Language Model Vision Language Remote Sensing Image Vision Language Benchmark
VoCo-LLaMA: Towards Vision Compression with Large Language Models
Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, Yansong Tang
Vision Language Model Visual Token Token Compression Vision Compression
The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge
Hongpeng Pan, Shifeng Yi, Shouwei Yang, Lei Qi, Bing Hu, Yi Xu, Yang Yang
Vision Language Model Object Detection Multimodal Large Language Model Pseudo Label Solution Path Iterative Pseudo Labeling

June 17, 2024

SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model
Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, Jing Shao
Vision Language Model Multimodal Information Visual Semantics
Unveiling Encoder-Free Vision-Language Models
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, Xinlong Wang
Vision Language Model Vision Language Representation Vision Language Benchmark
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
Geewook Kim, Minjoon Seo
Vision Language Model Read V Code Efficiency Vision Module Visually Grounded Vision Assistant
See It from My Perspective: Diagnosing the Western Cultural Bias of Large Vision-Language Models in Image Understanding
Amith Ananthram, Elias Stengel-Eskin, Carl Vondrick, Mohit Bansal, Kathleen McKeown
Vision Language Model Large Vision Language Model Visual Perspective Image Understanding Diverse Image Visual Task Cultural Bias
They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias
Salma Abdel Magid, Jui-Hsien Wang, Kushal Kafle, Hanspeter Pfister
Vision Language Model Text to Image Model CLIP Model Counterfactual Image Internal Medicine DOCTOR Visual Appearance Biased Association Diverse Counterfactuals
GUICourse: From General Vision Language Models to Versatile GUI Agents
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, Maosong Sun
Vision Language Model Course Specific Context User Interface Agent
BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models
Xuefeng Hu, Ke Zhang, Min Sun, Albert Chen, Cheng-Hao Kuo, Ram Nevatia
Vision Language Model Test Time Adaptation Visual Embeddings Actor Loss Class Embeddings

June 16, 2024

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, Bill Yuchen Lin
Vision Language Model Wild Challenge Human Preference Multimodal Interaction Biological Vision Contextual Cue Real World Interaction Visual Imagination
Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models
Yikai Zhang, Qianyu He, Xintao Wang, Siyu Yuan, Jiaqing Liang, Yanghua Xiao
Vision Language Model Image Text Pair Quantum Shadow Light Work Multi Modal Knowledge Graph Tail Entity
AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models
Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shrivastava, Furong Huang, Jordan Lee Boyd-Graber, Tianyi Zhou, Dinesh Manocha
Vision Language Model Large Vision Language Model Automatic Generation Model Hallucination LLM Hallucination Hallucination Benchmark

June 15, 2024

RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, Dieter Fox
Vision Language Model Robotics Domain Affordance Learning Robot Behavior
Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data
Jiahan Zhang, Qi Wei, Feng Liu, Lei Feng
Vision Language Model Vision Language Pseudo Label Prompt Tuning Pseudo Labeling Unlabeled Data

June 14, 2024

From Pixels to Prose: A Large Dataset of Dense Image Captions
Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, Tom Goldstein
Vision Language Model Large Vision Language Model Tetromino Pixel Dense Caption
Long Story Short: Story-level Video Understanding from 20K Short Films
Ridouane Ghermi, Xi Wang, Vicky Kalogeiton, Ivan Laptev
Vision Language Model New Benchmark Video Understanding Video Analysis Movie Dataset Short Video
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar
Reinforcement Learning Vision Language Model Mobile Device Effective Reinforcement Learning Digital Human Autonomous Reinforcement Learning
DevBench: A multimodal developmental benchmark for language learning
Alvin Wei Ming Tan, Sunny Yu, Bria Long, Wanjing Anya Ma, Tonya Murray, Rebecca D. Silverman, Jason D. Yeatman, Michael C. Frank
Language Model Vision Language Model Multimodal Benchmark Language Learning Large Scale Naturalistic Language Development

Vision Language Model

Papers

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model

VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding

VoCo-LLaMA: Towards Vision Compression with Large Language Models

The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge

SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model

Unveiling Encoder-Free Vision-Language Models

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning

See It from My Perspective: Diagnosing the Western Cultural Bias of Large Vision-Language Models in Image Understanding

They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias

GUICourse: From General Vision Language Models to Versatile GUI Agents

BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models

AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models

RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics

Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data

From Pixels to Prose: A Large Dataset of Dense Image Captions

Long Story Short: Story-level Video Understanding from 20K Short Films

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

DevBench: A multimodal developmental benchmark for language learning