Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

November 27, 2024

Evaluating Vision-Language Models as Evaluators in Path Planning
Mohamed Aghzal, Xiang Yue, Erion Plaku, Ziyu Yao
Vision Language Model Fine Tuning Path Planning Human Evaluation
Embodied Red Teaming for Auditing Robotic Foundation Models
Sathwik Karnik, Zhang-Wei Hong, Nishant Abhangi, Yen-Chen Lin, Tsun-Hsuan Wang, Pulkit Agrawal
Vision Language Model Human Instruction Natural Language Instruction Red Teaming Language Conditioned
FactCheXcker: Mitigating Measurement Hallucinations in Chest X-ray Report Generation Models
Alice Heiman, Xiaoman Zhang, Emma Chen, Sung Eun Kim, Pranav Rajpurkar
Vision Language Model Chest X Ray Fact Checking Mitigating Hallucination Radiology Report Medical Information Mart for Intensive Report Generation Medical Report Generation
Visual Adversarial Attack on Vision-Language Models for Autonomous Driving
Tianyuan Zhang, Lu Wang, Xinwei Zhang, Yitong Zhang, Boyi Jia, Siyuan Liang, Shengshan Hu, Qiang Fu, Aishan Liu, Xianglong Liu
Adversarial Attack Autonomous Driving Vision Language Model Legal Autonomy Driving Context Effective Adversarial
SCoTT: Wireless-Aware Path Planning with Vision Language Models and Strategic Chains-of-Thought
Aladin Djuhera, Vlad C. Andrei, Amin Seffo, Holger Boche, Walid Saad
Vision Language Model Chain of Thought Path Planning Planning Problem Ray Based Wireless Environment
From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects
Zizhao Li, Zhengkang Xiang, Joseph West, Kourosh Khoshelham
Vision Language Model Object Detection World Event Open Vocabulary Open Vocabulary Object Detection Novel Object Object Embeddings Open World Semi Supervised
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
Di Zhang, Junxian Li, Jingdi Lei, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, Peng Ye, Wanli Ouyang, Dongzhan Zhou
Vision Language Model Multimodal Reasoning
DistinctAD: Distinctive Audio Description Generation in Contexts
Bo Fang, Wenhao Wu, Qiangqiang Wu, Yuxin Song, Antoni B. Chan
Vision Language Model Context Information Movie Dataset Audio Description CLIP Adaptation
DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Zhanhui kang, Yu Wang
Vision Language Model Large Vision Language Model Content Hallucination Cross Modal Attention Hallucination Detection
Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysis
Weiqin Zhao, Ziyu Guo, Yinshuang Fan, Yuming Jiang, Maximus Yeung, Lequan Yu
Fine Grained Vision Language Model Multiple Instance Learning Whole Slide Image Image Analysis Concept Learning Knowledge Alignment
VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis
Donggoo Kang, Dasol Jeong, Hyunmin Lee, Sangwoo Park, Hasil Park, Sunkyu Kwon, Yeongjoon Kim, Joonki Paik
Vision Language Model Large Vision Language Model Human Object Interaction Interaction Evaluation
Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models
Shuyang Hao, Bryan Hooi, Jun Liu, Kai-Wei Chang, Zi Huang, Yujun Cai
Vision Language Model Adversarial Search Physical Inter Rule Vulnerability
Verbalized Representation Learning for Interpretable Few-Shot Generalization
Cheng-Fu Yang, Da Yin, Wenbo Hu, Nanyun Peng, Bolei Zhou, Kai-Wei Chang
Vision Language Model Strong Generalization Representation Learning Interpretable Representation Interpretable Feature Human Labeled

November 26, 2024

What's in the Image? A Deep-Dive into the Vision of Vision Language Models
Omri Kaduri, Shai Bagon, Tali Dekel
Vision Language Model Full Model Vision Language Vision Paper Deep Dive Visual Complexity
CoA: Chain-of-Action for Generative Semantic Labels
Meng Wei, Zhongnian Li, Peng Ying, Xinzheng Xu
Zero Shot Vision Language Model Semantic Label Label Space
Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation
Chanyoung Kim, Dayun Ju, Woojung Han, Ming-Hsuan Yang, Seong Jae Hwang
Semantic Segmentation Vision Language Model Open Vocabulary Open Vocabulary Semantic Segmentation Object Level

November 25, 2024

Probing the limitations of multimodal language models for chemistry and materials research
Nawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García, Indrajeet Mandal, Pranav Khetarpal, Hargun Singh Grover, N. M. Anoop Krishnan, Kevin Maik Jablonka
Vision Language Model Fundamental Limitation Experimental Study Multimodal Language Model Material SCIence Research Assistant Material Design Organic Chemistry
A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models
Manuel Schwonberg, Claus Werner, Hanno Gottschalk, Carsten Meyer
Semantic Segmentation Domain Adaptation Vision Language Model Vision Language Study Feature Unsupervised Domain Adaptation Domain Shift New Era
Open-Vocabulary Octree-Graph for 3D Scene Understanding
Zhigang Wang, Yifei Su, Chenhui Li, Dong Wang, Yan Huang, Bin Zhao, Xuelong Li
Point Cloud Vision Language Model 3D Scene Understanding Octree Structure Semantic Octree
Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Models
Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh Afghah
Vision Language Model Prompt Learning Style Consistency Zero Shot Generalization

Vision Language Model

Papers

Evaluating Vision-Language Models as Evaluators in Path Planning

Embodied Red Teaming for Auditing Robotic Foundation Models

FactCheXcker: Mitigating Measurement Hallucinations in Chest X-ray Report Generation Models

Visual Adversarial Attack on Vision-Language Models for Autonomous Driving

SCoTT: Wireless-Aware Path Planning with Vision Language Models and Strategic Chains-of-Thought

From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects

Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

DistinctAD: Distinctive Audio Description Generation in Contexts

DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models

Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysis

VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis

Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models

Verbalized Representation Learning for Interpretable Few-Shot Generalization

What's in the Image? A Deep-Dive into the Vision of Vision Language Models

CoA: Chain-of-Action for Generative Semantic Labels

Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation

Probing the limitations of multimodal language models for chemistry and materials research

A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models

Open-Vocabulary Octree-Graph for 3D Scene Understanding

Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Models