Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

November 27, 2024

SCoTT: Wireless-Aware Path Planning with Vision Language Models and Strategic Chains-of-Thought
Aladin Djuhera, Vlad C. Andrei, Amin Seffo, Holger Boche, Walid Saad
Vision Language Model Chain of Thought Path Planning Planning Problem Ray Based Wireless Environment
From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects
Zizhao Li, Zhengkang Xiang, Joseph West, Kourosh Khoshelham
Vision Language Model Object Detection World Event Open Vocabulary Open Vocabulary Object Detection Novel Object Object Embeddings Open World Semi Supervised
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
Di Zhang, Junxian Li, Jingdi Lei, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, Peng Ye, Wanli Ouyang, Dongzhan Zhou
Vision Language Model Multimodal Reasoning
DistinctAD: Distinctive Audio Description Generation in Contexts
Bo Fang, Wenhao Wu, Qiangqiang Wu, Yuxin Song, Antoni B. Chan
Vision Language Model Context Information Movie Dataset Audio Description CLIP Adaptation
DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Zhanhui kang, Yu Wang
Vision Language Model Large Vision Language Model Content Hallucination Cross Modal Attention Hallucination Detection
Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysis
Weiqin Zhao, Ziyu Guo, Yinshuang Fan, Yuming Jiang, Maximus Yeung, Lequan Yu
Fine Grained Vision Language Model Multiple Instance Learning Whole Slide Image Image Analysis Concept Learning Knowledge Alignment
VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis
Donggoo Kang, Dasol Jeong, Hyunmin Lee, Sangwoo Park, Hasil Park, Sunkyu Kwon, Yeongjoon Kim, Joonki Paik
Vision Language Model Large Vision Language Model Human Object Interaction Interaction Evaluation
Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models
Shuyang Hao, Bryan Hooi, Jun Liu, Kai-Wei Chang, Zi Huang, Yujun Cai
Vision Language Model Adversarial Search Physical Inter Rule Vulnerability
Verbalized Representation Learning for Interpretable Few-Shot Generalization
Cheng-Fu Yang, Da Yin, Wenbo Hu, Nanyun Peng, Bolei Zhou, Kai-Wei Chang
Vision Language Model Strong Generalization Representation Learning Interpretable Representation Interpretable Feature Human Labeled

November 26, 2024

What's in the Image? A Deep-Dive into the Vision of Vision Language Models
Omri Kaduri, Shai Bagon, Tali Dekel
Vision Language Model Full Model Vision Language Vision Paper Deep Dive Visual Complexity
CoA: Chain-of-Action for Generative Semantic Labels
Meng Wei, Zhongnian Li, Peng Ying, Xinzheng Xu
Zero Shot Vision Language Model Semantic Label Label Space
Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation
Chanyoung Kim, Dayun Ju, Woojung Han, Ming-Hsuan Yang, Seong Jae Hwang
Semantic Segmentation Vision Language Model Open Vocabulary Open Vocabulary Semantic Segmentation Object Level

November 25, 2024

Probing the limitations of multimodal language models for chemistry and materials research
Nawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García, Indrajeet Mandal, Pranav Khetarpal, Hargun Singh Grover, N. M. Anoop Krishnan, Kevin Maik Jablonka
Vision Language Model Fundamental Limitation Experimental Study Multimodal Language Model Material SCIence Research Assistant Material Design Organic Chemistry
A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models
Manuel Schwonberg, Claus Werner, Hanno Gottschalk, Carsten Meyer
Semantic Segmentation Domain Adaptation Vision Language Model Vision Language Study Feature Unsupervised Domain Adaptation Domain Shift New Era
Open-Vocabulary Octree-Graph for 3D Scene Understanding
Zhigang Wang, Yifei Su, Chenhui Li, Dong Wang, Yan Huang, Bin Zhao, Xuelong Li
Point Cloud Vision Language Model 3D Scene Understanding Octree Structure Semantic Octree
Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Models
Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh Afghah
Vision Language Model Prompt Learning Style Consistency Zero Shot Generalization

November 24, 2024

Improving Medical Diagnostics with Vision-Language Models: Convex Hull-Based Uncertainty Analysis
Ferhat Ozgur Catak, Murat Kuzlu, Taylor Patrick
Vision Language Model Visual Question Answering Uncertainty Estimation LLM Model
ResCLIP: Residual Attention for Training-free Dense Vision-language Inference
Yuhang Yang, Jinhong Deng, Wen Li, Lixin Duan
Vision Language Model Scientific Inference Self Attention Layer Residual Attention Entity Attention
Test-time Alignment-Enhanced Adapter for Vision-Language Models
Baoshun Tong, Kaiyu Song, Hanjiang Lai
Vision Language Model Test Time Adaptation Pre Trained Vision Language Model Test Time Image Text Alignment
Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks
Peng Xie, Yequan Bie, Jianda Mao, Yangqiu Song, Yang Wang, Hao Chen, Kani Chen
Adversarial Attack Native Robustness Vision Language Model Adversarial Example Side Chain Adversarial Transferability Blackbox Attack

Vision Language Model

Papers

SCoTT: Wireless-Aware Path Planning with Vision Language Models and Strategic Chains-of-Thought

From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects

Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

DistinctAD: Distinctive Audio Description Generation in Contexts

DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models

Aligning Knowledge Concepts to Whole Slide Images for Precise Histopathology Image Analysis

VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis

Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models

Verbalized Representation Learning for Interpretable Few-Shot Generalization

What's in the Image? A Deep-Dive into the Vision of Vision Language Models

CoA: Chain-of-Action for Generative Semantic Labels

Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation

Probing the limitations of multimodal language models for chemistry and materials research

A Study on Unsupervised Domain Adaptation for Semantic Segmentation in the Era of Vision-Language Models

Open-Vocabulary Octree-Graph for 3D Scene Understanding

Style-Pro: Style-Guided Prompt Learning for Generalizable Vision-Language Models

Improving Medical Diagnostics with Vision-Language Models: Convex Hull-Based Uncertainty Analysis

ResCLIP: Residual Attention for Training-free Dense Vision-language Inference

Test-time Alignment-Enhanced Adapter for Vision-Language Models

Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks