Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

July 22, 2024

Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models
Raza Imam, Hanan Gani, Muhammad Huzaifa, Karthik Nandakumar
Zero Shot Vision Language Model Test Time Adaptation Pre Trained Vision Language Model Low Rank Adaptation Test Time Prompt Tuning Confidence Minimization
Zero-Shot Embeddings Inform Learning and Forgetting with Vision-Language Encoders
Laura Niss, Kevin Vogt-Lowell, Theodoros Tsiligkaridis
Vision Language Model Attention Weight Modality Gap
HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning
Zhecan Wang, Garrett Bingham, Adams Yu, Quoc Le, Thang Luong, Golnaz Ghiasi
Vision Language Model Content Hallucination Multimodal Reasoning Hallucination Dataset Multimodal Hallucination Hallucination Rate
In-Context Learning Improves Compositional Understanding of Vision-Language Models
Matteo Nulli, Anesa Ibrahimi, Avik Pal, Hoshe Lee, Ivona Najdenkoska
Vision Language Model Context Learning Compositional Structure Compositional Data Contrastive Model
WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding
Quan Kong, Yuki Kawana, Rajat Saini, Ashutosh Kumar, Jingjing Pan, Ta Gu, Yohei Ozao, Balazs Opra, David C. Anastasiu, Yoichi Sato, Norimasa Kobori
Autonomous Driving Fine Grained Vision Language Model Autonomous Vehicle Pedestrian Data Woven Composite

July 21, 2024

Assessing Brittleness of Image-Text Retrieval Benchmarks from Vision-Language Models Perspective
Mariya Hendriksen, Shuo Zhang, Ridho Reinanda, Mohamed Yahya, Edgar Meij, Maarten de Rijke
Fine Grained Vision Language Model Information Retrieval Image Text Retrieval Retrieval Benchmark Brittle Fracture
Failures to Find Transferable Image Jailbreaks Between Vision-Language Models
Rylan Schaeffer, Dan Valentine, Luke Bailey, James Chua, Cristóbal Eyzaguirre, Zane Durante, Joe Benton, Brando Miranda, Henry Sleight, John Hughes, Rajashree Agrawal, Mrinank Sharma, Scott Emmons, Sanmi Koyejo, Ethan Perez
Vision Language Model Jailbreak Attack Transferable Adversarial Attack Transfer Attack Vision Backbone
Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts
Yi Liu, Chengjun Cai, Xiaoli Zhang, Xingliang Yuan, Cong Wang
Vision Language Model Large Vision Language Model Jailbreak Attack Red Teaming

July 20, 2024

Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models
Md Zarif Hossain, Ahmed Imteaj
Vision Language Model Robust Version Robust Representation Adversarial Fine Tuning CLIP Vision Encoder
Falcon2-11B Technical Report
Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campesan, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, Mohammed Al-Yafeai, Hamza Alobeidli, Leen Al Qadi, Mohamed El Amine Seddik, Kirill Fedyanin, Reda Alami, Hakim Hacid
Vision Language Model Multi Modal Technical Report High Quality Open Source Model Image to Text
Can VLMs be used on videos for action recognition? LLMs are Visual Reasoning Coordinators
Harsh Lunia
Large Language Model Vision Language Model Medical LLM Action Recognition Gameplay Video Visual Reasoning Capability

July 19, 2024

DEAL: Disentangle and Localize Concept-level Explanations for VLMs
Tang Li, Mengmeng Ma, Xi Peng
Vision Language Model Pre Trained Vision Language Model Concept Based Explanation Disentangle Content Level Explanation
Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models
Dionis Totsila, Quentin Rouxel, Jean-Baptiste Mouret, Serena Ivaldi
Vision Language Model Foundation Model Human Instruction Customer Service Language Guided Robotic Manipulation Contact Prediction

July 18, 2024

Simultaneous Localization and Affordance Prediction for Tasks in Egocentric Video
Zachary Chavis, Hyun Soo Park, Stephen J. Guy
Vision Language Model Affordance Learning Simultaneous Localization New Task Egocentric Video Spatial Understanding Ego Pose
BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models
Moon Ye-Bin, Nam Hyeon-Woo, Wonseok Choi, Tae-Hyun Oh
Vision Language Model Content Hallucination Image Text Visual Encoder LLM Hallucination Hallucination Dataset
Which objects help me to act effectively? Reasoning about physically-grounded affordances
Anne Kemmeren, Gertjan Burghouts, Michael van Bekkum, Wouter Meijer, Jelle van Mil
Vision Language Model Affordance Learning Robot Embodiment

July 17, 2024

ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, Wayne Zhang
Semantic Segmentation Vision Language Model Open Vocabulary Semantic Segmentation Segmentation Quality CLIP Representation
VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptions
Seokha Moon, Hyun Woo, Hongbeen Park, Haeji Jung, Reza Mahjourian, Hyung-gun Chi, Hyerin Lim, Sangpil Kim, Jinkyu Kim
Vision Language Model Autonomous Vehicle Trajectory Prediction Textual Description

July 16, 2024

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun Zhu, Qing Li
Data Set Vision Language Model Human Feedback Multimodal Model Multi Turn Conversation Feedback Mechanism Fire Occurrence Target Refinement Objective Conversational Feedback
Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities
Xu Zheng, Yuanhuiyi Lyu, Lin Wang
Semantic Segmentation Vision Language Model Different Modality Multi Modal Feature Image Modality Modality Agnostic

Vision Language Model

Papers

Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models

Zero-Shot Embeddings Inform Learning and Forgetting with Vision-Language Encoders

HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning

In-Context Learning Improves Compositional Understanding of Vision-Language Models

WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding

Assessing Brittleness of Image-Text Retrieval Benchmarks from Vision-Language Models Perspective

Failures to Find Transferable Image Jailbreaks Between Vision-Language Models

Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts

Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models

Falcon2-11B Technical Report

Can VLMs be used on videos for action recognition? LLMs are Visual Reasoning Coordinators

DEAL: Disentangle and Localize Concept-level Explanations for VLMs

Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models

Simultaneous Localization and Affordance Prediction for Tasks in Egocentric Video

BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models

Which objects help me to act effectively? Reasoning about physically-grounded affordances

ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference

VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptions

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models

Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities