Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

October 9, 2024

VHELM: A Holistic Evaluation of Vision Language Models
Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, Percy Liang
Vision Language Model Evaluation Metric Visual Perception Holistic Evaluation Bias Benchmark
Pixtral 12B
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang
Vision Language Model Multimodal LLM Multimodal Task Multimodal Benchmark
Preference Fine-Tuning for Factuality in Chest X-Ray Interpretation Models Without Human Feedback
Dennis Hein, Zhihong Chen, Sophie Ostmeier, Justin Xu, Maya Varma, Eduardo Pontes Reis, Arne Edward Michalson, Christian Bluethgen, Hyun Joo Shin, Curtis Langlotz, Akshay S Chaudhari
Vision Language Model Medical Image Human Feedback Factual Claim Preference Alignment Preference Fine Tuning
Compositional Entailment Learning for Hyperbolic Vision-Language Models
Avik Pal, Max van Spengler, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, Pascal Mettes
Vision Language Model Vision Language Representation Hyperbolic Representation Entailment Tree
Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency
Mingliang Liang, Martha Larson
Vision Language Model Image Text Pair Edge Pruning Pruning Method Text Image Pair Text Distribution Word Frequency
ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
Yi Ding, Bolian Li, Ruqi Zhang
Vision Language Model Adversarial Input Inference Time Multimodal AI Temporal Alignment Safety Alignment Modal Attack Estimated Time of Arrival
From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
Yang Bai, Yang Zhou, Jun Zhou, Rick Siow Mong Goh, Daniel Shu Wei Ting, Yong Liu
Vision Language Model Large Vision Language Model Task Specific Model Generalist Learner Expertise Level Contrastive Tuning

October 8, 2024

BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulation
Rutav Shah, Albert Yu, Yifeng Zhu, Yuke Zhu, Roberto Martín-Martín
Vision Language Model Complex Reasoning Mobile Manipulation Long Horizon Task
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
M. Jehanzeb Mirza, Mengjie Zhao, Zhuoyuan Mao, Sivan Doveh, Wei Lin, Paul Gavrikov, Michael Dorkenwald, Shiqi Yang, Saurav Jha, Hiromi Wakaki, Yuki Mitsufuji, Horst Possegger, Rogerio Feris, Leonid Karlinsky, James Glass
Vision Language Model Vision Language Zero Shot Classification Dual Encoder
Beyond Captioning: Task-Specific Prompting for Improved VLM Performance in Mathematical Reasoning
Ayush Singh, Mansi Gupta, Shivank Garg, Abhinav Kumar, Vansh Agrawal
Vision Language Model Visual Question Answering Mathematical Reasoning Task Specific Generated Caption
CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection
Mingyi Guo, Yuyang Liu, Zongying Lin, Peixi Peng, Yonghong Tian
Vision Language Model COCO Dataset Vision Language Foundation Model Rich Attribute Incremental Object Detection Class Incremental Attribute Composition
NegMerge: Consensual Weight Negation for Strong Machine Unlearning
Hyoseo Kim, Dongyoon Han, Junsuk Choe
Vision Language Model Fine Tuning Unlearning Framework Fine Tuned Model Weighting Function

October 7, 2024

Fine-Tuning CLIP's Last Visual Projector: A Few-Shot Cornucopia
Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, Raoul de Charette
Vision Language Model Shot Classification Camera Projector Shot Class CLIP Adaptation Shot Classification Benchmark
TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models
Rabin Adhikari, Safal Thapaliya, Manish Dhakal, Bishesh Khanal
Vision Language Model Prompt Tuning Visual Prompt Tuning Vision Language Segmentation
LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulation
Zhijie Wang, Zhehua Zhou, Jiayang Song, Yuheng Huang, Zhan Shu, Lei Ma
Vision Language Model Robotic Manipulation Manipulation Task Code Language Model Natural Language Input Vision Language Action Evaluation Platform
HE-Drive: Human-Like End-to-End Driving with Vision Language Models
Junming Wang, Xingyu Zhang, Zebin Xing, Songen Gu, Xiaoyang Guo, Yang Hu, Ziying Song, Qian Zhang, Xiaoxiao Long, Wei Yin
Autonomous Driving Vision Language Model End to End Trajectory Planner Trajectory Prediction Error
AnyAttack: Targeted Adversarial Attacks on Vision-Language Models toward Any Images
Jiaming Zhang, Junhong Ye, Xingjun Ma, Yige Li, Yunfan Yang, Jitao Sang, Dit-Yan Yeung
Adversarial Attack Vision Language Model Adversarial Example Adversarial Noise Adversarial Image Unsupervised Generation
Generating CAD Code with Vision-Language Models for 3D Designs
Kamel Alrashedy, Pradyumna Tambwekar, Zulfiqar Zaidi, Megan Langwasser, Wei Xu, Matthew Gombolay
Vision Language Model 3D Shape Computer Aided Design

October 6, 2024

Is What You Ask For What You Get? Investigating Concept Associations in Text-to-Image Models
Salma Abdel Magid, Weiwei Pan, Simon Warchol, Grace Guo, Junsik Kim, Mahia Rahman, Hanspeter Pfister
Vision Language Model Text to Image Model Text to Image Interpretable Concept Semantic Association
CoVLM: Leveraging Consensus from Vision-Language Models for Semi-supervised Multi-modal Fake News Detection
Devank, Jayateja Kalla, Soma Biswas
Vision Language Model Pseudo Label Fake News Fake News Detection Consensus Group Decision Misinformation Detection Robust Pseudo Label

Vision Language Model

Papers

VHELM: A Holistic Evaluation of Vision Language Models

Pixtral 12B

Preference Fine-Tuning for Factuality in Chest X-Ray Interpretation Models Without Human Feedback

Compositional Entailment Learning for Hyperbolic Vision-Language Models

Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning

BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulation

GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models

Beyond Captioning: Task-Specific Prompting for Improved VLM Performance in Mathematical Reasoning

CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection

NegMerge: Consensual Weight Negation for Strong Machine Unlearning

Fine-Tuning CLIP's Last Visual Projector: A Few-Shot Cornucopia

TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models

LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulation

HE-Drive: Human-Like End-to-End Driving with Vision Language Models

AnyAttack: Targeted Adversarial Attacks on Vision-Language Models toward Any Images

Generating CAD Code with Vision-Language Models for 3D Designs

Is What You Ask For What You Get? Investigating Concept Associations in Text-to-Image Models

CoVLM: Leveraging Consensus from Vision-Language Models for Semi-supervised Multi-modal Fake News Detection