Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

October 12, 2024

CLIP-SCGI: Synthesized Caption-Guided Inversion for Person Re-Identification
Qianru Han, Xinwei He, Zhi Liu, Sannyuya Liu, Ying Zhang, Jinhai Xiang
Vision Language Model Person Re Identification Contrastive Language Image Person Name Critical Synthesis Whistleblower Re Identification Image Captioning Model
Debiasing Vison-Language Models with Text-Only Training
Yunfan Yang, Chaoquan Jiang, Zhiyu Lin, Jinlin Xiao, Jiaming Zhang, Jitao Sang
Large Language Model Vision Language Model Multi Target Visual Bias Text Only Training

October 11, 2024

Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos
Harsh Mahesheka, Zhixian Xie, Zhaoran Wang, Wanxin Jin
Vision Language Model Bilevel Optimization Task Specific Model Reward Learning Complex Behavior Reward Design Online Video
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models
Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba
Vision Language Model Multimodal Representation Safety Alignment Unraveling Key Factor
Calibrated Cache Model for Few-Shot Vision-Language Model Adaptation
Kun Ding, Qiang Yu, Haojian Zhang, Gaofeng Meng, Shiming Xiang
Vision Language Model Pre Trained Projection Based Margin Calibration
RoRA-VLM: Robust Retrieval-Augmented Vision Language Models
Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, Lifu Huang
Vision Language Model Vision Language Task Retrieval Augmented Retrieval Augmentation
VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model
Beichen Wang, Juexiao Zhang, Shuwen Dong, Irving Fang, Chen Feng
Language Model Vision Language Model Robot Person Robot Learning Visual Perception Human Demonstration Robot Plan
Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models
Reza Abbasi, Sernam Lim
Large Language Model Vision Language Model Artificial Intelligence Model Large Model Based Model GPU Memory Large AI Model Stage Pipeline
Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models
Mengyuan Chen, Junyu Gao, Changsheng Xu
Vision Language Model Distribution Detection Semantic Scale Zero Shot Out of Distribution Distribution Identification
Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP
Eunji Kim, Kyuhong Shim, Simyung Chang, Sungroh Yoon
Vision Language Model Jina Embeddings Single CLIP Text Embeddings Text Encoder Interpretable Way Semantic Token Text Control

October 10, 2024

Q-VLM: Post-training Quantization for Large Vision-Language Models
Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, Jiwen Lu
Vision Language Model Large Vision Language Model Multi Modal Quantization Operator Post Training Quantization Optimal Quantization
Unsupervised Data Validation Methods for Efficient Model Training
Yurii Paniv
Natural Language Processing Vision Language Model Unsupervised Learning Low Resource Language Model Training Speech to Text Cross Lingual Transfer Learning
HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter
Yumiao Zhao, Bo Jiang, Xiao Wang, Qin Xu, Jin Tang
Vision Language Model Heterogeneous Graph Heterogeneous Graph Neural Network Task Specific Knowledge Multi Modal Knowledge
A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
Hoin Jung, Taeuk Jang, Xiaoqian Wang
Vision Language Model Constructive Approach New Task Different Modality Zero Shot Classification Text to Image Retrieval Tabular Data Imputation
How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?
Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee, Hoyeon Chang, Sue Hyun Park, Minjoon Seo
Vision Language Model Human SAFETY Safety Fine Tuning Safety Layer

October 9, 2024

Towards Interpreting Visual Information Processing in Vision-Language Models
Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, Fazl Barez
Vision Language Model Vision Model Visual Token Visual Processing Task Encoding Token
VHELM: A Holistic Evaluation of Vision Language Models
Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, Percy Liang
Vision Language Model Evaluation Metric Visual Perception Holistic Evaluation Bias Benchmark
Pixtral 12B
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang
Vision Language Model Multimodal LLM Multimodal Task Multimodal Benchmark
Preference Fine-Tuning for Factuality in Chest X-Ray Interpretation Models Without Human Feedback
Dennis Hein, Zhihong Chen, Sophie Ostmeier, Justin Xu, Maya Varma, Eduardo Pontes Reis, Arne Edward Michalson, Christian Bluethgen, Hyun Joo Shin, Curtis Langlotz, Akshay S Chaudhari
Vision Language Model Medical Image Human Feedback Factual Claim Preference Alignment Preference Fine Tuning
Compositional Entailment Learning for Hyperbolic Vision-Language Models
Avik Pal, Max van Spengler, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, Pascal Mettes
Vision Language Model Vision Language Representation Hyperbolic Representation Entailment Tree

Vision Language Model

Papers

CLIP-SCGI: Synthesized Caption-Guided Inversion for Person Re-Identification

Debiasing Vison-Language Models with Text-Only Training

Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videos

Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models

Calibrated Cache Model for Few-Shot Vision-Language Model Adaptation

RoRA-VLM: Robust Retrieval-Augmented Vision Language Models

VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model

Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models

Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP

Q-VLM: Post-training Quantization for Large Vision-Language Models

Unsupervised Data Validation Methods for Efficient Model Training

HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter

A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks

How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?

Towards Interpreting Visual Information Processing in Vision-Language Models

VHELM: A Holistic Evaluation of Vision Language Models

Pixtral 12B

Preference Fine-Tuning for Factuality in Chest X-Ray Interpretation Models Without Human Feedback

Compositional Entailment Learning for Hyperbolic Vision-Language Models