Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

November 12, 2024

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, Chong Ruan
Large Language Model Vision Language Model Generative Modeling Faithful Generation Autoregressive Model Autoregressive Language Model Multimodal Understanding Image Understanding Rectified Flow
BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, Etash Guha, Silvio Savarese, Ludwig Schmidt, Yejin Choi, Caiming Xiong, Ran Xu
Vision Language Model Large Vision Language Model Vision Language Task Synthetic Caption Knowledge Augmentation Dense Captioning

November 11, 2024

UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models
Jiachen Liang, Ruibing Hou, Minyang Hu, Hong Chang, Shiguang Shan, Xilin Chen
Vision Language Model Full Model Unsupervised Setting Visual Encoder Domain Specific Feature Feature Calibration Domain Calibration
Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning
Hongsheng Zhang, Zhong Ji, Jingren Liu, Yanwei Pang, Jungong Han
Vision Language Model Continual LEArning Knowledge Transfer Knowledge Integration Multi Stage Framework Feature Forgetting
Renaissance: Investigating the Pretraining of Vision-Language Encoders
Clayton Fields, Casey Kennington
Vision Language Model Vision Language Task Transformer Encoders

November 9, 2024

Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models
Arshia Hemmat, Adam Davies, Tom A. Lamb, Jianhao Yuan, Philip Torr, Ashkan Khakzar, Francesco Pinto
Vision Language Model Large Vision Language Model Object Recognition Hidden Knowledge Plain Sight Robust Perception
Aquila: A Hierarchically Aligned Visual-Language Model for Enhanced Remote Sensing Image Comprehension
Kaixuan Lu, Ruiqian Zhang, Xiao Huang, Yuxing Xie
Vision Language Model Large Vision Language Model Remote Sensing Image Visual Language Model Visual Language Language Foundation Model

November 8, 2024

End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
Dylan Goetting, Himanshu Gaurav Singh, Antonio Loquercio
Zero Shot Vision Language Model Complex Reasoning Spatial Domain GPS Trajectory Navigation Policy
Poze: Sports Technique Feedback under Data Constraints
Agamdeep Singh, Sujit PB, Mayank Vatsa
Vision Language Model Human Motion Technique Recognition
Enhancing Visual Classification using Comparative Descriptors
Hankyeol Lee, Gawon Seo, Wonseok Choi, Geunyoung Jung, Kyungwoo Song, Jiyoung Jung
Vision Language Model Novel Descriptor Image Embeddings Visual Classification Zero Shot Classification Task

November 7, 2024

On Erroneous Agreements of CLIP Image Embeddings
Siting Li, Pang Wei Koh, Simon Shaolei Du
Vision Language Model Visual Reasoning Human Answer Mistake Vision Language Alignment Image Embeddings CLIP Embeddings
A Reinforcement Learning-Based Automatic Video Editing Method Using Pre-trained Vision-Language Model
Panwen Hu, Nan Xiao, Feifei Li, Yongquan Chen, Rui Huang
Vision Language Model Positive Reinforcement Video Editing Scene Level Representation Editing Editing Method
In the Era of Prompt Learning with Vision-Language Models
Ankit Jha
Vision Language Model Domain Generalization Unsupervised Domain Adaptation Prompt Learning New Era Domain Invariant Prompt
Vision Language Models are In-Context Value Learners
Yecheng Jason Ma, Joey Hejna, Ayzaan Wahid, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Jonathan Tompson, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, Fei Xia
Vision Language Model Context Learning Value Function Visuomotor Policy Pixel Trajectory Temporal Evolution
BendVLM: Test-Time Debiasing of Vision-Language Embeddings
Walter Gerych, Haoran Zhang, Kimia Hamidieh, Eileen Pan, Maanas Sharma, Thomas Hartvigsen, Marzyeh Ghassemi
Vision Language Model Text Guided Image Generation Encode Bias

November 6, 2024

Unfair Alignment: Examining Safety Alignment Across Vision Encoder Layers in Vision-Language Models
Saketh Bachu, Erfan Shayegani, Trishna Chakraborty, Rohit Lal, Arindam Dutta, Chengyu Song, Yue Dong, Nael Abu-Ghazaleh, Amit K. Roy-Chowdhury
Large Language Model Vision Language Model Multi Layer Safety Alignment Last Layer
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?
Daniel P. Jeong, Saurabh Garg, Zachary C. Lipton, Michael Oberst
Large Language Model Vision Language Model Large Language Much Progress Domain Specific Model Biomedical Corpus Medical Task Adaptation
RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models
Maya Varma, Jean-Benoit Delbrouck, Zhihong Chen, Akshay Chaudhari, Curtis Langlotz
Fine Grained Vision Language Model Spurious Correlation Zero Shot Classification LLM Robustness
Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval
Davide Buoso, Luke Robinson, Giuseppe Averta, Philip Torr, Tim Franzmeyer, Daniele De Martini
Vision Language Model Task Planning Autonomous Navigation Path Planning Memory Trace Training Free
Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models
Minh Duc Bui, Katharina von der Wense, Anne Lauscher
Vision Language Model Hate Speech Multimodal Phenomenon Multilingual Text

Vision Language Model

Papers

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions

UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models

Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

Renaissance: Investigating the Pretraining of Vision-Language Encoders

Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models

Aquila: A Hierarchically Aligned Visual-Language Model for Enhanced Remote Sensing Image Comprehension

End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering

Poze: Sports Technique Feedback under Data Constraints

Enhancing Visual Classification using Comparative Descriptors

On Erroneous Agreements of CLIP Image Embeddings

A Reinforcement Learning-Based Automatic Video Editing Method Using Pre-trained Vision-Language Model

In the Era of Prompt Learning with Vision-Language Models

Vision Language Models are In-Context Value Learners

BendVLM: Test-Time Debiasing of Vision-Language Embeddings

Unfair Alignment: Examining Safety Alignment Across Vision Encoder Layers in Vision-Language Models

Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?

RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models

Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval

Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models