Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

October 1, 2024

September 30, 2024

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models
Qi Wu, Zipeng Fu, Xuxin Cheng, Xiaolong Wang, Chelsea Finn
Vision Language Model Legged Robot Open World Quadrupedal Locomotion Robot Dog Legged Mobile Manipulation
Do Vision-Language Models Really Understand Visual Language?
Buse Giledereli, Yifan Hou, Yilei Tu, Mrinmaya Sachan
Vision Language Model Visual Reasoning Visual Language Fundamental Diagram Visual Query Scientific Chart
UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models
Qiaojun Yu, Siyuan Huang, Xibin Yuan, Zhengkai Jiang, Ce Hao, Xin Li, Haonan Chang, Junbo Wang, Liu Liu, Hongsheng Li, Peng Gao, Cewu Lu
Vision Language Model Robotic Manipulation Affordance Learning Vision Based Object Centric Representation Unified Representation Tool Usage Topic Focus Articulation
Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environments
Mohamed Elnoor, Kasun Weerakoon, Gershom Seneviratne, Ruiqi Xian, Tianrui Guan, Mohamed Khalid M Jaffar, Vignesh Rajagopal, Dinesh Manocha
Vision Language Model Robot Navigation Real World Environment Proprioceptive Sensing Traversability Estimation Terrain Traversability Outdoor Environment
World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering
Jiacong Wang, Bohong Wu, Haiyong Jiang, Xun Zhou, Xin Xiao, Haoyuan Guo, Jun Xiao
Vision Language Model Real World Code World Event Online Filtering Multi Modal Generation Multi Modal Alignment Cross Modal Information
Resolving Positional Ambiguity in Dialogues by Vision-Language Models for Robot Navigation
Kuan-Lin Chen, Tzu-Ti Wei, Li-Tzu Yeh, Elaine Kao, Yu-Chee Tseng, Jen-Jee Chen
Vision Language Model Large Vision Language Model Robot Navigation Dialogue Utterance Autonomous Robot Navigation Natural Language Command
Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function
Chenyi Zhuang, Ying Hu, Pan Gao
Vision Language Model Text to Image Diffusion Model Jina Embeddings Text Embeddings Permanent Magnet Bias Attribute

September 29, 2024

Vision-Language Models are Strong Noisy Label Detectors
Tong Wei, Hao-Tian Li, Chun-Shu Li, Jiang-Xin Shi, Yu-Feng Li, Min-Ling Zhang
Vision Language Model Vision Language Noisy Label Noisy Label Detection
Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method
Bikang Pan, Wei Huang, Ye Shi
Vision Language Model Practical Method Theoretical Analysis Feature Learning Vision Language Foundation Model

September 28, 2024

FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Models
Diego A. B. Moreira, Alef Iury Ferreira, Gabriel Oliveira dos Santos, Luiz Pereira, João Medrado Gondim, Gustavo Bonil, Helena Maia, Nádia da Silva, Simone Tiemi Hashiguti, Jefersson A. dos Santos, Helio Pedrini, Sandra Avila
Vision Language Model Multimodal Model Visual Language Model Image Datasets Topic Bias CLIP Model
DOTA: Distributional Test-Time Adaptation of Vision-Language Models
Zongbo Han, Jialong Yang, Junfan Li, Qinghua Hu, Qianli Xu, Mike Zheng Shou, Changqing Zhang
Vision Language Model Catastrophic Forgetting Vision Language Foundation Model Multiplayer Online Battle Arena
TrojVLM: Backdoor Attack Against Vision Language Models
Weimin Lyu, Lu Pang, Tengfei Ma, Haibin Ling, Chao Chen
Vision Language Model Backdoor Attack Multimodal Model Image to Text Generation

September 27, 2024

Individuation in Neural Models with and without Visual Grounding
Alexey Tikhonov, Lisa Bylinina, Ivan P. Yamshchikov
Vision Language Model Latent Representation Visual Grounding Neural Model CLIP Embeddings Hierarchical Concept Data Driven Identification
Image-guided topic modeling for interpretable privacy classification
Alina Elena Baia, Andrea Cavallaro
Vision Language Model Topic Analysis Privacy Preserving Image Novel Descriptor Private Prediction Private Data Analysis

September 26, 2024

SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation
Xin Li, Siyuan Huang, Qiaojun Yu, Zhengkai Jiang, Ce Hao, Yimeng Zhu, Hongsheng Li, Peng Gao, Cewu Lu
Vision Language Model Assistive Robot Semantic Keypoint Garment Manipulation
DARE: Diverse Visual Question Answering with Robustness Evaluation
Hannah Sterz, Jonas Pfeiffer, Ivan Vulić
Vision Language Model Visual Question Robustness Evaluation Based Model Multi Modal Input VQA Benchmark Text Only Language Model
The Hard Positive Truth about Vision-Language Compositionality
Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, Ranjay Krishna
Vision Language Model Vision Language High Quality Distractors

Vision Language Model

Papers

Vision Language Models See What You Want but not What You See

Ask, Pose, Unite: Scaling Data Acquisition for Close Interactions with Vision Language Models

VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models

Do Vision-Language Models Really Understand Visual Language?

UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models

Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environments

World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering

Resolving Positional Ambiguity in Dialogues by Vision-Language Models for Robot Navigation

Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function

Vision-Language Models are Strong Noisy Label Detectors

Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Models

DOTA: Distributional Test-Time Adaptation of Vision-Language Models

TrojVLM: Backdoor Attack Against Vision Language Models

Individuation in Neural Models with and without Visual Grounding

Image-guided topic modeling for interpretable privacy classification

SKT: Integrating State-Aware Keypoint Trajectories with Vision-Language Models for Robotic Garment Manipulation

DARE: Diverse Visual Question Answering with Robustness Evaluation

The Hard Positive Truth about Vision-Language Compositionality