Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

July 12, 2024

OVExp: Open Vocabulary Exploration for Object-Oriented Navigation
Meng Wei, Tai Wang, Yilun Chen, Hanqing Wang, Jiangmiao Pang, Xihui Liu
Vision Language Model Open Vocabulary Scene Representation Goal Conditioned Exploration
LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models
Yabin Zhang, Wenjie Zhu, Chenhang He, Lei Zhang
Vision Language Model Distribution Detection Prompt Learning Prompt Based Pseudo

July 11, 2024

CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting
Naman Sharma
Vision Language Model Large Vision Language Model High Uncertainty Anticipation Chest Radiograph CXR Lt
Extracting Training Data from Document-Based VQA Models
Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr, Federico Tombari
Vision Language Model Training Data Visual Question Answering Limited Memorization Page Document VQA Visual Argument
NODE-Adapter: Neural Ordinary Differential Equations for Better Vision-Language Reasoning
Yi Zhang, Chun-Wun Cheng, Ke Yu, Zhihai He, Carola-Bibiane Schönlieb, Angelica I. Aviles-Rivero
Vision Language Model Neural Ordinary Differential Equation Visual Reasoning Vision Language Reasoning Modal Prototype
Specialist vision-language models for clinical ophthalmology
Robbie Holland, Thomas R. P. Taylor, Christopher Holmes, Sophie Riedl, Julia Mai, Maria Patsiamanidi, Dimitra Mitsopoulou, Paul Hager, Philip Müller, Hendrik P. N. Scholl, Hrvoje Bogunović, Ursula Schmidt-Erfurth, Daniel Rueckert, Sobha Sivaprasad, Andrew J. Lotery, Martin J. Menten
Vision Language Model Vision Language Real World Clinical Clinical Task
Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization
Jinlong Li, Dong Zhao, Zequn Jie, Elisa Ricci, Lin Ma, Nicu Sebe
Native Robustness Zero Shot Vision Language Model Fine Tuning Consistency Regularization Orthogonal Statistical Learning Orthogonal Fine Tuning Orthogonal Training
AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization
Shixiong Xu, Chenghao Zhang, Lubin Fan, Gaofeng Meng, Shiming Xiang, Jieping Ye
Vision Language Model Visual Localization Image Matching Address Matching
Live Fitness Coaching as a Testbed for Situated Interaction
Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, Roland Memisevic
Vision Language Model Human AI Interaction Test Bed Situated Reasoning Conversational AI Assistant AI Coach

July 10, 2024

Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison
Qian Yang, Weixiang Yan, Aishwarya Agrawal
Vision Language Model Vision Language Task Microbial Decomposition Self Consistency Answer Correctness Task Consistency Goal Consistency Measure
PaliGemma: A versatile 3B VLM for transfer
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, Xiaohua Zhai
Language Model Vision Language Model Formality Transfer Open World
Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
Zhifang Zhang, Beibei Li
Vision Language Model Image Text Pair High Quality Representation Candidate Label Label Disambiguation Prompt Alignment
Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systems
Chashi Mahiul Islam, Shaeke Salman, Montasir Shams, Xiuwen Liu, Piyush Kumar
Vision Language Model Robot Navigation Vision and Language Navigation Language Navigation Real World eXploitation Malicious Attack

July 9, 2024

CoLA: Conditional Dropout and Language-driven Robust Dual-modal Salient Object Detection
Shuang Hao, Chunlin Zhong, He Tang
Vision Language Model Structured Dropout Salient Object Detection
Vision language models are blind
Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, Anh Totti Nguyen
Vision Language Model Vision Benchmark Low Level Vision Task Vision Capability

July 8, 2024

Multi-Object Hallucination in Vision-Language Models
Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David F. Fouhey, Joyce Chai
Vision Language Model Large Vision Language Model Linear Probing Visual Hallucination Object Hallucination
Vision-Language Models under Cultural and Inclusive Considerations
Antonia Karamolegkou, Phillip Rust, Yong Cao, Ruixiang Cui, Anders Søgaard, Daniel Hershcovich
Vision Language Model Large Vision Language Model Local Culture Vision Assistant Cross CUltural Understanding Benchmark Inclusive Approach
Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding
Aaron Lohner, Francesco Compagno, Jonathan Francis, Alessandro Oltramari
Vision Language Model Scene Graph Traffic Accident Accident Datasets

July 7, 2024

Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models
Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Hantao Zhou, Hengshuang Zhao, Xiu Li, Jiaya Jia
Knowledge Distillation Vision Language Model Pre Trained Vision Language Continual Learning Incremental Learning Human Mind Catastrophic Interference Knowledge Integration Continual Learning Scenario
WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, Alexandre Drouin
Vision Language Model Autonomous Agent Planning Domain

Vision Language Model

Papers

OVExp: Open Vocabulary Exploration for Object-Oriented Navigation

LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models

CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting

Extracting Training Data from Document-Based VQA Models

NODE-Adapter: Neural Ordinary Differential Equations for Better Vision-Language Reasoning

Specialist vision-language models for clinical ophthalmology

Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization

AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization

Live Fitness Coaching as a Testbed for Situated Interaction

Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison

PaliGemma: A versatile 3B VLM for transfer

Tuning Vision-Language Models with Candidate Labels by Prompt Alignment

Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systems

CoLA: Conditional Dropout and Language-driven Robust Dual-modal Salient Object Detection

Vision language models are blind

Multi-Object Hallucination in Vision-Language Models

Vision-Language Models under Cultural and Inclusive Considerations

Enhancing Vision-Language Models with Scene Graphs for Traffic Accident Understanding

Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models

WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks