Generative Vision Language Model

Generative Vision-Language Models (VLMs) aim to create systems that seamlessly understand and generate both visual and textual information, bridging the gap between image and text modalities. Current research focuses on improving the quality and reliability of VLM outputs, addressing issues like hallucinations and biases through techniques such as direct preference optimization and multi-modal mutual information decoding, often within transformer-based architectures. These advancements are significant because they enable more robust and reliable applications in diverse fields, including robotics, medical image analysis, and content creation, while also providing valuable tools for analyzing and improving the explainability of other AI models.

Papers

October 16, 2024

Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts
Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, Hassan Sajjad
Vision Language Model Human Language Complex Prompt Prompt Tuning Semantics Surfaced Semantic Change Network Sensitivity Generative Vision Language Model

September 25, 2024

Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset
Andrew Goldberg, Kavish Kondap, Tianshuang Qiu, Zehan Ma, Letian Fu, Justin Kerr, Huang Huang, Kaiyuan Chen, Kuan Fang, Ken Goldberg
Robot Person Generative Question Physical Simulation Generative AI System Generative Vision Language Model Global Reset Feature Robotic Experiment Part Assembly

August 24, 2024

Explainable Concept Generation through Vision-Language Preference Learning
Aditya Taparia, Som Sagar, Ransalu Senanayake
Preference Learning Explanation Generation Concept Based Concept Based Explanation Generative Vision Language Model

June 10, 2024

Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation
Oishi Banerjee, Hong-Yu Zhou, Subathra Adithan, Stephen Kwak, Kay Wu, Pranav Rajpurkar
Direct Preference Optimization Content Hallucination Radiology Report Generation Chest X Ray Report Generation Generative Vision Language Model

April 1, 2024

Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
Rongjie Li, Yu Wu, Xuming He
LeArning Abstract Zero Shot Related Task Language Correction Vision Language Reasoning Generative Vision Language Model Zero Shot Vision Language Conditioned Caption

March 20, 2024

Multi-Modal Hallucination Control by Visual Information Grounding
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, Stefano Soatto
Mutual Information Visual Grounding Content Hallucination Generative Vision Language Model

February 11, 2024

Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy
Simon Ging, María A. Bravo, Thomas Brox
Vision Language Model Vision Language Visual Question Answering Generative Vision Language Model Classification Datasets Semantic Hierarchy

December 17, 2023

p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models
Haoyuan Wu, Xinyun Zhang, Peng Xu, Peiyu Liao, Xufeng Yao, Bei Yu
Graph Neural Network Vision Language Model Graph Attention Network Attention Matrix Attention Graph Generative Vision Language Model

December 14, 2023

VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, Ying Shan
Language Understanding Faithful Generation Visual Question Answering Text to Image Generation Generative Question Generative Vision Language Model

December 13, 2023

ToViLaG: Your Visual-Language Generative Model is Also An Evildoer
Xinpeng Wang, Xiaoyuan Yi, Han Jiang, Shanlin Zhou, Zhihua Wei, Xing Xie
Image Text Pair Non Toxic Toxic Comment Toxicity Annotation Generative Vision Language Model

October 5, 2023

Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency
Tianhong Li, Sangnie Bhardwaj, Yonglong Tian, Han Zhang, Jarred Barber, Dina Katabi, Guillaume Lajoie, Huiwen Chang, Dilip Krishnan
Text to Image Generation Image Text Unpaired Data Image to Text Cycle Consistency Generative Vision Language Model Joint Image Text Image Text Correlation

August 21, 2023

An Examination of the Compositionality of Large Generative Vision-Language Models
Teli Ma, Rong Li, Junwei Liang
Multimodal Reasoning Compositional Language Exam Document Generative Vision Language Model Syntactic Inductive Bias SYNTAX Score

July 12, 2023

Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning
Gengyuan Zhang, Yurui Zhang, Kerui Zhang, Volker Tresp
Vision Language Model Full Model Visual Encoder Temporal Reasoning Generative Vision Language Model Location Analysis Late Time

June 29, 2023

CLIPAG: Towards Generator-Free Text-to-Image Generation
Roy Ganz, Michael Elad
Vision Language Model Gradient Alignment Generative Vision Language Model

June 2, 2023

Revisiting the Role of Language Priors in Vision-Language Models
Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, Deva Ramanan
Vision Language Model Integral Role Vision Language Understanding Generative Pre Training Vision Language Benchmark Language Prior Generative Vision Language Model

May 12, 2023

WEDGE: A multi-weather autonomous driving dataset built from generative vision-language models
Aboli Marathe, Deva Ramanan, Rahee Walambe, Ketan Kotecha
Adverse Weather Weather Image Generative Vision Language Model Autonomous Perception Weather Benchmark

May 3, 2023

Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime
Chuhan Zhang, Antoine Miech, Jiajun Shen, Jean-Baptiste Alayrac, Pauline Luc
Visual Language Model Task Adaptation Low Data Regime Visual Captioning Generative Vision Language Model Visual Language Task

February 23, 2023

Vision-Language Generative Model for View-Specific Chest X-ray Generation
Hyungyung Lee, Da Young Lee, Wonjae Kim, Jin-Hwa Kim, Tackeun Kim, Jihang Kim, Leonard Sunwoo, Edward Choi
Chest X Ray X Ray Image Medical Information Mart for Intensive Generative Vision Language Model

January 31, 2023

Debiasing Vision-Language Models via Biased Prompts
Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, Stefanie Jegelka
Generative Model Vision Language Model Vision Language Foundation Model Generative Vision Language Model Prompt Bias

September 9, 2022

Pre-training image-language transformers for open-vocabulary tasks
AJ Piergiovanni, Weicheng Kuo, Anelia Angelova
Visual Question Answering Open Vocabulary Transformer Language Model Generative Vision Language Model Language Transformer Visual Entailment Caption Data

Generative Vision Language Model

Papers

Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts

Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset

Explainable Concept Generation through Vision-Language Preference Learning

Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation

Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning

Multi-Modal Hallucination Control by Visual Information Grounding

Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy

p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models

VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation

ToViLaG: Your Visual-Language Generative Model is Also An Evildoer

Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency

An Examination of the Compositionality of Large Generative Vision-Language Models

Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning

CLIPAG: Towards Generator-Free Text-to-Image Generation

Revisiting the Role of Language Priors in Vision-Language Models

WEDGE: A multi-weather autonomous driving dataset built from generative vision-language models

Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime

Vision-Language Generative Model for View-Specific Chest X-ray Generation

Debiasing Vision-Language Models via Biased Prompts

Pre-training image-language transformers for open-vocabulary tasks