Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

May 31, 2024

StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyond
Pengyuan Lyu, Yulin Li, Hao Zhou, Weihong Ma, Xingyu Wan, Qunyi Xie, Liang Wu, Chengquan Zhang, Kun Yao, Errui Ding, Jingdong Wang
Vision Language Model Knowledge Comprehension Capability Text Rich Image Efficient Vision Language Model
InsightSee: Advancing Multi-agent Vision-Language Models for Enhanced Visual Understanding
Huaxiang Zhang, Yaojia Mu, Guo-Niu Zhu, Zhongxue Gan
Vision Language Model Visual Understanding Multimodal Understanding Visual Processing Insight Generation Visual Interpretation
Information Theoretic Text-to-Image Alignment
Chao Wang, Giulio Franzese, Alessandro Finamore, Massimo Gallo, Pietro Michiardi
Vision Language Model Text to Image Image Text Alignment Model Alignment Optimal Alignment
Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images
Mansi Kakkar, Dattesh Shanbhag, Chandan Aladahalli, Gurunath Reddy M
Vision Language Model Medical Image Contrastive Language Image Radiology Report Generation Anatomical Segmentation Multi Modal Magnetic Resonance Language Augmentation

May 30, 2024

OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation
Gonca Yilmaz, Songyou Peng, Marc Pollefeys, Francis Engelmann, Hermann Blum
Domain Adaptation Vision Language Model Open Vocabulary Open Vocabulary Segmentation Open Domain Generalization
Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training
Aisha Urooj Khan, John Garrett, Tyler Bradshaw, Lonie Salkowski, Jiwoong Jason Jeong, Amara Tariq, Imon Banerjee
Vision Language Model Domain Shift Visual Language Model Medical Domain Text to Image Retrieval Adaptation Strategy

May 29, 2024

Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding
Shenghuan Sun, Alexander Schubert, Gregory M. Goldgof, Zhiqing Sun, Thomas Hartvigsen, Atul J. Butte, Ahmed Alaa
Vision Language Model Visual Instruction Tuning Clinical Reasoning Symbol Grounding
Evaluating Vision-Language Models on Bistable Images
Artemis Panagopoulou, Coby Melkin, Chris Callison-Burch
Vision Language Model Visual Stimulus Bistable Image
I Bet You Did Not Mean That: Testing Semantic Importance via Betting
Jacopo Teneggi, Jeremias Sulam
Vision Language Model Feature Importance Significance Testing Semantic Importance
Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design
Markus J. Buehler
Large Language Model Vision Language Model Product Design Image Text Pair Text to 3D Larger OCTOPUS Multimodal Vision Bio Inspired Material
Enhancing Vision-Language Model with Unmasked Token Alignment
Jihao Liu, Jinliang Zheng, Boxiao Liu, Yu Liu, Hongsheng Li
Vision Language Model Image Text Pair Contrastive Pre Training Token Alignment Unmasked Token

May 28, 2024

Low-Rank Few-Shot Adaptation of Vision-Language Models
Maxime Zanella, Ismail Ben Ayed
Vision Language Model Shot Learning Adaptation Concern Shot Adaptation Shot Vision Language
WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization
Jiawei Ma, Yulei Niu, Shiyuan Huang, Guangxing Han, Shih-Fu Chang
Vision Language Model Domain Invariant Representation Single Source Domain Generalization Shape Disentanglement
Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?
Yifan Bai, Dongming Wu, Yingfei Liu, Fan Jia, Weixin Mao, Ziheng Zhang, Yucheng Zhao, Jianbing Shen, Xing Wei, Tiancai Wang, Xiangyu Zhang
Language Model Autonomous Driving Vision Language Model Image Tokenizer Reliable Autonomous Driving
Frustratingly Easy Test-Time Adaptation of Vision-Language Models
Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, Elisa Ricci
Vision Language Model Test Time Adaptation Vision Encoders Time Adaptation Test Time Prompt Tuning
Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
Xin Xiao, Bohong Wu, Jiacong Wang, Chunyuan Li, Xun Zhou, Haoyuan Guo
Vision Language Model Cross Modal Alignment Contrastive Alignment Object Correlation

May 27, 2024

Cross-Modal Safety Alignment: Is textual unlearning all you need?
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit K. Roy-Chowdhury, Chengyu Song
Vision Language Model Multi Modal Model Sentence Level Attack
An Introduction to Vision-Language Modeling
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie, Pietro Astolfi, Reyhane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri, Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, Vikas Chandra
Generative Model Vision Language Model Gentle Introduction
Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View
Jin Wang, Shichao Dong, Yapeng Zhu, Kelu Yao, Weidong Zhao, Chao Li, Ping Luo
Vision Language Model Game Theoretic Compositional Structure Compositional Reasoning Compositional Representation
Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks
Yunqi Zhang, Songda Li, Chunyuan Deng, Luyi Wang, Hui Zhao
Vision Language Model Gender Bias Vision Language Task Two Stage

Vision Language Model

Papers

StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyond

InsightSee: Advancing Multi-agent Vision-Language Models for Enhanced Visual Understanding

Information Theoretic Text-to-Image Alignment

Language Augmentation in CLIP for Improved Anatomy Detection on Multi-modal Medical Images

OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation

Knowledge-grounded Adaptation Strategy for Vision-language Models: Building Unique Case-set for Screening Mammograms for Residents Training

Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding

Evaluating Vision-Language Models on Bistable Images

I Bet You Did Not Mean That: Testing Semantic Importance via Betting

Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design

Enhancing Vision-Language Model with Unmasked Token Alignment

Low-Rank Few-Shot Adaptation of Vision-Language Models

WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization

Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?

Frustratingly Easy Test-Time Adaptation of Vision-Language Models

Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

Cross-Modal Safety Alignment: Is textual unlearning all you need?

An Introduction to Vision-Language Modeling

Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View

Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks