Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

October 16, 2024

Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions
Zhenyu Jiang, Yuqi Xie, Jinhan Li, Ye Yuan, Yifeng Zhu, Yuke Zhu
Vision Language Model Human Language Humanoid Robot Description Library Motion Prior Whole Body Motion Body Motion Generation
WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
João Matos, Shan Chen, Siena Placino, Yingya Li, Juan Carlos Climent Pardo, Daphna Idan, Takeshi Tohyama, David Restrepo, Luis F. Nakayama, Jose M. M. Pascual-Leone, Guergana Savova, Hugo Aerts, Leo A. Celi, A. Ian Wong, Danielle S. Bitterman, Jack Gallifant
Vision Language Model Multimodal Phenomenon Multiple Choice Multimodal Language Model Medical VQA
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Ching Lam Cheng, Daud Abolade, Emmanuele Chersoni, Enrico Santus, Fariz Ikhwantri, Garry Kuwanto, Hanyang Zhao, Haryo Akbarianto Wibowo, Holy Lovenia, Jan Christian Blaise Cruz, Jan Wira Gotama Putra, Junho Myung, Lucky Susanto, Maria Angelica Riera Machin, Marina Zhukova, Michael Anugraha, Muhammad Farid Adilazuarda, Natasha Santosa, Peerat Limkonchotiwat, Raj Dabre, Rio Alexander Audino, Samuel Cahyawijaya, Shi-Xiong Zhang, Stephanie Yulia Salim, Yi Zhou, Yinxuan Gui, David Ifeoluwa Adelani, En-Shiun Annie Lee, Shogo Okada, Ayu Purwarianti, Alham Fikri Aji, Taro Watanabe, Derry Tanti Wijaya, Alice Oh, Chong-Wah Ngo et al. (6 additional authors not shown) You must enabled JavaScript to view entire author list.
Vision Language Model Language Understanding Visual Question Large Scale Benchmark Food Development Cultural Background VQA Benchmark
VividMed: Vision Language Model with Versatile Visual Grounding for Medicine
Lingxiao Luo, Bingda Tang, Xuanzhong Chen, Rong Han, Ting Chen
Vision Language Model Visual Grounding
Tokens on Demand: Token Condensation as Training-free Test-time Adaptation
Zixin Wang, Dong Gong, Sen Wang, Zi Huang, Yadan Luo
Vision Language Model Adaptation Concern K TOKEN Image Token Video Game Demand Token Sparsification Graph Out of Distribution Adaptation
Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety
Lucas Choi, Ross Greer
Zero Shot Vision Language Model Data Detection Detection Method Safety Climate Perception Safety Mechanism

October 15, 2024

Enabling Data-Driven and Empathetic Interactions: A Context-Aware 3D Virtual Agent in Mixed Reality for Enhanced Financial Customer Experience
Cindy Xu, Mengyu Chen, Pranav Deshpande, Elvir Azanli, Runqing Yang, Joseph Ligman
Vision Language Model Mixed Reality Situational Awareness Empathetic Dialogue Customer Experience
Dynamic Open-Vocabulary 3D Scene Graphs for Long-term Language-Guided Mobile Manipulation
Zhijie Yan, Shufei Li, Zuoxu Wang, Lixiu Wu, Han Wang, Jun Zhu, Lijiang Chen, Jihong Liu
Vision Language Model Real World Long Term Mobile Manipulation 3D Scene Graph
PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model
Shang-Ching Liu, Van Nhiem Tran, Wenkai Chen, Wei-Lun Cheng, Yen-Lin Huang, I-Bin Liao, Yung-Hui Li, Jianwei Zhang
Point Cloud Vision Language Model Affordance Learning
Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs
Shuo Li, Tao Ji, Xiaoran Fan, Linsheng Lu, Leyi Yang, Yuming Yang, Zhiheng Xi, Rui Zheng, Yuran Wang, Xiaohui Zhao, Tao Gui, Qi Zhang, Xuanjing Huang
Vision Language Model Study Feature Visual Language Model Visual Attention Visual Token Visual Sycophancy
Tree of Attributes Prompt Learning for Vision-Language Models
Tong Ding, Wanhua Li, Zhongqi Miao, Hanspeter Pfister
Zero Shot Vision Language Model Prompt Learning Visual Attribute

October 14, 2024

Locality Alignment Improves Vision-Language Models
Ian Covert, Tony Sun, James Zou, Tatsunori Hashimoto
Vision Language Model Pre Trained Backbone Vision Backbone Local Alignment Image Semantics
Towards Foundation Models for 3D Vision: How Close Are We?
Yiming Zuo, Karhan Kayan, Maggie Wang, Kevin Jeon, Jia Deng, Thomas L. Griffiths
Vision Language Model Foundation Model 3D Perception 3D Reasoning 3D Vision Task 3D Visual Understanding
LG-CAV: Train Any Concept Activation Vector with Language Guidance
Qihan Huang, Jie Song, Mengqi Xue, Haofei Zhang, Bingde Hu, Huiqiong Wang, Hao Jiang, Xingen Wang, Mingli Song
Vision Language Model Explainable AI Pre Trained Vision Language Model Concept Activation Vector
LOBG:Less Overfitting for Better Generalization in Vision-Language Model
Chenhao Ding, Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, Alex Kot, Yihong Gong
Vision Language Model Strong Generalization Model Overfitting Feature Space Better Generalization Prompt Learning Method
MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, Lijuan Wang, Huaxiu Yao
Vision Language Model Large Vision Language Model Multimodal Learning Multimodal Comprehension Multimodal Query
Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
Jun Luo, Chen Chen, Shandong Wu
Vision Language Model Vision Language Mixture Component Expert Knowledge Prompt Optimization Personalized Subject Federated Prompt

October 13, 2024

ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
Xinxin Zhao, Wenzhe Cai, Likun Tang, Teng Wang
Vision Language Model Visual Navigation Visual Navigation Task Embodied Visual Navigation
MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
Hang Hua, Yunlong Tang, Ziyun Zeng, Liangliang Cao, Zhengyuan Yang, Hangfeng He, Chenliang Xu, Jiebo Luo
Fine Grained Vision Language Model Large Vision Language Model Cross Modal Retrieval Multimodal Understanding Compositional Language

October 12, 2024

Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Dataset
Haoming Lu, Feifei Zhong
Vision Language Model Case Study Human Annotation Annotation Tool

Vision Language Model

Papers

Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions

WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation

WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines

VividMed: Vision Language Model with Versatile Visual Grounding for Medicine

Tokens on Demand: Token Condensation as Training-free Test-time Adaptation

Evaluating Cascaded Methods of Vision-Language Models for Zero-Shot Detection and Association of Hardhats for Increased Construction Safety

Enabling Data-Driven and Empathetic Interactions: A Context-Aware 3D Virtual Agent in Mixed Reality for Enhanced Financial Customer Experience

Dynamic Open-Vocabulary 3D Scene Graphs for Long-term Language-Guided Mobile Manipulation

PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model

Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs

Tree of Attributes Prompt Learning for Vision-Language Models

Locality Alignment Improves Vision-Language Models

Towards Foundation Models for 3D Vision: How Close Are We?

LG-CAV: Train Any Concept Activation Vector with Language Guidance

LOBG:Less Overfitting for Better Generalization in Vision-Language Model

MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models

Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models

ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination

MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models

Can Vision-Language Models Replace Human Annotators: A Case Study with CelebA Dataset