Vision Language Model

Vision-language models (VLMs) integrate visual and textual information to perform complex tasks, aiming to bridge the gap between computer vision and natural language processing. Current research focuses on improving VLM efficiency and robustness through techniques like prompt tuning, which optimizes textual or visual prompts for specific tasks, and sparse token optimization to reduce computational overhead. These advancements are significant because they enable VLMs to be applied to diverse real-world applications, including robotics, autonomous driving, medical image analysis, and fake news detection, while addressing challenges like hallucinations and model miscalibration.

Papers

December 5, 2024

Discriminative Fine-tuning of LVLMs
Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis, Ioannis Maniadis Metaxas, Georgios Tzimiropoulos, Brais Martinez
Vision Language Model Large Vision Language Model Vision Language Representation Vision Language Reasoning
3D Part Segmentation via Geometric Aggregation of 2D Visual Features
Marco Garosi, Riccardo Tedoldi, Davide Boscaini, Massimiliano Mancini, Nicu Sebe, Fabio Poiesi
Vision Language Model Arbitrary Object 2 Dimensional Part Segmentation Geometric Structure Spatial Global Local Aggregation
MegaCOIN: Enhancing Medium-Grained Color Perception for Vision-Language Models
Ming-Chang Chiu, Shicheng Wen, Pin-Yu Chen, Xuezhe Ma
Vision Language Model Full Model Multimodal Phenomenon Human Annotation Visual Task Human Labeled Bitcoin User Color Perception
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance
Chu Myaet Thwal, Ye Lin Tun, Minh N. H. Nguyen, Eui-Nam Huh, Choong Seon Hong
Vision Language Model Contrastive Language Image Cross Modal Alignment Contrastive Learning Objective Contrastive Self Supervision Efficient Vision Language Model Guidance Network
LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model
Yuan Xue, Qi Zhang, Chuanmin Jia, Shiqi Wang
Vision Language Model Image Compression Linear Compression Computer Vision Model Reference Image Quality Assessment

December 4, 2024

Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension
Wang Xiyao, Yang Zhengyuan, Li Linjie, Lu Hongjin, Xu Yuancheng, Lin Chung-Ching Lin, Lin Kevin, Huang Furong, Wang Lijuan
Vision Language Model Vision Model Large Relevance Improvement Visual Understanding Inference Time Multimodal Benchmark
PaliGemma 2: A Family of Versatile VLMs for Transfer
Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, Xiaohua Zhai
Language Model Vision Language Model Formality Transfer Vision Encoders Single Parent Family
A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for accelerating Large VLMs
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, Yang You
Vision Language Model Large Vision Language Model Attention Map Global Attention Invisible Stitch
AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?
Shouwei Ruan, Hanqin Liu, Yao Huang, Xiaoqi Wang, Caixin Kang, Hang Su, Yinpeng Dong, Xingxing Wei
Vision Language Model Robust 3D Image Variation Adversarial Transformation Adversarial Optimization
Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis
Po-Hsuan Huang, Jeng-Lin Li, Chin-Po Chen, Ming-Ching Chang, Wei-Chao Chen
Vision Language Model Large Vision Language Model Content Hallucination Human Saliency Visual Input Multimodal Hallucination Causality Analysis

December 3, 2024

Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty
Roman Colman, Minh Vu, Manish Bhattarai, Martin Ma, Hari Viswanathan, Daniel O'Malley, Javier E. Santos
Language Model Vision Language Model App to App Retrieval Optical Character Recognition Model Uncertainty Information Extraction Task Information Accuracy
CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs
Abhas Kumar, Kapil Pathak, Rajesh Kavuru, Prabhakar Srinivasan
Language Model Vision Language Model High Efficiency Model Performance Sustainable Artificial Intelligence
OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation
Junyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang, Zichen Wen, Ying Li, Ka-Ho Chow, Conghui He, Wentao Zhang
Vision Language Model Retrieval Augmented Generation Optical Character Recognition Cascade Model Unstructured Information Semantic Noise
Remote Sensing Temporal Vision-Language Models: A Comprehensive Survey
Chenyang Liu, Jiafan Zhang, Keyan Chen, Man Wang, Zhengxia Zou, Zhenwei Shi
Vision Language Model Comprehensive Survey Vision Language Task Temporal Image Remote Sensing Vision Language
SJTU:Spatial judgments in multimodal models towards unified segmentation through coordinate detection
Joongwon Chae, Zhenyu Wang, Peiwu Qin
Vision Language Model Multimodal Model Multi Modal Model Spatial Reasoning Vision Language Understanding General Object Unified Segmentation
Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks
Jinjin Cai, Kexin Meng, Baijian Yang, Gang Shao
Vision Language Model Large Vision Language Model Attention Network Multimodal Representation Scene Classification Understanding Cross Remote Sensing Scene Classification
BYE: Build Your Encoder with One Sequence of Exploration Data for Long-Term Dynamic Scene Understanding
Chenguang Huang, Shengchao Yan, Wolfram Burgard
Point Cloud Vision Language Model Scene Understanding Sequence of Sequence Dynamic Object Security Closure
Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation
Sepand Dyanatkar, Angran Li, Alexander Dungate
Vision Language Model Computer Vision Visual Concept Kg Rag Scalable Model Marine Environment Environmental Conservation Ocean Data

December 2, 2024

COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Sanghwan Kim, Rui Xiao, Mariana-Iuliana Georgescu, Stephan Alaniz, Zeynep Akata
Vision Language Model Vision Language Contrastive Loss Self Distillation
VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval
Dhiman Paul, Md Rizwan Parvez, Nabeel Mohammed, Shafin Rahman
Vision Language Model Moment Retrieval Video Text Alignment Task Alignment Video Highlight Detection Cross Modal Fusion Network

Vision Language Model

Papers

Discriminative Fine-tuning of LVLMs

3D Part Segmentation via Geometric Aggregation of 2D Visual Features

MegaCOIN: Enhancing Medium-Grained Color Perception for Vision-Language Models

CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

LL-ICM: Image Compression for Low-level Machine Vision via Large Vision-Language Model

Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension

PaliGemma 2: A Family of Versatile VLMs for Transfer

A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for accelerating Large VLMs

AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?

Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis

Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty

CEGI: Measuring the trade-off between efficiency and carbon emissions for SLMs and VLMs

OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation

Remote Sensing Temporal Vision-Language Models: A Comprehensive Survey

SJTU:Spatial judgments in multimodal models towards unified segmentation through coordinate detection

Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks

BYE: Build Your Encoder with One Sequence of Exploration Data for Long-Term Dynamic Scene Understanding

Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation

COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training

VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval