Large Vision Language Model

Large Vision-Language Models (LVLMs) integrate computer vision and natural language processing to enable machines to understand and reason about images and text simultaneously. Current research focuses on improving LVLMs' accuracy, efficiency, and robustness, particularly addressing issues like hallucinations (generating inaccurate information), and enhancing their ability to perform multi-level visual perception and reasoning tasks, including quantitative spatial reasoning and mechanical understanding. These advancements are significant for various applications, including medical image analysis, robotics, and autonomous driving, by enabling more reliable and insightful multimodal data processing.

Papers

December 10, 2024

Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios
Jiaqi Fan, Jianhua Wu, Hongqing Chu, Quanbo Ge, Bingzhao Gao
Vision Language Model Large Vision Language Model Traffic Scenario Multimodal Understanding Object Hallucination Reducing Hallucination Semantic Dataset Semantic Enhancement
Maya: An Instruction Finetuned Multilingual Multimodal Model
Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze-Yin Chan, S M Iftekhar Uddin, Shayekh Bin Islam, Roshan Santhosh, Snegha A, Drishti Sharma, Chen Liu, Isha Chaturvedi, Genta Indra Winata, Ashvanth.S, Snehanshu Mukherjee, Alham Fikri Aji
Large Vision Language Model Human Instruction Vision Language Task Multilingual Multimodal

December 9, 2024

Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models
Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu
Large Vision Language Model Mitigating Hallucination Contrastive Decoding Contrastive Pair Contrastive Data
From Uncertainty to Trust: Enhancing Reliability in Vision-Language Models with Uncertainty-Guided Dropout Decoding
Yixiong Fang, Ziran Yang, Zhaorun Chen, Zhuokai Zhao, Jiawei Zhou
Full Model Large Vision Language Model High Uncertainty Anticipation Appropriate Trust Visual Token Perception Error Dropout Method
Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models
Wei Suo, Ji Ma, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, Yanning Zhang
Large Vision Language Model Inference Efficiency
iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
Lianyu Hu, Fanhua Shang, Liang Wan, Wei Feng
Large Vision Language Model Multimodal Model Single Image Image Encoder Multi Image Lossless Performance

December 5, 2024

Discriminative Fine-tuning of LVLMs
Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis, Ioannis Maniadis Metaxas, Georgios Tzimiropoulos, Brais Martinez
Vision Language Model Large Vision Language Model Vision Language Representation Vision Language Reasoning
Unified Framework for Open-World Compositional Zero-shot Learning
Hirunima Jayasekara, Khoi Pham, Nirat Saini, Abhinav Shrivastava
Large Vision Language Model Unified Framework Language Image

December 4, 2024

PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
Ao Wang, Hui Chen, Jianchao Tan, Kefeng Zhang, Xunliang Cai, Zijia Lin, Jungong Han, Guiguang Ding
Full Model Large Vision Language Model Vision Paper Key Value Cache KV Cache Key Value Efficient Generation
A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for accelerating Large VLMs
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Zhikai Li, Yibing Song, Kai Wang, Zhangyang Wang, Yang You
Vision Language Model Large Vision Language Model Attention Map Global Attention Invisible Stitch
Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation
Gianni Franchi, Dat Nguyen Trong, Nacim Belkhir, Guoxuan Xia, Andrea Pilzer
Generative Model Uncertainty Quantification Faithful Generation Large Vision Language Model Human Understanding Text to Image Model Text to Image Text to Image Generation Model
Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis
Po-Hsuan Huang, Jeng-Lin Li, Chin-Po Chen, Ming-Ching Chang, Wei-Chao Chen
Vision Language Model Large Vision Language Model Content Hallucination Human Saliency Visual Input Multimodal Hallucination Causality Analysis

December 3, 2024

Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks
Jinjin Cai, Kexin Meng, Baijian Yang, Gang Shao
Vision Language Model Large Vision Language Model Attention Network Multimodal Representation Scene Classification Understanding Cross Remote Sensing Scene Classification
VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning
Xueqing Wu, Yuheng Ding, Bingxuan Li, Pan Lu, Da Yin, Kai-Wei Chang, Nanyun Peng
Large Vision Language Model Visual Reasoning Language Correction Leg Failure Self Improvement Model Criticism

December 2, 2024

X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models
Zeyi Sun, Ziyang Chu, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuanjun Xiong, Dahua Lin, Jiaqi Wang
Large Vision Language Model Context Generation Autoregressive Vision Language Model
[CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster
Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, Shanghang Zhang
Human Attention Large Vision Language Model Visual Token Visual Encoder Token Pruning Video Token
FastRM: An efficient and automatic explainability framework for multimodal generative models
Gabriela Ben-Melech Stan, Estelle Aflalo, Man Luo, Shachar Rosenman, Tiep Le, Sayak Paul, Shao-Yen Tseng, Vasudev Lal
Explainable AI Large Vision Language Model Multimodal Generative Model Relevance Map
Understanding the World's Museums through Vision-Language Reasoning
Ada-Astrid Balauca, Sanjana Garai, Stefan Balauca, Rasesh Udayakumar Shetty, Naitik Agrawal, Dhwanil Subhashbhai Shah, Yuqian Fu, Xi Wang, Kristina Toutanova, Danda Pani Paudel, Luc Van Gool
Large Language Model Fine Grained Vision Language Large Vision Language Model World Event Vision Language Reasoning

December 1, 2024

VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information
Ryo Kamoi, Yusen Zhang, Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Rui Zhang
Large Vision Language Model Perception Aware Visual Perception Fine Grained Visual Geometric Information
Exploring Large Vision-Language Models for Robust and Efficient Industrial Anomaly Detection
Kun Qian, Tianyu Sun, Wenhong Wang
Anomaly Detection High Efficiency Large Vision Language Model Robust Version Industrial Anomaly Detection Visual Anomaly Detection Anomaly Localization

Large Vision Language Model

Papers

Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios

Maya: An Instruction Finetuned Multilingual Multimodal Model

Delve into Visual Contrastive Decoding for Hallucination Mitigation of Large Vision-Language Models

From Uncertainty to Trust: Enhancing Reliability in Vision-Language Models with Uncertainty-Guided Dropout Decoding

Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models

iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models

Discriminative Fine-tuning of LVLMs

Unified Framework for Open-World Compositional Zero-shot Learning

PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation

A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for accelerating Large VLMs

Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation

Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis

Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks

VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning

X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models

[CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster

FastRM: An efficient and automatic explainability framework for multimodal generative models

Understanding the World's Museums through Vision-Language Reasoning

VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information

Exploring Large Vision-Language Models for Robust and Efficient Industrial Anomaly Detection