VLM Evaluation

VLM: Vision-Language Model

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving
May 1, 2025Zhijie Qiao, Haowei Li, Zhong Cao +1VLM Evaluation Autonomous Driving Benchmarks
Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA
SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs
Mar 11, 2025Parsa Hosseini, Sumit Nawathe, Mazda Moayeri +2VLM Evaluation VLM Robustness
Department of Computer Science University of Maryland
Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
Feb 23, 2025Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang +7VLM Evaluation Vision-Language Models
CUHK · PKU · CUHKSZ +3
Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions
Feb 18, 2025Aditya K Surikuchi, Raquel Fernández, Sandro PezzelleVisual Storytelling Multimodal Learning
Institute for Logic, Language and Computation (ILLC), University of Amsterdam, Netherlands
PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?
Feb 6, 2025Mennatullah SiamLarge Vision-Language Models Vision-Language Grounding
Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking
Dec 30, 2024Sangyun Chung, Youngjoon Yu, Se Yeon Kim +2VLM Evaluation Vision-Language Models
Integrated Vision Language Lab., School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea
Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner
Dec 30, 2024Yitong Zhou, Mingyue Cheng, Qingyang Mao +3Table Structure Recognition VLM Evaluation
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Artificial Intelligence Research Institute, iFLYTEK Co., Ltd
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Jul 16, 2024Haodong Duan, Xinyu Fang, Junming Yang +41VLM Evaluation LLM Evaluation
Privacy-Aware Visual Language Models
May 27, 2024Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1VLM Evaluation VLM Adaptation
Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg

May 1, 2025J/K move · Enter open · S save

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

Zhijie Qiao, Haowei Li, Zhong Cao, Henry X. Liu

Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI 48109, USA · University of Michigan Transportation Research Institute, Ann Arbor, MI 48109, USA

VLM Evaluation Autonomous Driving Benchmarks VLMs for Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving

Open paper AskPDF arXiv

Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assumption is that successive VLM generations will continually improve driving performance and eventually outperform state-of-the-art methods. To systematically examine this assumption, we introduce LightEMMA, a longitudinal framework for evaluating the autonomous driving performance of VLMs. LightEMMA uses a lightweight, unified evaluation protocol that assesses each model's intrinsic driving capability without model-specific fine-tuning, architectural changes, or prompt engineering. Using this protocol, we evaluate 15 models from five major families on the challenging nuScenes prediction benchmark. Empirical findings show that, despite increased model scale and enhanced general reasoning capabilities, successive VLM generations do not consistently achieve better driving performance. Further analysis of driving scenarios reveals recurring failure modes, including overreliance on historical actions and difficulty reconciling conflicting visual cues. These findings highlight the need for domain-specific adaptation to improve the safety of VLM-based autonomous driving systems. The source code is available at https://github.com/michigan-traffic-lab/LightEMMA.