VLM Evaluation

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

CardsList

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

  1. RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation

    Mar 26, 2026Jiajun Zhang, Yuying Li, Zhixun Li +13Chart-to-Code GenerationCode Generation

    USTC · CASIA · THU +6

  2. Humans vs Vision-Language Models: A Unified Measure of Narrative Coherence

    Mar 26, 2026Nikolai Ilinykh, Hyewon Jang, Shalom Lappin +2Visual StorytellingVLM Evaluation

    Dept. of Philosophy, Linguistics, and Theory of Science, University of Gothenburg · School of Electronic Engineering and Computer Science, Queen Mary University of London · Dept. of Informatics, King’s College London

  3. THEMIS: Towards Holistic Evaluation of MLLMs for Scientific Paper Fraud Forensics

    Mar 26, 2026Tzu-Yen Ma, Bo Zhang, Zichen Tang +20VLM EvaluationImage Forgery Detection

    Beijing University of Posts and Telecommunications

  4. Mechanistically Interpreting Compression in Vision-Language Models

    Mar 26, 2026Veeraraju Elluru, Arth Singh, Roberto Aguero +3Cultural Safety Evaluation of VLMsLVLM Pruning

    Indian Institute of Technology Jodhpur · AIM Intelligence · National Institute of Technology Agartala +3

  5. How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning

    Mar 25, 2026Luyu Yang, Yutong Dai, An Yan +3VLM Evaluation

    Salesforce AI Research

  6. Evaluating Vision-Language Models for Image Quality Assessment using Psychophysical Data

    Mar 25, 2026Imran Mehmood, Imad Ali Shah, Ming Ronnier Luo +1Image Quality AssessmentVLM Evaluation

    School of Engineering, University of Galway, Ireland · Lero the Research Ireland Centre for Software, Ireland · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou, China.

  7. VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

    Mar 25, 2026Qijia He, Xunmei Liu, Hammaad Memon +6Vector Graphics GenerationVision-Language Models

    University of Washington · Allen Institute for Artificial Intelligence · UNC-Chapel Hill

  8. Ukrainian Visual Word Sense Disambiguation Benchmark

    Mar 24, 2026Yurii Laba, Yaryna Mohytych, Ivanna Rohulia +4VLM EvaluationMultilingual VLM Evaluation

    Ukrainian Catholic University 2A Kozelnytska st., Lviv, Ukraine, 79026

  9. MedObvious: Exposing the Medical Moravec's Paradox in VLMs via Clinical Triage

    Mar 24, 2026Ufaq Khan, Umair Nawaz, L D M S S Teja +5VLM EvaluationMedical VLMs

    Mohamed bin Zayed University of Artificial Intelligence, UAE · National Institute of Technology, Silchar · Birmingham City University, UK

  10. ARGENT: Adaptive Hierarchical Image-Text Representations

    Mar 24, 2026Chuong Huynh, Hossein Souri, Abhinav Kumar +3Vision-Language ModelsHierarchical Representation Learning

    University of Maryland, College Park · Samsung Research America, AI Center – Mountain View

  11. Quantized Vision-Language Models for Damage Assessment: A Comparative Study of LLaVA-1.5-7B Quantization Levels

    Mar 24, 2026Takato YasunoVLM QuantizationVLM Evaluation

  12. Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

    Mar 24, 2026Mehmet Kaan ErolCompositional Reasoning in VLMsVLM Evaluation

    Marmara University, Institute of Pure and Applied Sciences

  13. TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration

    Mar 24, 2026Chunxiao Li, Lijun Li, Jing ShaoCultural Safety Evaluation of VLMsAutomated Red Teaming

    Shanghai Artificial Intelligence Laboratory

  14. How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos

    Mar 24, 2026Wentao Lu, Alexander Senchenko, Alan Sayle +2VLM EvaluationVideo Anomaly Detection

    University of Alberta Edmonton, Alberta, Canada · Electronic Arts Vancouver, British Columbia, Canada · Electronic Arts Guildford, Surrey, United Kingdom

  15. To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

    Mar 23, 2026OFM Riaz Rahman Aranya, Kevin DesaiSycophancy in VLMsMedical VLMs

    Department of Computer Science, The University of Texas at San Antonio

  16. SteelDefectX: A Multi-Form Vision-Language Dataset and Benchmark for Steel Surface Defect Analysis

    Mar 23, 2026Shuxian Zhao, Jie Gui, Baosheng Yu +1VLM EvaluationVision-Language Grounding

    Southeast University · Purple Mountain Laboratories · Nanyang Technological University

  17. MIRAGE: The Illusion of Visual Understanding

    Mar 23, 2026Mohammad Asadi, Jack W. O'Sullivan, Fang Cao +5VLM EvaluationFaithfulness in VLMs

    Department of Electrical Engineering, Stanford University, CA, USA · Division of Cardiology, Department of Medicine, Stanford University, CA, USA · Department of Biomedical Data Science, Stanford University, CA, USA +3

  18. When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations

    Mar 23, 2026Harsh Nishant Lalai, Raj Sanjay Shah, Hanspeter Pfister +2VLM EvaluationChart Understanding

    Birla Institute of Technology and Science, Pilani · Georgia Institute of Technology · Harvard University

  19. Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs

    Mar 23, 2026Efthymios Tsaprazlis, Tiantian Feng, Anil Ramakrishna +3VLM EvaluationPrivacy Auditing

    University of Southern California, Los Angeles CA 90089, USA · Meta Superintelligence Labs · Amazon AGI

  20. Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects

    Mar 22, 2026Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Shubhashis Roy Dipta +6VLM EvaluationMultilingual VLM Evaluation

  21. Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models

    Mar 22, 2026Binesh Sadanandan, Vahid BehzadanVLM EvaluationMedical VLMs

    SAIL Lab, University of New Haven, West Haven, CT, USA

  22. When Negation Is a Geometry Problem in Vision-Language Models

    Mar 20, 2026Fawaz Sammani, Tzoulio Chamiti, Paul Gavrikov +1VLM ReasoningVLM Evaluation

    ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium · Independent Researcher

  23. CREG: Compass Relational Evidence Graph for Characterizing Directional Structure in VLM Spatial-Reasoning Attribution

    Mar 20, 2026Kaizhen Tan, Yang Feng, Heqing DuVLM EvaluationGradient-Based Attribution

    Carnegie Mellon University Pittsburgh, PA, USA · Columbia University New York, NY, USA

  24. From Masks to Pixels and Meaning: A New Taxonomy, Benchmark, and Metrics for VLM Image Tampering

    Mar 20, 2026Xinyi Shang, Yi Tang, Jiacheng Cui +9Image Forgery DetectionImage Forgery Localization

    Mohamed bin Zayed University of Artificial Intelligence · University College London

  25. FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs

    Mar 20, 2026Zhihan Yin, Jianxin Liang, Yueqian Wang +3VLM HallucinationVLM Evaluation

    Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence

  26. Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders

    Mar 19, 2026Shang-Jui Ray Kuo, Paola Cascante-BonillaVision-Language ModelsState Space Models

    Stony Brook University

  27. SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues

    Mar 19, 2026Carlos Hinojosa, Clemens Grange, Bernard GhanemCultural Safety Evaluation of VLMsVision-Language Models

    King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Technical University of Munich (TUM), Germany

  28. MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model

    Mar 19, 2026Youngwan Lee, Soojin Jang, Yoorhim Cho +3Spatial Reasoning BenchmarksSpatial Reasoning

    Electronics and Telecommunications Research Institute, South Korea · Korea Advanced Institute of Science and Technology, South Korea · Sungkyunkwan University, South Korea +1

  29. Do Vision Language Models Understand Human Engagement in Games?

    Mar 19, 2026Ziyi Wang, Qizan Guo, Rishitosh Singh +1VLM EvaluationVideo-Language Models

    Arizona State University

  30. Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models

    Mar 17, 2026Weijie Qiu, Dai Guan, Junxin Wang +6Multimodal Reward ModelingVLM Evaluation

    Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences