VLM Evaluation

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

CardsList

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

  1. NTIRE 2026 The 3rd Restore Any Image Model (RAIM) Challenge: Professional Image Quality Assessment (Track 1)

    Apr 14, 2026Guanyi Qin, Jie Liang, Bingbing Zhang +50Image Quality AssessmentVLM Evaluation

  2. Beyond Perception Errors: Semantic Fixation in Large Vision-Language Models

    Apr 13, 2026Md Tanvirul AlamVLM ReasoningVLM Evaluation

    Rochester Institute of Technology, Rochester, NY, USA

  3. Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs

    Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM EvaluationConfidence Estimation for VLMs

    Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA

  4. Benchmarking Deflection and Hallucination in Large Vision-Language Models

    Apr 13, 2026Nicholas Moratelli, Christopher Davis, Leonardo F. R. Ribeiro +2VLM EvaluationVision-Language Models

    University of Modena and Reggio Emilia · Amazon AGI · University of Cambridge

  5. GazeVaLM: A Multi-Observer Eye-Tracking Benchmark for Evaluating Clinical Realism in AI-Generated X-Rays

    Apr 13, 2026David Wong, Zeynep Isik, Bin Wang +22Medical Image BenchmarksVLM Evaluation

    Northwestern University, Evanston, IL, United States · Northwestern University, Evanston, IL, USA · Université Sorbonne Paris Nord, Paris, France +5

  6. CArtBench: Evaluating Vision-Language Models on Chinese Art Understanding, Interpretation, and Authenticity

    Apr 13, 2026Xuefeng Wei, Zhixuan Wang, Xuan Zhou +5VLM EvaluationVLM Reasoning

    Nara Institute of Science and Technology · Liaoning Normal University

  7. Sign Language Recognition in the Age of LLMs

    Apr 13, 2026Vaclav Javorek, Jakub Honzik, Ivan Gruber +2Sign Language RecognitionVLM Evaluation

    University of West Bohemia, Czech Republic · Eindhoven University of Technology, The Netherlands

  8. Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding

    Apr 13, 2026Shivam Sharma, Sankalp Nagaonkar, Ashish Choithani +1VLM ReasoningVLM Evaluation

    VideoDB∗

  9. RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Models

    Apr 13, 2026Yakun Yu, Ashley Wiens, Adrián Barahona-Ríos +4VLM EvaluationReference-Based Video Anomaly Detection

    University of Alberta · Sony Interactive Entertainment

  10. TraversalBench: Challenging Paths to Follow for Vision Language Models

    Apr 13, 2026Clara Petrova, Zhuo Chen, Marin SoljačićVLM EvaluationVisual Grounding

    Massachusetts Institute of Technology, Department of Physics · NSF AI Institute for Artificial Intelligence and Fundamental Interactions · Massachusetts Institute of Technology, Institute for Data, Systems, and Society

  11. Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models

    Apr 13, 2026Sameera Horawalavithana, Lauren Phillips, Ian Stewart +2Vision-Language ModelsVLM Reasoning

    Pacific Northwest National Laboratory, Richland, WA 99352, USA

  12. BareBones: Benchmarking Zero-Shot Geometric Comprehension in VLMs

    Apr 12, 2026Aaditya Baranwal, Vishal Yadav, Abhishek RajoraVLM EvaluationVLM Reasoning

    (Independent Researcher) · University of Calgary

  13. DiningBench: A Hierarchical Multi-view Benchmark for Perception and Reasoning in the Dietary Domain

    Apr 12, 2026Song Jin, Juntian Zhang, Xun Zhang +6VLM EvaluationFine-Grained Visual Perception

    Gaoling School of Artificial Intelligence, Renmin University of China · Meituan · Wuhan University

  14. Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts

    Apr 11, 2026Ruijia Li, Mingzi Zhang, Zengyi Yu +2Cultural Bias in VLMsVLM Evaluation

    School of Computer Science and Technology, East China Normal University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · Faculty of Education, East China Normal University

  15. Counting to Four is still a Chore for VLMs

    Apr 11, 2026Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van VoAttention Intervention in VLMsVision-Language Models

    MBZUAI

  16. Cross-Cultural Value Attribution in Large Vision-Language Models

    Apr 10, 2026Phillip Howard, Xin Su, Kathleen C. FraserVLM EvaluationCounterfactual Evaluation of VLMs

    Thoughtworks · University of Ottawa

  17. From Scientific Crop Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping

    Apr 10, 2026Yu Wu, Guangzeng Han, Ibra Niang Niang +6Plant PhenotypingVLM Evaluation

    Computer Science, University of Memphis, Memphis, 38111, TN, United States · Fisher Delta Research Extension and Education Center, University of Missouri, Portageville, 63873, MO, United States · University of Arkansas Division of Agriculture, Batesville, 72501, AR, United States +1

  18. Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models

    Apr 10, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMsVLM Evaluation

    SAIL Lab, University of New Haven, West Haven, CT, USA

  19. SenBen: Sensitive Scene Graphs for Explainable Content Moderation

    Apr 9, 2026Fatih Cagatay Akyon, Alptekin TemizelContent ModerationMultimodal Content Moderation

    Graduate School of Informatics, METU, Turkiye · Ultralytics, Inc., United States

  20. Can Vision Language Models Judge Action Quality? An Empirical Evaluation

    Apr 9, 2026Miguel Monte e Freitas, Rui Henriques, Ricardo Rei +1Action Quality AssessmentVision-Language Models

    Sword Health · Instituto Superior T´ecnico, Universidade de Lisboa · INESC-ID

  21. Vision-Language Foundation Models for Comprehensive Automated Pavement Condition Assessment

    Apr 9, 2026Blessing Agyei Kyem, Joshua Kofi Asamoah, Anthony Dontoh +1Multimodal Instruction TuningVLM Evaluation

    Department of Civil, Construction and Environmental Engineering, North Dakota State University, Fargo, 58102, ND, USA · Department of Civil, Construction and Environmental Engineering, University of Memphis, Memphis, 38152, TN, USA

  22. MARINER: A 3E-Driven Benchmark for Fine-Grained Perception and Complex Reasoning in Open-Water Environments

    Apr 9, 2026Xingming Liao, Ning Chen, Muying Shu +5VLM EvaluationVisual Question Answering

    Guangdong University of Technology, Guangzhou, China

  23. VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models

    Apr 8, 2026Pavan Kumar Anasosalu Vasu, Cem Koc, Fartash Faghri +6VLM EvaluationVideo-Language Model Evaluation

    Apple

  24. Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

    Apr 8, 2026Yuechen Jiang, Enze Zhang, Md Mohsinul Kabir +4Cultural Knowledge in Language ModelsVLM Evaluation

    University of Manchester Manchester, UK · School of Artificial Intelligence, Wuhan University Wuhan, Hubei, China · Getty Conservation Institute Los Angeles, CA, USA

  25. How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study

    Apr 8, 2026Roberto Brusnicki, Mattia Piccinini, Johannes BetzVisual Evidence Sensitivity in VLMsTemporal Reasoning in VLMs

    Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich, Munich, Germany

  26. ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs

    Apr 7, 2026Zhipin Wang, Christoph Leiter, Christian Frey +3Cultural Knowledge in Language ModelsVLM Evaluation

    University of Technology Nuremberg (UTN), Germany

  27. SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future

    Apr 7, 2026Timothy B. Higgins, Antonios Mamalakis, Chirag AgarwalVLM EvaluationForecasting Benchmarks

    Department of Environmental Sciences, University of Virginia, Charlottesville, 22903, Virginia, United States. · School of Data Science, University of Virginia, Charlottesville, 22903, Virginia, United States.

  28. "I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

    Apr 7, 2026Naen Xu, Jiayi Sheng, Changjiang Li +7VLM EvaluationMultimodal Understanding

    Zhejiang University · Beihang University · Palo Alto Networks +4

  29. AICA-Bench: Holistically Examining the Capabilities of VLMs in Affective Image Content Analysis

    Apr 7, 2026Dong She, Xianrong Yao, Liqun Chen +3VLM EvaluationVLM Calibration

    School of Future Technology,South China University of Technology,Guangzhou

  30. DetailVerifyBench: A Benchmark for Dense Hallucination Localization in Long Image Captions

    Apr 7, 2026Xinran Wang, Yuxuan Zhang, Xiao Zhang +7VLM HallucinationVLM Evaluation

    Beijing University of Posts and Telecommunications, Beijing, China