VLM Evaluation

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

CardsList

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

  1. Shape vs. Context: Examining Human--AI Gaps in Ambiguous Japanese Character Recognition

    Feb 27, 2026Daichi HaraguchiVLM EvaluationVision-Language Alignment

    CyberAgent, Tokyo, Japan

  2. SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses

    Feb 26, 2026Zhuohang Jiang, Xu Yuan, Haohao Qu +4Visual Question AnsweringKnowledge-Based VQA

    The Hong Kong Polytechnic University

  3. PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models

    Feb 24, 2026Binesh Sadanandan, Vahid BehzadanMedical VLMsVLM Evaluation

    SAIL Lab, University of New Haven, West Haven, CT, USA.

  4. VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation

    Feb 24, 2026Seongheon Park, Changdae Oh, Hyeong Kyu Choi +2Confidence Estimation for VLMsVLM Evaluation

    University of Wisconsin-Madison · Nanyang Technological University

  5. Diagnosing Causal Reasoning in Vision-Language Models via Structured Relevance Graphs

    Feb 24, 2026Dhita Putri Pratama, Soyeon Caren Han, Yihao DingVLM ReasoningVLM Evaluation

    University of Melbourne Melbourne, Australia · The University of Western Australia Perth, Australia

  6. How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective

    Feb 24, 2026Bo Peng, Pi Bu, Keyu Pan +7Embodied Agent BenchmarksEmbodied AI

    University of Science and Technology of China · Alibaba Group · Independent Researcher +1

  7. How Do Inpainting Artifacts Propagate to Language?

    Feb 24, 2026Pratham Yashwante, Davit Abrahamyan, Shresth Grover +1VLM RobustnessVLM Evaluation

    UC San Diego

  8. Do Large Language Models Understand Data Visualization Principles?

    Feb 23, 2026Martin Sinnona, Valentin Bonas, Viviana Siless +1LLM EvaluationVLM Evaluation

    Universidad Torcuato Di Tella, Buenos Aires, Argentina · Universidad de Buenos Aires, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina

  9. VectorGym: A Multi-Task Benchmark for SVG Code Generation, Sketching and Editing

    Feb 22, 2026Joan Rodriguez, Haotian Zhang, Abhay Puri +14VLM EvaluationCode Generation

    Mila, Quebec AI Institute · ÉTS Montréal · QuiverAI +11

  10. MiSCHiEF: A Benchmark in Minimal-Pairs of Safety and Culture for Holistic Evaluation of Fine-Grained Image-Caption Alignment

    Feb 21, 2026Sagarika Banerjee, Tangatar Madi, Advait Swaminathan +4Cultural Safety Evaluation of VLMsVLM Evaluation

    Algoverse AI Research · PocketFM

  11. OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models

    Feb 20, 2026Ling Lin, Yang Bai, Heng Su +5VLM EvaluationVLM Robustness

    School of Artificial Intelligence and Data Science, University of Science and Technology of China, Hefei 230000, China · Suzhou Institute for Advanced Research, USTC, Suzhou 215123, China · Institute of High Performance Computing (IHPC), A*STAR, Singapore +1

  12. Understanding the Fine-Grained Knowledge Capabilities of Vision-Language Models

    Feb 19, 2026Dhruba Ghosh, Yuhui Zhang, Ludwig SchmidtFine-Grained Visual PerceptionVLM Evaluation

    Stanford University

  13. Sketch2Feedback: Grammar-in-the-Loop Framework for Rubric-Aligned Feedback on Student STEM Diagrams

    Feb 19, 2026Aayam BansalVLM Hallucination MitigationVLM Evaluation

    IEEE

  14. Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families

    Feb 17, 2026Yuval LeventalVision-Language ModelsVLM Reasoning

    Rochester Institute of Technology

  15. ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery

    Feb 16, 2026Ayush Shrivastava, Kirtan Gangani, Laksh Jain +2VLM EvaluationThermal Imaging

    Indian Institute of Technology, Gandhinagar, India · Carnegie Mellon University, Pittsburgh, USA

  16. Error Patterns in Historical OCR: A Comparative Analysis of TrOCR and a Vision-Language Model

    Feb 16, 2026Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen +1Optical Character RecognitionVLM Evaluation

    University of Helsinki, Department of Computer Science, Finland · University of Helsinki, Department of Digital Humanities, Finland

  17. An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation

    Feb 13, 2026Giang Son Nguyen, Zi Pong Lim, Sarthak Ketanbhai Modi +2VLM EvaluationCode Generation

    Nanyang Technological University · VinUniversity · AUMOVIO Singapore

  18. VideoSTF: Stress-Testing Output Repetition in Video Large Language Models

    Feb 11, 2026Yuxin Cao, Wei Song, Shangzhi Xu +2VLM EvaluationVLM Robustness

    National University of Singapore, Singapore · University of New South Wales, Australia · CSIRO’s Data61, Australia

  19. Egocentric Bias in Vision-Language Models

    Feb 10, 2026Maijunxian Wang, Yijiang Li, Bingyang Wang +6VLM EvaluationVisual Spatial Reasoning

    Cognitive Science Program, University of California, Berkeley · Department of Electrical and Computer Engineering, University of California San Diego · School of Computer Science, Georgia Institute of Technology & Emory University +6

  20. Out of the box age estimation through facial imagery: A Comprehensive Benchmark of Vision-Language Models vs. out-of-the-box Traditional Architectures

    Feb 8, 2026Simiao Ren, Xingyu Shen, Ankit Raj +8Large Vision-Language ModelsVLM Evaluation

    Manlin · Reality Inc. · New York University +2

  21. Same Answer, Different Representations: Hidden instability in VLMs

    Feb 6, 2026Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena +6VLM EvaluationVLM Robustness

    Sapienza University of Rome · University of Edinburgh · University of Oxford +5

  22. VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

    Feb 5, 2026Tina Khezresmaeilzadeh, Jike Zhong, Konstantinos PsounisVisual ReasoningVLM Reasoning

    University of Southern California, Los Angeles, USA

  23. Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raters

    Feb 1, 2026Lana Do, Gio Jung, Juvenal Francisco Barajas +5VLM EvaluationHuman-in-the-Loop Evaluation

    Northeastern University · United States · San Francisco State University +2

  24. Model Specific Task Similarity for Vision Language Model Selection via Layer Conductance

    Feb 1, 2026Wei Yang, Hong Xie, Tao Tan +3VLM EvaluationVision-Language Models

    School of AI and Data Science USTC

  25. TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs

    Jan 30, 2026Baiqi Li, Kangyi Zhao, Ce Zhang +3VLM EvaluationVideo Understanding

    University of North Carolina at Chapel Hill · University of Pittsburgh · Carnegie Mellon University

  26. A Comparative Evaluation of Large Vision-Language Models for 2D Object Detection under SOTIF Conditions

    Jan 30, 2026Ji Zhou, Yilin Ding, Yongqi Zhao +4VLM EvaluationLarge Vision-Language Models

    Institute of Automotive Engineering, Graz University of Technology, 8010 Graz, Austria · Independent Researcher, 200000 Shanghai, China · School of Intelligent Connected Vehicle, Hubei University of Automotive Technology, 442002 Shiyan, China +1

  27. DisasterInsight: A Building-Centric Benchmark for Evaluating Vision--Language Models in Disaster Response

    Jan 26, 2026Sara Tehrani, Yonghao Xu, Leif Haglund +3Building Damage AssessmentRemote Sensing Image Understanding

    Computer Vision and Learning System, Linköping University, Sweden · Vantor, Linköping, Sweden

  28. Semantic Misalignment in Vision-Language Models under Perceptual Degradation

    Jan 13, 2026Guo Cheng, Huang LiVLM EvaluationVLM Robustness

  29. BabyVision: Visual Reasoning Beyond Language

    Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM EvaluationVisual Reasoning

    1UniPat AI · 6Peking University · 3Alibaba Group +8

  30. SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models

    Jan 8, 2026Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi +1Video RetrievalVLM Evaluation

    Universitat de Barcelona and Computer Vision Center Gran Via de les Corts Catalanes 585, Barcelona Campus UAB, Edifici O, Cerdanyola del Vall`es · Milestone Systems A/S Banemarksvej 50, Brøndby · Aalborg Universitet Fredrik Bajers Vej 7K, Aalborg Øst