VLM Evaluation

VLM: Vision-Language Model

Ask about this topic

Overview

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

CardsList

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

  1. It Takes Two to Probe, Thousands to Retrieve: Putting Vision--Language Probes to the Retrieval Test

    Oct 29, 2026Ivana Beňová, Michal Gregor, Marián ŠimkoVLM EvaluationVision-Language Retrieval

    Kempelen Institute of Intelligentz Technologies, Bratislava, Slovakia

  2. Where Do Vision Language Models Fail at 3D Reasoning?

    Oct 29, 2026Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo +11Reasoning for VLMsVLM Evaluation

  3. Evaluating Cultural Knowledge Editing in Vision-Language Models

    Oct 29, 2026Malvina Nikandrou, Georgios Pantazopoulos, Ioannis Konstas +1Vision-Language ModelsKnowledge Editing

    University of Edinburgh · Heriot-Watt University

  4. Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe

    Oct 29, 2026Jiawei ChenVLM EvaluationVisual Evidence Sensitivity in VLMs

    Netflix

  5. OneScene-Bench: Do Vision-Language Models Integrate Evidence Across Views?

    Oct 28, 2026Swathi Suhas, Anurag Das, Bernt Schiele +1VLM EvaluationReasoning for VLMs

    Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany

  6. inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild

    Oct 28, 2026Sidra HanifVLM EvaluationVision-Language Grounding

    San Diego State University

  7. V-CAPE: Visual Consistency Assessment for Product Image Editing

    Oct 28, 2026Iaroslav Melekhov, Roberto Alcover Couso, Francesco Taioli +4Image Editing EvaluationImage Editing

    Amazon · Faculty

  8. Automatic Iconographic Annotation of Medieval Illuminations: Comparing Generative Vision-Language Models and Supervised Discriminative Classifiers

    Oct 28, 2026Vincent Jolivet, Lucas Terriel, Chahan Vidal-Gor{è}neImage ClassificationVLM Evaluation

    ENC, CJM · ENC · CJM, ENC

  9. The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks

    Oct 28, 2026Shuo Wen, Beixi Du, Edwin Meriaux +5Visual ReasoningVideo Reasoning

    McGill University · Mila – Quebec AI Institute · Cornell University +1

  10. Learning to Read Braille: Benchmarking and Improving Braille Recognition Capabilities of VLMs

    Oct 26, 2026Kartik Jangra, Atharva Gawane, Siddharth Pulipaka +2VLM EvaluationSynthetic Data Generation

    Nilekani Centre at AI4Bharat · Indian Institute of Technology, Madras

  11. Improving VLMs on Scientific Chart Analysis via Autonomous Evolution

    Oct 26, 2026Abhinav Sudhakar Dubey, Vidushi Goel, Kiran Rana +2Chart QAVLM Evaluation

    University of California, Santa Cruz · Airbnb Inc.

  12. Good Rankings, Poor Choices? Rethinking Language-Only Vision-Language Model Selection

    Oct 26, 2026Hyuk Kwon, Jaebyeong Jeon, Hyeonseo Jang +1Model SelectionVLM Evaluation

    Yonsei University

  13. Does the Outfit Change the Price? Context Halo in Vision-Language Model Valuation of an Identical Low-Cost Necklace

    Oct 25, 2026Brianne LeeFraming Effects in Language ModelsVision-Language Models

    Independent Researcher

  14. VTC-AUDIT: An Interventional Reliability Audit for Visual-Token Compression under Paired Image Shifts

    Oct 25, 2026Isaac, LiVisual Token CompressionVLM Evaluation

    Rucheng · University of Pittsburgh

  15. Execution Logs Sway VLM Judges of Robot Progress

    Oct 25, 2026Weiyan Zhao, Sunny YangVLM RobustnessVision-Language Model-as-a-Judge

    NLPQuant AI Inc. · University of Illinois Urbana-Champaign

  16. What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

    Oct 25, 2026Chaitat Utintu, Ahmed Bourouis, Arkaprabha Basu +1Vision-Language ModelsVLM Interpretability

    SketchX, Surrey Institute for People-Centred Artificial Intelligence (PAI), University of Surrey · UKRI AI: CDT in AI for Digital Media Inclusion

  17. All the Facts, Still the Wrong Answer: Integration Gaps in Vision-Language Models

    Oct 24, 2026Yuyang Dai, Yuxia Wang, Zhuohan XieVLM EvaluationCompositional Reasoning in VLMs

    Florida State University · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI

  18. What Remains OOD? OODConceptBench for Open-World Concept Recognition in Vision-Language Models

    Oct 24, 2026Zhiran Luo, Zhiyong Yang, Zitai Wang +3VLM EvaluationOOD Generalization

    School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +2

  19. Image Clustering with Vision-Language Model Embeddings

    Oct 23, 2026João P. Matos-Carvalho, Nuno FachadaImage ClusteringVLM Evaluation

    LASIGE, Faculdade de Ciências, Universidade de Lisboa, Lisboa, 1749-016, Portugal · COPELABS, Lusófona University, Lisboa, 1700-921, Portugal · INESC INOV, Lisboa, 1000-029, Portugal

  20. Of Chains and Gains: What Reinforcement Learning changes on Reasoning-based Image Quality Assessment

    Oct 23, 2026Yixuan Li, Yuxuan Jiang, Hanwei Zhu +4Image Quality AssessmentReinforcement Learning with Verifiable Rewards

    University of Bristol, UK · Jiangxi University of Finance and Economics, China · China University of Mining and Technology, China +1

  21. PCFBench: How Far Can Large Vision-Language Models Go in Physics-aware Photonic Inverse Design?

    Oct 23, 2026Shengchao Chen, Ting Shu, Sufen RenInverse DesignVLM Evaluation

    Australian AI Institute, University of Technology Sydney · School of Artificial Intelligence, Shenzhen University · School of Information and Communication Engineering, Hainan University

  22. Search-Based Testing of Modality Sensitivity in Vision Language Models

    Oct 23, 2026Oliver Weißl, Eduardo Breitembach, Andrea StoccoVLM EvaluationVLM Robustness

    Technical University of Munich, Garching near Munich, Germany · fortiss GmbH, Munich, Germany

  23. SAGE: Sense-Aligned Multimodal Grounding Evaluation with Gloss-Augmented Reasoning

    Oct 22, 2026Sunjae Kwon, Suhaas Kiran DG, Zonghai Yao +3VLM EvaluationVision-Language Grounding

    UMass Amherst · UMass Lowell · UMass Chan Medical School +1

  24. CellTrack-Bench: Microscopic Cell Tracking Benchmark for Vision-Language Models

    Oct 22, 2026Ilan Theodoro, Jordão Bragantini, Loïc A. RoyerVLM EvaluationMicroscopy Image Analysis

    Biohub San Francisco, CA

  25. AgentVQA: A Unified Benchmark for Agentic Visual Understanding

    Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM EvaluationAgent Benchmarks

    University of Maryland, College Park · Princeton University

  26. IRON-Meme: A Benchmark for Structured Understanding of Identities and Roles in Online Meme Narratives

    Oct 22, 2026Gabriella Bollici, Stefano De Giorgis, Wouter Van Atteveldt +1Political Discourse AnalysisVLM Evaluation

    Department of Computer Science, Faculty of Science, Vrije Universiteit Amsterdam · Department of Communication Science, Faculty of Social Science, Vrije Universiteit Amsterdam

  27. VLTest: Finding Joint Image-Text Failures in Vision-Language Models

    Oct 22, 2026Bin Duan, Jintao Lin, Guowei YangVLM EvaluationMutation Testing

    The University of Queensland, Brisbane, Australia

  28. Where Did the Light Come From? Diagnosing Cast-Shadow Reasoning in VLMs and Borrowing the Missing Shadow Prior from Image Generation Models

    Oct 22, 2026Ryugo Morita, Shunsuke Kitada, Stanislav Frolov +5VLM ReasoningPhysical Reasoning

    DFKI, Germany · RPTU Kaiserslautern-Landau, Germany · Hosei University, Japan

  29. Evaluating Visual Language Models on Visual Grounding for Visually Impaired People

    Oct 21, 2026Fairooz Azim, Lydia Achour, Claudia Borg +1Vision-Language GroundingVLM Evaluation

    Orange Research Lannion, France · University of Malta, Msida, Malta

  30. A Unified Evaluation Framework of Physical Adversarial Robustness in the Edge VLM Era

    Oct 21, 2026Eric Liang, Alan Luo, Kaiwen YuanVLM RobustnessAdversarial Attacks on VLMs

    Saratoga High School · Google · University of Illinois Urbana-Champaign