VLM Evaluation

VLM: Vision-Language Model

Overview

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

CardsList

1,060 papers77 papers in the last four weeks, down 40% on the four weeks before. 0.6% of all new papers.

  1. What Does Anti-Hallucination Training Buy? Saying Less, Cheap Pruning, and the Rest

    Nov 19, 2026Lingkai Bu, Qian GaoObject Hallucination in VLMsVLM Hallucination Mitigation

  2. Characterizing Age-Conditioned Affective Reasoning in Vision-Language Models

    Nov 19, 2026Shahriyar Zaman Ridoy, Azmine Toushik Wasi, Lin Gu +1Demographic Bias in VLMsVLM Evaluation

  3. Pixel2Perspective: A Benchmark for Multi-Modal and Multi-View Generalization in Vision-Language Models

    Nov 18, 2026Aditya Gandhamal, Aniruddh Sikdar, Arya Kishor +1VLM RobustnessVLM Evaluation

  4. Diagnostic Accuracy and Navigation Behavior of Vision-Language Model Agents Reading Emergency CT and MRI Examinations

    Nov 17, 2026Jonathan Lichtle, Aymen Meddeb, Antonin Birot +10Radiology VLMsVLM Evaluation

  5. Bits, Not Boxes: Label-Free Layer Selection for Visual Localization in VLMs

    Nov 17, 2026Haisheng Fu, Gexin Huang, Haikai Zhao +4Vision-Language GroundingVLM Interpretability

  6. Is model self-assessment enough for diverse LVLM ensembles?

    Nov 17, 2026Michael Smith, Frank P. FerrieVLM EnsemblingVLM Evaluation

  7. Know When to Zoom: How Small Is Too Small for Vision-Language Models?

    Nov 17, 2026Siddharth Vohra, Min XuVision-Language ModelsVisual Evidence Sensitivity in VLMs

  8. Lost in Read-out: Locating the Image Quality Gap in Vision-Language Models with a Closed-Form Edit

    Nov 17, 2026Zhuangzi Li, Shilv Cai, Jian Jin +3VLM EvaluationImage Quality Assessment

  9. M3^3-Test: Coverage Gaps in Joint-Perturbation Testing of Visual Question Answering

    Nov 17, 2026Enhui Wan, Xiaowan WangVLM EvaluationVLM Robustness

  10. Portrait Fitness as a Pretext: Probing Implicit Demographic Bias in VLMs

    Nov 16, 2026Alexandre Fournier-Montgieux, Akram Elbouanani, Aboubacar Tuo +3Demographic Bias in VLMsVLM Evaluation

  11. AnachroBench: Evaluating Historical Consistency in Large Vision-Language Models

    Nov 16, 2026Keunhyeung Park, Byeonggeuk Lim, Kahyeon Kim +5VLM EvaluationTemporal Reasoning in VLMs

  12. Seeing Isn't Resolving: Visual and Linguistic Disambiguation with Conversational Grounding

    Nov 16, 2026Edoardo Cecchinato, Shun Inadumi, Angel García Contreras +2Vision-Language GroundingVLM Evaluation

  13. CivilBench: Benchmarking Vision-Language Models for Multimodal Information Integration in Civil Engineering

    Nov 16, 2026Dan Yuan, Yue Wen, Zhili He +3VLM EvaluationNumerical Reasoning in Language Models

  14. EEGProbe: Do Vision-Language Models Really Understand EEG? A Benchmark for Signal-Grounded EEG Signals Evaluation

    Nov 16, 2026Haozhe Zhu, Dan Yuan, Chenghao Gu +3VLM EvaluationMultimodal Model Evaluation

  15. From Images to Grounded Edits: A Real-World Benchmark for Chinese Grammatical Error Correction in Handwritten Essays

    Nov 13, 2026Weiru Zhang, Yuzhong Hong, Junwei Bao +2Grammatical Error CorrectionVision-Language Grounding

  16. In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment

    Nov 13, 2026Monica Riedler, Jasmin Orth, Raffaella Bernardi +1VLM Evaluation

  17. OmniScore-Bench: Evaluating Vision-Language Models on Musical Score Reading

    Nov 13, 2026Congren Dai, Zhancheng Guo, Danni Zhao +3VLM EvaluationOptical Music Recognition

  18. Validation-Guided Prompt Selection for Differential Morphing Attack Detection

    Nov 12, 2026Raghavendra Ramachandra, Hailin LiFace Morphing Attack DetectionPrompt Optimization

  19. How Far Can Frozen Vision-Language Models Go on Event Data? A Training-Free Study on EventBench

    Nov 11, 2026Md Kamrujjaman Mobin, Md Salman Shamil, Md Adnan ArefeenEvent-Based VisionVLM Evaluation

  20. Supervision-Aware Evaluation of MedGemma for Reference-Assisted Radiotherapy Contour Ranking

    Nov 11, 2026Jaden Moon, Caiwen Jiang, Songlin Zhao +1VLM EvaluationMedical VLMs

  21. Negation Through Affirmation: Leveraging Positive Evidence in Vision-Language Models

    Nov 11, 2026Aashish Pokhrel, Tejas Anvekar, Shivanand Venkanna SheshappanavarVLM ReasoningVision-Language Retrieval

  22. Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models

    Nov 11, 2026Shreya Ajay Kale, Angad Ripudaman Singh BajwaMedical VLMsVLM Evaluation

  23. RSLiveBench-UAV: A Continually Evolving Vision-Language Benchmark for Aerial Spatial Intelligence

    Nov 11, 2026Yue Zhou, Jiawei Liu, Jue Chen +9Continual Learning BenchmarkingVLM Evaluation

  24. ZoomGain: Mining Global-to-Zoom Perception Gaps for Fine-Grained Visual Supervision

    Nov 10, 2026Haozhan Shen, Liangbo He, Jun Lan +4Fine-Grained Visual PerceptionVLM Evaluation