VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

71 papers in the last four weeks, up 69% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 649

All topics

Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA
Jul 28, 2026Carlos Celemin, Benedict Wilkins, Adrián Barahona-Ríos +2VLM Evaluation VLM Robustness
Sony Interactive Entertainment, London, United Kingdom
Forensic Reproducibility Audit of a Radiology Vision-Language Model Benchmark: From Intended Protocol to Released Artifact
Jul 28, 2026Mateusz KozłowskiVLM Evaluation Radiology VLMs
Independent Researcher, Kraków, Poland
ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams
Jul 27, 2026Ali Ansari, Yasmin Mohammadi, Farnoush Nili +3VLM Evaluation Document Image Understanding
Temple University · Department of Computer and Information Sciences · Philadelphia, PA, USA
When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents
Jul 27, 2026Marina Gardella, Camilo Mariño, Diego Belzarena +3VLM Evaluation VLM Hallucination
CB · Université Paris-Saclay, ENS Paris-Saclay, CNRS, Centre Borelli, France · UDELAR, CB +4
Cheap Probes Predict Expensive Training in 3D-CT Vision--Language Models
Jul 24, 2026Renjie LiangVLM Evaluation Medical VLMs
University of Florida, Gainesville, FL, USA
Be Consistent! Enhancing Robust Visual Reasoning in LVLMs with Consistency Constraints
Jul 23, 2026Liqiang Jing, Xiong Zhou, Siddharth Varia +3VLM Evaluation Visual Reasoning
University of Texas at Dallas · Amazon Web Services
When Are Reasoning-Based Guardrails Not Efficient? ResponseGuard: A Fast Vision-Language Guard for Real-Time Moderation
Jul 23, 2026Dongbin NaVLM Evaluation Efficient VLM Inference
POSTECH
CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA
Jul 23, 2026Hanseok Oh, Parishad BehnamGhader, Benno Krojer +4VLM Evaluation Multimodal RAG
New York University, New York NY, USA · Mila - Quebec AI Institute, Montreal, Canada · McGill University, Montreal, Canada +3
V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding-Refusal Coupling Failure
Jul 23, 2026Zhetong Zhang, Honghao Fu, Miao Xu +2VLM Evaluation VLM Robustness
University of Queensland · University of California, Merced
Do Pathology Vision-Language Models Truly See Pathology?
Jul 23, 2026Chengyang Zhang, Wenchuan Zhang, Bo Li +10VLM Evaluation Computational Pathology
College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer Science, School of Computing, National University of Singapore +2
MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement
Jul 23, 2026Daiqing Wu, Dongbao Yang, Jiashu Yao +4VLM Evaluation Multimodal Emotion Recognition
Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing 100084, China · Nankai University, Tianjin 300071, China · Beijing Institute of Technology, Beijing 100081, China +1
ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?
Jul 23, 2026Han Li, Si Liu, Zehao Huang +6VLM Evaluation Spatial Reasoning Benchmarks
School of Artificial Intelligence, Beihang University, Beijing 100191, China · Zhongguancun Academy, Beijing 100094, China · School of Engineering, Westlake University, Hangzhou 310030, China
ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
Jul 22, 2026Karan Goyal, Afreen Hossain, Debojyoti Das +1VLM Evaluation Vision-Language Models
IIIT Delhi, India · Dr. Ambedkar Institute of Technology, India · Heritage Institute of Technology, India +1
PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image
Jul 21, 2026Dankai Liao, Tianyi Zhang, Yufeng Wu +6VLM Evaluation Medical VLMs
Department of Biomedical Engineering, National University of Singapore, Singapore 117417, Singapore
Bounding Boxes to Improve Small Language Model Performance on Vision-Based Grading Tasks
Jul 21, 2026Lachlan McGinnessVLM Evaluation Educational Assessment
Australian National University, Acton, ACT, Australia
MissingBench-Verified: Probing Vision-Language Models' Inability to Detect Missing Object Parts
Jul 21, 2026Wenqi Marshall Guo, Qingyun Qian, Shiyu Zhou +2VLM Evaluation Object Hallucination in VLMs
Department of CMPS, University of British Columbia, Canada · Weathon Software, Canada · Department of Computer Science, University of British Columbia, Canada
Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs
Jul 20, 2026Binesh Sadanandan, Vahid BehzadanVLM Evaluation VLM Interpretability
SAIL Lab, University of New Haven, West Haven, CT, USA
The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric
Jul 20, 2026Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann +4VLM Evaluation T2I Generation Evaluation
1Carnegie Mellon University · 2Adobe Research · 3UC Berkeley
Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQA
Jul 20, 2026Yuhao Liu, Cheng Zhao, Guanghui YueVLM Evaluation Endoscopy
School of Biomedical Engineering, Shenzhen University, Shenzhen, China
Hazard or Anomaly? Evaluating VLMs for Understanding Dangers and Discrepancies
Jul 18, 2026Murali Indukuri, Mohammad Eskandari, Sree Nitya Kollu +2VLM Evaluation
Interactive Robotics and Language Lab, University of Maryland Baltimore County, Computer Science and Electrical Engineering Department, Baltimore, MD, USA. · DEVCOM Army Research Laboratory, Adelphi, MD, USA.
An Exam for Active Observers
Jul 17, 2026Jiarui Zhang, Muzi Tao, Shangshang Wang +3VLM Evaluation Multimodal Large Language Models
University of Southern California
Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling
Jul 17, 2026Bo-An Chang, Yu-Chih ChenVLM Evaluation Reward Modeling
National Tsing Hua University, Hsinchu 300044, Taiwan · National Yang Ming Chiao Tung University, Hsinchu 300093, Taiwan
AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery
Jul 16, 2026Zahratu Shabrina, Muhammad Asa, Jin Rui +2VLM Evaluation VLM Reasoning
Department of Geography, King’s College London, UK · Resilience Development Initiative, Indonesia · School of Computer Science and Electronic Engineering, University of Surrey, UK +1
VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance
Jul 16, 2026Yunfeng Liu, Yuandong Yang, Jiarui Han +5VLM Evaluation Video Understanding
1Nanjing University · 2Shanghai AI Laboratory
SportD: How do VLMs physically strategize?
Jul 16, 2026Jasin Cekinmez, Addison J. Wu, Haotian Xia +6VLM Evaluation Vision-Language Models
Princeton University · Rice University · UC Irvine +2
Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions
Jul 16, 2026Yijiang Li, Huiqi Zou, Bingyang Wang +1VLM Evaluation VLM Hallucination
UC San Diego · Northeastern University · Georgia Institute of Technology +1
Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks
Jul 14, 2026Jae Joong LeeVLM Evaluation Video-Language Model Evaluation
Department of Computer Science Purdue University West Lafayette, Indiana, USA
CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models
Jul 14, 2026Lin Peng, Cong Wan, Zeyu Guo +2VLM Evaluation Visual Reasoning
Xi’an Jiaotong University Xi’an, China
Lost in Visual Translation: A VLM-Assisted Perceptual-Semantic Coherence Framework for EEG-to-Image Reconstruction
Jul 14, 2026Sukriti Tiwari, BHVSP Subrahmanyam, Nidhi Goyal +1VLM Evaluation EEG-Based Visual Decoding
Department of Computer Science and Engineering, Mahindra University, Hyderabad, India · MU-VT Interdisciplinary Advanced Research Centre for Transformative Technologies, Mahindra University, Hyderabad, India · Avyakt Ehsaas
What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
Jul 14, 2026Farrukh RahmanVLM Evaluation Temporal Reasoning
Microsoft · Georgia Institute of Technology

Jul 28, 2026, cs.CVJ/K move · Enter open · S save

Evaluating VLMs for Autonomous Agent-Driven Geometry Clipping Detection in Video Game QA

Carlos Celemin, Benedict Wilkins, Adrián Barahona-Ríos, Saman Zadtootaghaj+1

Sony Interactive Entertainment, London, United Kingdom

VLM Evaluation VLM Robustness Automated Software Testing Zero-Shot Anomaly Detection

Open paper AskPDF arXiv

In this work, we study the use of Vision-Language Models (VLMs) for anomaly detection in an agent-driven game Quality Assurance (QA) pipeline focusing on geometry clipping. In this evaluation, a custom exploration agent navigates a game level to collect visual observations, while the automatic annotation pipeline provides frame-level clipping labels. This setup allows us to evaluate recent VLMs on a controlled anomaly detection task without manual annotation. We benchmark six recent VLMs (Gemini, GPT, Qwen, Gemma, Llama, and Ministral) under a zero-shot prompting setting and analyse their sensitivity to four prompt variants. Our results show that while the VLMs can capture visual cues associated with geometry clipping, they all produce substantial false positives on visually ambiguous frames such as near-contact geometry and partial occlusions. Gemini-3.1-Flash achieves the best overall accuracy and is the most robust to prompt variation, while open-source models exhibit large precision--recall swings depending on the prompt design. These findings suggest that current VLMs are best suited as high-recall candidate filters within multi-stage QA pipelines rather than as standalone bug detectors.

Momentum

Related topics

Latest papers 649