VLM Evaluation

VLM: Vision-Language Model

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

SF20K Competition 2025: Summary and findings
May 2, 2026Ridouane Ghermi, Xi Wang, Vicky Kalogeiton +1VLM Evaluation Video Understanding
VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models
May 2, 2026Pang Liu, Yingjie LaoVLM Evaluation VLM Robustness
Department of Electrical and Computer Engineering, Tufts University
VISTA: Video Interaction Spatio-Temporal Analysis Benchmark
May 2, 2026Alejandro Aparcedo, Akash Kumar, Aaryan Garg +5VLM Evaluation Video Understanding
University of Central Florida · BITS Pilani · Ho Chi Minh City University of Science +1
Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification
May 1, 2026Charles Weng, Dingwen Li, Alexander MartinVLM Evaluation Confidence Estimation for VLMs
Johns Hopkins University · Independent
Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception
Apr 30, 2026Neemias B da Silva, Rodrigo Minetto, Daniel Silver +1VLM Evaluation Personality Modeling in Language Models
Universidade Tecnol´ogica Federal do Paran´a (UTFPR), Curitiba, Brazil · University of Toronto, Toronto, Canada
ClimateVID -- Social Media Videos Analysis and Challenges Involved
Apr 30, 2026Shiqi Xu, Moritz Burmester, Katharina Prasse +3VLM Evaluation Social Media Analysis
University of Mannheim · Technical University of Munich · University of Mannheim, Max-Planck-Institute for Informatics, Saarland Informatics Campus
The Effects of Visual Priming on Cooperative Behavior in Vision-Language Models
Apr 30, 2026Kenneth J. K. OngVLM Evaluation VLM Robustness
1ST Engineering, Singapore.
Auditing Frontier Vision-Language Models for Trustworthy Medical VQA: Grounding Failures, Format Collapse, and Domain Adaptation
Apr 30, 2026Xupeng Chen, Binbin Shi, Chenqian Le +5VLM Evaluation Healthcare
New York University, New York, USA · Tsinghua University, Beijing, China · Columbia University, New York, USA +1
WaferSAGE: Large Language Model-Powered Wafer Defect Analysis via Synthetic Data Generation and Rubric-Guided Reinforcement Learning
Apr 30, 2026Ke Xu, Zhongyuan LianVLM Evaluation Visual Question Answering
Shanghai Huahong Grace Semiconductor Manufacturing Corporation, Shanghai, 201203, China · Dept. of Automation, School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China
Revealing the Impact of Visual Text Style on Attribute-based Descriptions Produced by Large Visual Language Models
Apr 30, 2026Xiaomeng Wang, Martha Larson, Zhengyu ZhaoVLM Evaluation Vision-Language Models
Radboud University · Nijmegen, Netherlands · Xi’an Jiaotong University +1
Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations
Apr 28, 2026Chen Liang, Xirui Jiang, Naihao Deng +2VLM Evaluation Video Understanding
University of Michigan
QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding
Apr 28, 2026Shuxiang Cao, Zijian Zhang, Abhishek Agarwal +29VLM Evaluation In-Context Learning
1NVIDIA · University of Toronto · 13Vector Institute for Artificial Intelligence +10
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
Apr 28, 2026Divake Kumar, Sina Tayebati, Devashri Naik +2VLM Evaluation Vision-Language Models
University of Illinois at Chicago · 2AI Labs at Capital One
DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Apr 28, 2026Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande +4VLM Evaluation Visual Reasoning
Arizona State University · Adobe Research · University of Maryland
FCMBench-Video: Benchmarking Document Video Intelligence
Apr 28, 2026Runze Cui, Fangxin Shang, Yehui Yang +3VLM Evaluation Document Understanding
AI Lab, Qifu Technology, Beijing, China · School of Future Technology, South China University of Technology, Guangzhou, China · Pazhou Lab, Guangzhou, China +1
One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations
Apr 28, 2026Ravikumar Balakrishnan, Sanket MendaparaVLM Evaluation VLM Robustness
Cisco Systems
Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics
Apr 27, 2026Hai Wang, Xiaochen Yang, Mingzhi Dong +1VLM Evaluation VLM Adaptation
Department of Statistical Science, University College London, UK · School of Mathematics and Statistics, University of Glasgow, UK · Department of Computer Science, University of Bath, UK.
A systematic evaluation of vision-language models for observational astronomical reasoning tasks
Apr 27, 2026Wenke Ren, Hengxiao Guo, Wenwen Zuo +1VLM Evaluation Vision-Language Models
Shanghai Astronomical Observatory, Chinese Academy of Sciences, Shanghai, China · Harvard University, Boston, Massachusetts, USA
AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
Apr 27, 2026Hongxin Li, Xiping Wang, Jingran Su +4VLM Evaluation GUI Agents
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3
SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters
Apr 27, 2026Arya Shah, Deepali Mishra, Chaklam SilpasuwanchaiVLM Evaluation Vision-Language Models
IIT Gandhinagar Gandhinagar, India · IIT Kanpur Kanpur, India · Asian Institute of Technology Bangkok, Thailand
ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning
Apr 27, 2026Yiming Zhang, Jiacheng Chen, Jiaqi Tan +3VLM Evaluation Spatial Reasoning Benchmarks
1Simon Fraser University · 2Hong Kong University of Science and Technology · University of Waterloo +1
PushupBench: Your VLM is not good at counting pushups
Apr 25, 2026Shengzhi Li, Jiarun Chen, Karun Sharma +2Multimodal Understanding Video-Language Model Evaluation
GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI
Apr 25, 2026Duaa Alim, Mogtaba Alim, Liam ChalcroftVLM Evaluation Rare Disease Diagnosis
Imperial College London, UK · University of Toronto, Canada · University College London, UK
When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMs
Apr 23, 2026Pegah Khayatan, Jayneel Parekh, Arnaud Dapogny +3VLM Evaluation LLM Hallucination Mitigation
1ISIR, Sorbonne Universit´e, Paris, France · 2Valeo.ai, Paris, France
From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media
Apr 23, 2026Katharina Prasse, Steffen Jung, Isaac Bravo +4VLM Evaluation Social Media Analysis
Chair of Machine Learning, University of Mannheim Mannheim, BW 68131, Germany · Department of Governance, Technical University Munich München, BA 80333, Germany · Clausthal University of Technology Clausthal-Zellerfeld, NI 38678, Germany +1
Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
Apr 23, 2026Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1VLM Evaluation VLM Robustness
♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad
VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought
Apr 23, 2026Byeonggeuk Lim, Kyeonghyun Kim, JungMin Yun +1VLM Evaluation CoT Reasoning
Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University1 · Department of Artificial Intelligence, Chung-Ang University2
Can MLLMs "Read" What is Missing?
Apr 23, 2026Jindi Guo, Chaozheng Huang, Xi FangVLM Evaluation Document Understanding
DP Technology
OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model
Apr 22, 2026Qiguang Chen, Chengyu Luan, Jiajun Wu +7VLM Evaluation Visual Reasoning
Research Center for Social Computing and Interactive Robotics, Harbin Institute of Technology · Central South University · Fudan University +3
The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
Apr 22, 2026Karan GoyalVLM Evaluation Multimodal Reasoning
IIIT Delhi, India

May 2, 2026J/K move · Enter open · S save

SF20K Competition 2025: Summary and findings

Ridouane Ghermi, Xi Wang, Vicky Kalogeiton, Ivan Laptev

VLM Evaluation Video Understanding Video QA Long-Video QA

Open paper AskPDF arXiv

This report presents the results and findings of the first edition of the Short-Films 20K (SF20K) Competition, held in conjunction with the SLoMO Workshop at ICCV 2025. The competition is designed to advance story-level video understanding beyond short-clip action recognition, introducing an open-ended video question-answering task built on a corpus of amateur short films. This setup ensures that models must rely on multimodal understanding rather than memorization of popular movies. Evaluation is conducted using the SF20K-Test benchmark (95 movies, 979 question-answer pairs) and scored via LLM-QA-Eval, an automated judge based on GPT-4.1-nano. The competition attracted 22 teams and 286 submissions across two tracks: a Main Track with unrestricted model size and a Special Track limited to models under 8 billion parameters. The winning team achieved 65.7% accuracy on the Main Track and 48.7% on the Special Track, against a human performance ceiling of 91.7%. Our analysis reveals several key findings: narrative-aware, shot-level processing consistently outperforms uniform frame sampling; well-designed multi-stage pipelines using smaller models can match or exceed end-to-end inference with models over 30x larger; and subtitle quality is a dominant factor in performance. These results highlight that the primary bottleneck in long-form video QA lies in information selection and reasoning structure rather than raw model capacity, and that a substantial gap remains between current methods and human-level narrative comprehension.