VLM Evaluation

VLM: Vision-Language Model

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models
Mar 10, 2026Tengjin Weng, Wenhao Jiang, Jingyi Wang +3Fine-Grained Visual Perception Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua Shenzhen International Graduate School +2
Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning
Mar 9, 2026Heesup Yun, Isaac Kazuo Uyehara, Earl Ranario +4VLM Evaluation Agricultural Image Analysis
University of California, Davis
MEGC2026: Micro-Expression Grand Challenge on Visual Question Answering
Mar 9, 2026Xinqi Fan, Jingting Li, John See +3Micro-Expression Recognition Video QA
Department of Computing and Mathematics, Manchester Metropolitan University · State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, CAS & Department of Psychology, University of the Chinese Academy of Sciences · School of Mathematical and Computer Sciences, Heriot-Watt University Malaysia
Reading $\neq$ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models
Mar 9, 2026Heng Zhou, Ao Yu, Li Kang +5Vision-Language Models VLM Evaluation
Can Vision-Language Models Solve the Shell Game?
Mar 9, 2026Tiedong Liu, Wee Sun LeeVisual Object Tracking Temporal Video Understanding
National University of Singapore
Text to Automata Diagrams: Comparing TikZ Code Generation with Direct Image Synthesis
Mar 9, 2026Ethan Young, Zichun Wang, Aiden Taylor +6Code Generation VLM Evaluation
Marshall University · West Virginia State University
VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?
Mar 9, 2026Minkyu Kim, Sangheon Lee, Dongmin ParkVLM Evaluation VLM Reasoning
KRAFTON · KAIST
Hospitality-VQA: Decision-Oriented Informativeness Evaluation for Vision-Language Models
Mar 9, 2026Jeongwoo Lee, Baek Duhyeong, Eungyeol Han +5VLM Evaluation Visual Question Answering
Yonsei University · Yanolja NEXT
OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
Mar 8, 2026Yusuke Tozaki, Hisashi MiyamoriVLM Evaluation VLM Reasoning
Division of Frontier Informatics, Kyoto Sangyo University, Kyoto, Japan
Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework
Mar 8, 2026Kaihua Tang, Jiaxin Qi, Jinli Ou +2VLM Robustness Test-Time Scaling for VLMs
Tongji University, China · Computer Network Information Center, CAS, China · University of Chinese Academy of Sciences, China +1
Prompt-Based Caption Generation for Single-Tooth Dental Images Using Vision-Language Models
Mar 8, 2026Anastasiia Sukhanova, Aiden Taylor, Julian Myers +5Dental Imaging Image Captioning
Marshall University · West Virginia State University
Vision Language Models Cannot Reason About Physical Transformation
Mar 7, 2026Dezhi Luo, Yijiang Li, Maijunxian Wang +7Physical Reasoning Vision-Language Models
University of Michigan · University of California San Diego · University of California Berkeley +6
VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
Mar 7, 2026Xueqing Yu, Bohan Li, Yan Li +1Abstention in VLMs VLM Evaluation
Peking University · ByteDance
VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
Mar 6, 2026Rohit Saxena, Alessandro Suglia, Pasquale MinerviniVLM Robustness VLM Evaluation
University of Edinburgh, UK
Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models
Mar 6, 2026Nicoleta-Nina Basoc, Adrian Cosma, Emilian RadoiVLM Robustness Vision-Language Models
National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania · IDSIA, The Dalle Molle Institute for Artificial Intelligence, Lugano, Switzerland
Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe
Mar 4, 2026Chris Vorster, Mayug Maniparambil, Noel E. O'Connor +2VLM Evaluation Zero-Shot Image Classification
ML-Labs, Dublin City University, Dublin, Ireland
Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild
Mar 4, 2026Changda Zhou, Ziyue Gao, Xueqing Wang +4VLM Evaluation Document Parsing Benchmarks
PaddlePaddle Team, Baidu Inc. · The Hong Kong University of Science and Technology (Guangzhou)
DISCO: Document Intelligence Suite for COmparative Evaluation
Mar 4, 2026Kenza Benkirane, Dan Goldwater, Martin Asenov +1Document Understanding Document Image Understanding
Parexel AI Labs, London, United Kingdom · Parexel AI Labs, San Francisco, United-States
When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models
Mar 4, 2026Qianpu Chen, Derya Soydaner, Rob SaundersVLM Robustness VLM Evaluation
LIACS (Leiden Institute of Advanced Computer Science), Leiden University, Leiden, The Netherlands
VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images
Mar 3, 2026Neil TripathiVLM Evaluation VLM Reasoning
New York University
Logit-Level Uncertainty Quantification in Vision-Language Models for Histopathology Image Analysis
Mar 3, 2026Betul Yurdem, Ferhat Ozgur Catak, Murat Kuzlu +1Uncertainty Quantification Medical VLMs
Department of Electrical and Electronics Engineering, Izmir Bakircay University, 35665 Izmir, Turkey · Department of Electrical Engineering and Computer Science, University of Stavanger, 4021 Stavanger, Norway · Batten College of Engineering and Technology, Old Dominion University, Norfolk, VA 23529, USA
Structure-Aware Text Recognition for Ancient Greek Critical Editions
Mar 3, 2026Nicolas Angleraud, Antonia Karamolegkou, Benoît Sagot +1Optical Character Recognition VLM Evaluation
Inria, Paris, France
Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples
Mar 2, 2026Phillip Howard, Xin Su, Kathleen C. FraserVLM Evaluation Counterfactual Evaluation of VLMs
Thoughtworks · University of Ottawa
Semantic Similarity is a Spurious Measure of Comic Understanding: Lessons Learned from Hallucinations in a Benchmarking Experiment
Mar 2, 2026Christopher Driggers-Ellis, Nachiketh Tibrewal, Rohit Bogulla +4VLM Evaluation VLM Hallucination
University of Florida, Gainesville, FL 32611
Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation
Mar 2, 2026Aditya Parikh, Aasa Feragen, Sneha Das +1VLM Evaluation Radiology Report Evaluation
DTU Compute, Technical University of Denmark
Towards Policy-Adaptive Image Guardrail: Benchmark and Method
Mar 1, 2026Caiyong Piao, Zhiyuan Yan, Haoming Xu +4Cultural Safety Evaluation of VLMs VLM Evaluation
Fudan University · Tencent · Peking University
GameVerse: Can Vision-Language Models Learn from Video-based Reflection?
Mar 1, 2026Kuan Zhang, Dongchen Liu, Qiyue Zhao +5Vision-Language Models VLM Evaluation
College of AI, Tsinghua University, China.
Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective
Mar 1, 2026Arctanx An, Shizhao Sun, Danqing Huang +5VLM Evaluation
Peking University · Microsoft Research Asia · Microsoft +1
The Aftermath of DrawEduMath: Vision Language Models Underperform with Struggling Students and Misdiagnose Errors
Mar 1, 2026Li Lucy, Albert Zhang, Nathan Anderson +2AI in Education VLM Evaluation
University of Washington · Insource Services · Worcester Polytechnic Institute +1
Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks
Feb 27, 2026Qihua Dong, Kuo Yang, Lin Ju +6Referring Expression Comprehension VLM Evaluation
Northeastern University

Mar 10, 2026J/K move · Enter open · S save

OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models

Tengjin Weng, Wenhao Jiang, Jingyi Wang, Ming Li+2

College of Computer Science and Software Engineering, Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua Shenzhen International Graduate School+2

Fine-Grained Visual Perception Multimodal Large Language Models VLM Evaluation RL for VLMs

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved remarkable performance across a wide range of vision language tasks. However, their ability in low-level visual perception, particularly in detecting fine-grained visual discrepancies, remains underexplored and lacks systematic analysis. In this work, we introduce OddGridBench, a controllable benchmark for evaluating the visual discrepancy sensitivity of MLLMs. OddGridBench comprises over 1,400 grid-based images, where a single element differs from all others by one or multiple visual attributes such as color, size, rotation, or position. Experiments reveal that all evaluated MLLMs, including open-source families such as Qwen3-VL and InternVL3.5, and proprietary systems like Gemini-2.5-Pro and GPT-5, perform far below human levels in visual discrepancy detection. We further propose OddGrid-GRPO, a reinforcement learning framework that integrates curriculum learning and distance-aware reward. By progressively controlling the difficulty of training samples and incorporating spatial proximity constraints into the reward design, OddGrid-GRPO significantly enhances the model's fine-grained visual discrimination ability. We hope OddGridBench and OddGrid-GRPO will lay the groundwork for advancing perceptual grounding and visual discrepancy sensitivity in multimodal intelligence. Code and dataset are available at https://wwwtttjjj.github.io/OddGridBench/.