VLM Evaluation

VLM: Vision-Language Model

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality
Apr 7, 2026Yanming Xiu, Zhengyuan Jiang, Neil Zhenqiang Gong +1Augmented Reality VLM Robustness
Duke University
Can Vision-Language Models Think from the Sky? Unifying UAV Reasoning and Generation
Apr 7, 2026Jintao Sun, Gangyi Ding, Donglin Di +2VLM Reasoning Unified Multimodal Models
Beijing Institute of Technology · Harbin Institute of Technology · CSIRO Data61 +1
Watch Before You Answer: Learning from Visually Grounded Post-Training
Apr 6, 2026Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang +8VLM Post-Training Vision-Language Models
University of British Columbia · Vector Institute · Etude AI +4
DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs
Apr 6, 2026Dikshant Kukreja, Kshitij Sah, Karan Goyal +2Reasoning for VLMs VLM Evaluation
IIIT Delhi, India
Discovering Failure Modes in Vision-Language Models using RL
Apr 6, 2026Kanishk Jain, Qian Yang, Shravan Nayak +3VLM Evaluation VLM Robustness
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Michigan State University +2
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
Apr 6, 2026Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen +2VLM Robustness VLM Evaluation
Zhejiang University of Technology, Hangzhou, Zhejiang, China · National Institute of Informatics, Tokyo, Japan
Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models
Apr 6, 2026Yunkai Zhang, Linda Li, Yingxin Cui +5VLM Evaluation Fine-Grained Visual Perception
BAIR, UC Berkeley · UC Santa Cruz · Analogy AI
Evaluation of Embedding-Based and Generative Methods for LLM-Driven Document Classification: Opportunities and Challenges
Apr 5, 2026Rong Lu, Hao Liu, Song HouEarth Science Text Classification
GENFIG1: Visual Summaries of Scholarly Work as a Challenge for Vision-Language Models
Apr 5, 2026Yaohan Guan, Pristina Wang, Najim Dehak +3T2I Generation Evaluation VLM Evaluation
Johns Hopkins University
When Does Multimodal AI Help? Diagnostic Complementarity of Vision-Language Models and CNNs for Spectrum Management in Satellite-Terrestrial Networks
Apr 4, 2026Yuanhang LiVLM Evaluation Wireless Communications
VidNum: Diagnosing VLM Failure Modes in Video-Grounded Numerical Reasoning
Apr 4, 2026Shaoyang Cui, Lingbei Meng, Yaodi Luo +1VLM Evaluation Video Reasoning
School of Psychological and Cognitive Sciences, Peking University, Beijing, China · Shenzhen Loop Area Institute, Shenzhen, Guangdong, China · University of Electronic Science and Technology of China, Chengdu, Sichuan, China
Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs
Apr 3, 2026Zhangyun Tan, Zeliang Zhang, Jiani Liu +4VLM Evaluation VLM Unlearning
University of Rochester
Are VLMs Lost Between Sky and Space? LinkS $^2$ Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligence
Apr 2, 2026Dian Liu, Jie Feng, Di Li +4Visual Spatial Reasoning VLM Evaluation
Xidian University, China · Qinghai Normal University, China · Sun Yat-sen University, China
Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance
Apr 2, 2026Jason Qiu, Zachary Meurer, Xavier Thomas +1VLM Reasoning VLM Robustness
Boston University
Hidden Meanings in Plain Sight: RebusBench for Evaluating Cognitive Visual Reasoning
Apr 2, 2026Seyed Amir Kasaei, Arash Marioriyad, Mahbod Khaleti +3VLM Reasoning Visual Reasoning
Department of Computer Engineering Sharif University of Technology
True (VIS) Lies: Analyzing How Generative AI Recognizes Intentionality, Rhetoric, and Misleadingness in Visualization Lies
Apr 1, 2026Graziano Blasilli, Marco AngeliniData Visualization VLM Evaluation
Sapienza University of Rome, Rome, Italy. · Link Campus University, Rome, Italy.
JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
Apr 1, 2026Issa Sugiura, Koki Maeda, Shuhei Kurita +3VLM Evaluation Benchmark Construction
Kyoto University · NII LLMC · Institute of Science Tokyo +2
When VLMs 'Fix' Students: Identifying and Penalizing Over-Correction in the Evaluation of Multi-line Handwritten Math OCR
Apr 1, 2026Jin Seong, Wencke Liermann, Minho Kim +2Handwritten Mathematical Expression Recognition Self-Correction in VLMs
Electronics and Telecommunications Research Institute, Republic of Korea
TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios
Mar 31, 2026Qiucheng Yu, Ruijie Xu, Mingang Chen +2VLM Evaluation VLM Robustness
City University of Hong Kong · East China Normal University · Shanghai Development Center of Computer Software Technology +1
Video-Oasis: Rethinking Evaluation of Video Understanding
Mar 31, 2026Geuntaek Lim, Sungjune Park, Jaeyun Lee +5VLM Evaluation Temporal Video Understanding
Sejong University, South Korea · NAVER Cloud, South Korea
ChartDiff: A Large-Scale Benchmark for Comprehending Pairs of Charts
Mar 30, 2026Rongtian YeChart Understanding Visual Reasoning
Department of Computer Science, Aalto University
AMIGO: Agentic Multi-Image Grounding Oracle Benchmark
Mar 30, 2026Min Wang, Ata MahjoubfarVLM Evaluation Visual Question Answering
Target Corporation
SEA: Evaluating Sketch Abstraction Efficiency via Element-level Commonsense Visual Question Answering
Mar 30, 2026Jiho Park, Sieun Choi, Jaeyoon Seo +3Automated Evaluation VLM Evaluation
Dongguk University, Republic of Korea
CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models
Mar 30, 2026Kesheng Chen, Yamin Hu, Qi Zhou +2Emotional Hallucination in VLMs VLM Hallucination
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
Mar 30, 2026Koki Maeda, Naoaki OkazakiVLM Evaluation Scene Text Recognition
Institute of Science Tokyo, Tokyo, Japan · Research and Development Center for Large Language Models, National Institute of Informatics, Tokyo, Japan
Inference-Time Structural Reasoning for Compositional Vision-Language Understanding
Mar 28, 2026Amartya BhattacharyaCompositional Reasoning in VLMs VLM Reasoning
Biomedical Data Science, Geisel School of Medicine, Dartmouth College
Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark
Mar 28, 2026Seng Nam Chen, Hao Chen, Chenglam Ho +4Long-Video Understanding Vision-Language Models
CUHK (SZ) · University of Cambridge · UESTC +3
The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding
Mar 27, 2026Gillian Rosenberg, Skylar Stadhard, Bruce C. Hansen +1Large Vision-Language Models VLM Evaluation
Barnard College, Columbia University · Colgate University
HandVQA: Diagnosing and Improving Fine-Grained Spatial Reasoning about Hands in Vision-Language Models
Mar 27, 2026MD Khalequzzaman Chowdhury Sayem, Mubarrat Tajoar Chowdhury, Yihalem Yimolal Tiruneh +4VLM Reasoning Spatial Reasoning
UNIST · University of Aberdeen · University College London +1
VLAgeBench: Benchmarking Large Vision-Language Models for Zero-Shot Human Age Estimation
Mar 27, 2026Rakib Hossain Sajib, Md Kishor Morol, Rajan Das Gupta +2Large Vision-Language Models VLM Evaluation
Department of Computer Science and Engineering, Begum Rokeya University, Rangpur, Rangpur, Bangladesh · EliteLab.AI, Queens, New York, United States · Department of Computer Science, American International University - Bangladesh, Dhaka, Bangladesh

Apr 7, 2026J/K move · Enter open · S save

Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality

Yanming Xiu, Zhengyuan Jiang, Neil Zhenqiang Gong, Maria Gorlatova

Duke University

Augmented Reality VLM Robustness VLM Evaluation Adversarial Attacks on VLMs Adversarial Attacks

Open paper AskPDF arXiv

Augmented reality (AR) has rapidly expanded over the past decade. As AR becomes increasingly integrated into daily life, its security and reliability emerge as critical challenges. Among various threats, contradictory virtual content attacks, where malicious or inconsistent virtual elements are introduced into the user's view, pose a unique risk by misleading users, creating semantic confusion, or delivering harmful information. In this work, we systematically model such attacks and present ContrAR, a novel benchmark for evaluating the robustness of vision-language models (VLMs) against virtual content manipulation and contradiction in AR. ContrAR contains 312 real-world AR videos validated by 10 human participants. We further benchmark 11 VLMs, including both commercial and open-source models. Experimental results reveal that while current VLMs exhibit reasonable understanding of contradictory virtual content, room still remains for improvement in detecting and reasoning about adversarial content manipulations in AR environments. Moreover, balancing detection accuracy and latency remains challenging.