VLM Evaluation

VLM: Vision-Language Model

Vision-language model evaluation studies what these systems can infer from images and video, and whether their answers are supported by visual evidence. Researchers use broad benchmarks alongside controlled probes, counterfactual edits, and stage-wise tests covering tasks such as spatial reasoning, charts and documents, medical imaging, and long-video understanding. They assess not only accuracy but also grounding, hallucination, calibration, robustness, bias, and abstention, often comparing models with human or specialist judgments. A central challenge is that high benchmark scores can reflect language priors or shortcuts rather than reliable visual understanding, and performance often falters on fine-grained, temporal, compositional, or unfamiliar tasks.

Latest papers

999 papers122 papers in the last four weeks, up 12% on the four weeks before. 0.7% of all new papers.

Evaluating Remote Sensing Image Captions Beyond Metric Biases
Apr 22, 2026Ziyun Chen, Fan Liu, Liang Yao +3VLM Evaluation Remote Sensing Image Understanding
Hohai University Nanjing, China · Cardiff University Cardiff, UK
EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving
Apr 22, 2026Finn Rasmus Schäfer, Yuan Gao, Dingrui Wang +5VLM Evaluation Autonomous Driving Benchmarks
Professorship of Autonomous Vehicle Systems, Technical University of Munich, Munich, Germany · Bayerische Motoren Werke AG, Munich, Germany · Data Analytics and Machine Learning Group, Technical University of Munich, Munich, Germany
Culturally uneven urban perception in large language models
Apr 21, 2026Rong Zhao, Wanqi Liu, Zhizhou Sha +3VLM Evaluation Social Bias in Language Models
Centre for Advanced Spatial Analysis (CASA), UCL, London, UK · Department of Computer Science, UT Austin, Austin, TX, USA · School of Architecture, Tsinghua University, Beijing, China
DistortBench: Benchmarking Vision Language Models on Image Distortion Identification
Apr 21, 2026Divyanshu Goyal, Akhil Eppa, Vanya Bannihatti KumarVLM Evaluation Benchmark Construction
Adobe Inc.
Assessing VLM-Driven Semantic-Affordance Inference for Non-Humanoid Robot Morphologies
Apr 21, 2026Jess Jones, Raul Santos-Rodriguez, Sabine HauertVLM Evaluation Cross-Embodiment Robot Learning
Bristol Robotics Laboratory, University of Bristol Bristol, United Kingdom · University of Bristol Bristol, United Kingdom
Seeing Candidates at Scale: Multimodal LLMs for Visual Political Communication on Instagram
Apr 21, 2026Michael Achmann-Denkler, Mario Haim, Christian WolffVLM Evaluation Social Media Analysis
Media Informatics Group University of Regensburg Regensburg, Germany · Department of Media and Communication Ludwig-Maximilians-Universität Munich, Germany
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multimodal Model Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
Disparities In Negation Understanding Across Languages In Vision-Language Models
Apr 21, 2026Charikleia Moraitaki, Sarah Pan, Skyler Pulling +3VLM Evaluation Vision-Language Models
Massachusetts Institute of Technology
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models
Apr 20, 2026Zhiyuan Jiang, Weihao Hong, Xinlei Guan +8VLM Evaluation Hallucination Detection in VLMs
Kean University, USA · North Dakota State University, Fargo, ND, USA · McGill University, Montreal, Quebec, Canada +1
Revisiting Change VQA in Remote Sensing with Structured and Native Multimodal Qwen Models
Apr 20, 2026Yakoub Bazi, Mohamad M. Al Rahhal, Mansour Zuair +1VLM Evaluation Remote Sensing Image Understanding
Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia · Applied Computer Science Department, College of Applied Computer Science, King Saud University, Riyadh 11543, Saudi Arabia
Multilingual Training and Evaluation Resources for Vision-Language Models
Apr 20, 2026Daniela Baiamonte, Elena Fano, Matteo Gabburo +3VLM Evaluation Vision-Language Models
Aithlas
Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models
Apr 20, 2026Haiweng Xu, Sipeng Zheng, Hao Luo +3VLM Evaluation VLM Reasoning
1Peking University · 3BeingBeyond · 2Tsinghua University
Leveraging Vision-Language Models to Detect Attention in Educational Videos
Apr 20, 2026Gabriel Becquet, Sébastien Lallé, Vanda Luengo +1VLM Evaluation Educational Technology
LIP6, CNRS, SU · Sorbonne University, CNRS, LIP6 & PHENIX, F-75005 Paris, France · CNRS, LIP6, SU +2
Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models
Apr 20, 2026Ziyao Tang, Pengkun Jiao, Bin Zhu +3VLM Evaluation VLM Robustness
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
Apr 20, 2026Xiaohan Zou, Roshan Sridhar, Mohammadtaher Safarzadeh +1VLM Evaluation Vision-Language Alignment
The Pennsylvania State University · Oracle AI
Lost in Motion: Vision Language Models Fail the Dynamic Gauges Test
Apr 19, 2026Tairan Fu, Francisco Javier Santos-Martín, Javier Conde +2VLM Evaluation Industrial Inspection
When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models
Apr 19, 2026Cui Yakun, Xingqun Qi, TianTian Geng +3VLM Evaluation VLM Hallucination
The Hong Kong University of Science and Technology · Hong Kong, China · University of Birmingham
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
Apr 19, 2026Wei HeVLM Evaluation Figurative Language Understanding
IDSAI, University of Exeter Exeter, UK
DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models
Apr 18, 2026JiYang Wang, Jiawei Chen, Mengqi Xiao +3VLM Evaluation Hallucination Detection in VLMs
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy
Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalization
Apr 17, 2026Siddhant Bharadwaj, Ashish Vashist, Fahimul Aleem +1VLM Evaluation Large Vision-Language Models
University of Central Florida
HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning
Apr 17, 2026Yanbin Wei, Chun Kang, Siwei Li +11VLM Evaluation Large Vision-Language Models
1Southern University of Science and Technology · 2Hong Kong University of Science and Technology · 4Beihang University +1
Why Do Vision Language Models Struggle To Recognize Human Emotions?
Apr 16, 2026Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara +1VLM Evaluation Multimodal Emotion Recognition
The University of Edinburgh, UK
MetaDent: Labeling Clinical Images for Vision-Language Models in Dentistry
Apr 16, 2026Meng-Xun Li, Wen-Hui Deng, Zhi-Xing Wu +6VLM Evaluation Medical Image Analysis
Department of Prosthodontics, State Key Laboratory of Oral & Maxillofacial Reconstruction and Regeneration, Key Laboratory of Oral Biomedicine, Ministry of Education, Hubei Key Laboratory of Stomatology, School and Hospital of Stomatology, Wuhan University, Wuhan, Hubei, China · School of Artificial Intelligence, Wuhan University, Wuhan, Hubei, China · School of Computer Science, Wuhan University, Wuhan, Hubei, China +2
Zoom Consistency: A Free Confidence Signal in Multi-Step Visual Grounding Pipelines
Apr 15, 2026Keon Kim, Krish ChelikavadaConfidence Estimation for VLMs Vision-Language Grounding
Om Labs
FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images
Apr 15, 2026Sabab Ishraq, Aarushi Aarushi, Juncai Jiang +1VLM Evaluation
College of Engineering and Computer Science, University of Central Florida, Orlando, FL, USA · College of Business Administration, University of Central Florida, Orlando, FL, USA · Institute of Artificial Intelligence, University of Central Florida, Orlando, FL, USA
GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
Apr 15, 2026Yangyue Wang, Harshvardhan Sikka, Yash Mathur +3VLM Evaluation Domain Randomization
1Fig · 2Manifold Research
Seek-and-Solve: Benchmarking MLLMs for Visual Clue-Driven Reasoning in Daily Scenarios
Apr 15, 2026Xiaomin Li, Tala Wang, Zichen Zhong +7VLM Evaluation Multimodal Reasoning
Dalian University of Technology · WeChat, Tencent Inc. · Tsinghua University
SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis
Apr 14, 2026Kathakoli Sengupta, Kai Ao, Paola Cascante-Bonilla3D Scene Generation 3D Scene Layout Generation
Stony Brook University
Do VLMs Truly "Read" Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting
Apr 14, 2026Kaiqi Hu, Linda Xiao, Shiyue Xu +2Stock Price Forecasting Financial Forecasting
University of Leeds, Leeds, UK · Sun Yat-sen University, Zhuhai, China · Sun Yat-sen University, GuangZhou, China +1
Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs
Apr 14, 2026Jianhao Chen, Shiqin Wang, Haoyang Chen +4Adversarial Attacks on VLMs VLM Evaluation
School of Computer Science, Wuhan University, Wuhan, China · Zhongguancun Academy, Beijing, China · Tianjin University, Tianjin, China +1

Apr 22, 2026J/K move · Enter open · S save

Evaluating Remote Sensing Image Captions Beyond Metric Biases

Ziyun Chen, Fan Liu, Liang Yao, Chuanyi Zhang+2

Hohai University Nanjing, China · Cardiff University Cardiff, UK

VLM Evaluation Remote Sensing Image Understanding Remote Sensing Image Captioning LLM Self-Refinement Large Vision-Language Models

Open paper AskPDF arXiv

The core objective of image captioning is to achieve lossless semantic compression from visual signals into textual modalities. However, the reliance on manually curated reference texts for evaluation essentially forces models to mimic specific human annotation styles, thereby masking the true descriptive capabilities of advanced foundation models. This systemic misalignment prompts a critical question: Is task-specific fine-tuning truly necessary for Remote Sensing Image Captioning, or is the perceived performance gap merely an artifact of flawed evaluation criteria? To investigate this discrepancy, we propose ReconScore, a novel reference-free evaluation metric. Rather than computing textual similarities, we assess caption quality by its capability to reconstruct the original visual elements solely from the generated text, effectively neutralizing human annotation biases. Applying this metric, we uncover a profound, counterintuitive truth: inherently powerful, unfine-tuned MLLMs surpass their fine-tuned counterparts in authentic zero-shot RSIC tasks. Driven by this structural discovery, we introduce RemoteDescriber, a completely training-free generation methodology. By employing ReconScore as a self-correction mechanism, we iteratively refine the semantic precision of MLLM outputs without any computational fine-tuning overhead. Comprehensive experiments demonstrate that RemoteDescriber achieves state-of-the-art performance on three datasets. Furthermore, we validate ReconScore's reliability and analyze the flaws of traditional metrics. Our code is available at https://github.com/hhu-czy/RemoteDescriber.