VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

70 papers in the last four weeks, up 67% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 649

All topics

Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5VLM Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
Disparities In Negation Understanding Across Languages In Vision-Language Models
Apr 21, 2026Charikleia Moraitaki, Sarah Pan, Skyler Pulling +3VLM Evaluation Vision-Language Models
Massachusetts Institute of Technology
LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models
Apr 20, 2026Zhiyuan Jiang, Weihao Hong, Xinlei Guan +8VLM Evaluation Hallucination Detection in VLMs
Kean University, USA · North Dakota State University, Fargo, ND, USA · McGill University, Montreal, Quebec, Canada +1
Revisiting Change VQA in Remote Sensing with Structured and Native Multimodal Qwen Models
Apr 20, 2026Yakoub Bazi, Mohamad M. Al Rahhal, Mansour Zuair +1VLM Evaluation Remote Sensing Image Understanding
Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia · Applied Computer Science Department, College of Applied Computer Science, King Saud University, Riyadh 11543, Saudi Arabia
Multilingual Training and Evaluation Resources for Vision-Language Models
Apr 20, 2026Daniela Baiamonte, Elena Fano, Matteo Gabburo +3VLM Evaluation Vision-Language Models
Aithlas
Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models
Apr 20, 2026Haiweng Xu, Sipeng Zheng, Hao Luo +3VLM Evaluation VLM Reasoning
1Peking University · 3BeingBeyond · 2Tsinghua University
Leveraging Vision-Language Models to Detect Attention in Educational Videos
Apr 20, 2026Gabriel Becquet, Sébastien Lallé, Vanda Luengo +1VLM Evaluation Educational Technology
LIP6, CNRS, SU · Sorbonne University, CNRS, LIP6 & PHENIX, F-75005 Paris, France · CNRS, LIP6, SU +2
Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models
Apr 20, 2026Ziyao Tang, Pengkun Jiao, Bin Zhu +3VLM Evaluation VLM Robustness
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
Apr 20, 2026Xiaohan Zou, Roshan Sridhar, Mohammadtaher Safarzadeh +1VLM Evaluation Vision-Language Alignment
The Pennsylvania State University · Oracle AI
Lost in Motion: Vision Language Models Fail the Dynamic Gauges Test
Apr 19, 2026Tairan Fu, Francisco Javier Santos-Martín, Javier Conde +2VLM Evaluation Industrial Inspection
When Text Hijacks Vision: Benchmarking and Mitigating Text Overlay-Induced Hallucination in Vision Language Models
Apr 19, 2026Cui Yakun, Xingqun Qi, TianTian Geng +3VLM Evaluation VLM Hallucination
The Hong Kong University of Science and Technology · Hong Kong, China · University of Birmingham
More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage
Apr 19, 2026Wei HeVLM Evaluation Figurative Language Understanding
IDSAI, University of Exeter Exeter, UK
DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models
Apr 18, 2026JiYang Wang, Jiawei Chen, Mengqi Xiao +3VLM Evaluation Hallucination Detection in VLMs
Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University · Zhongguancun Academy
Where Do Vision-Language Models Fail? World Scale Analysis for Image Geolocalization
Apr 17, 2026Siddhant Bharadwaj, Ashish Vashist, Fahimul Aleem +1VLM Evaluation Large Vision-Language Models
University of Central Florida
HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning
Apr 17, 2026Yanbin Wei, Chun Kang, Siwei Li +11VLM Evaluation Large Vision-Language Models
1Southern University of Science and Technology · 2Hong Kong University of Science and Technology · 4Beihang University +1
Why Do Vision Language Models Struggle To Recognize Human Emotions?
Apr 16, 2026Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara +1VLM Evaluation Multimodal Emotion Recognition
The University of Edinburgh, UK
MetaDent: Labeling Clinical Images for Vision-Language Models in Dentistry
Apr 16, 2026Meng-Xun Li, Wen-Hui Deng, Zhi-Xing Wu +6VLM Evaluation Medical Image Analysis
Department of Prosthodontics, State Key Laboratory of Oral & Maxillofacial Reconstruction and Regeneration, Key Laboratory of Oral Biomedicine, Ministry of Education, Hubei Key Laboratory of Stomatology, School and Hospital of Stomatology, Wuhan University, Wuhan, Hubei, China · School of Artificial Intelligence, Wuhan University, Wuhan, Hubei, China · School of Computer Science, Wuhan University, Wuhan, Hubei, China +2
GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
Apr 15, 2026Yangyue Wang, Harshvardhan Sikka, Yash Mathur +3VLM Evaluation Domain Randomization
1Fig · 2Manifold Research
Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM Evaluation Confidence Estimation in Language Models
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA
RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Models
Apr 13, 2026Yakun Yu, Ashley Wiens, Adrián Barahona-Ríos +4VLM Evaluation Video Anomaly Detection
University of Alberta · Sony Interactive Entertainment
Cross-Cultural Value Attribution in Large Vision-Language Models
Apr 10, 2026Phillip Howard, Xin Su, Kathleen C. FraserVLM Evaluation Counterfactual Evaluation of VLMs
Thoughtworks · University of Ottawa
Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs
Apr 3, 2026Zhangyun Tan, Zeliang Zhang, Jiani Liu +4VLM Evaluation VLM Unlearning
University of Rochester
TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarios
Mar 31, 2026Qiucheng Yu, Ruijie Xu, Mingang Chen +2VLM Evaluation VLM Robustness
City University of Hong Kong · East China Normal University · Shanghai Development Center of Computer Software Technology +1
Video-Oasis: Rethinking Evaluation of Video Understanding
Mar 31, 2026Geuntaek Lim, Sungjune Park, Jaeyun Lee +5VLM Evaluation Temporal Video Understanding
Sejong University, South Korea · NAVER Cloud, South Korea
AMIGO: Agentic Multi-Image Grounding Oracle Benchmark
Mar 30, 2026Min Wang, Ata MahjoubfarVLM Evaluation Visual Question Answering
Target Corporation
Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs
Mar 23, 2026Efthymios Tsaprazlis, Tiantian Feng, Anil Ramakrishna +3VLM Evaluation Privacy Auditing
University of Southern California, Los Angeles CA 90089, USA · Meta Superintelligence Labs · Amazon AGI
CREG: Compass Relational Evidence Graph for Characterizing Directional Structure in VLM Spatial-Reasoning Attribution
Mar 20, 2026Kaizhen Tan, Yang Feng, Heqing DuVLM Evaluation Gradient-Based Attribution
Carnegie Mellon University Pittsburgh, PA, USA · Columbia University New York, NY, USA
Do Vision Language Models Understand Human Engagement in Games?
Mar 19, 2026Ziyi Wang, Qizan Guo, Rishitosh Singh +1VLM Evaluation Video-Language Models
Arizona State University
GroundSet: A Cadastral-Grounded Dataset for Spatial Understanding with Vector Data
Mar 15, 2026Roger Ferrod, Maël Lecene, Krishna Sapkota +4VLM Evaluation Remote Sensing Image Understanding
Université Paris Cité, LIPADE, F-75006 Paris, France · Google Research
vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
Mar 14, 2026Suhwan Choi, Yunsung Lee, Yubeen Park +4VLM Evaluation Vision-Language-Action Models
Allen Institute for AI (AI2) · Seoul National University

Apr 21, 2026, cs.CLJ/K move · Enter open · S save

Lost in Translation: Do LVLM Judges Generalize Across Languages?

Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem, Amran Bhuiyan+4

York University · University of Alberta · Nanyang Technological University+1

VLM Evaluation Multilingual VLM Evaluation Multimodal Reward Modeling Vision-Language Model-as-a-Judge Cross-Lingual Consistency in Language Models Multimodal Model Evaluation

Open paper AskPDF arXiv

Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.

Momentum

Related topics

Latest papers 649