VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

71 papers in the last four weeks, up 69% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 658

All topics

MMGist: A Comprehensive Multimodal Benchmark for 2027
Jun 21, 2026Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong +3VLM Evaluation Large Vision-Language Models
1Shanghai Jiao Tong University · 2Sichuan University
Zero-Shot Vision-Language Models for Classroom Engagement Recognition: A Benchmark Study of Prompt Sensitivity and Cross-Dataset Generalization
Jun 20, 2026Aman Goyal, Kshama Nitin Shah, Kemmannu Vineet Venkatesh RaoVLM Evaluation Zero-Shot Learning
Carnegie Mellon University · University of Michigan, Ann Arbor · Magna International
Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Models
Jun 19, 2026Bita Azari, Zoe Stanley, Avneet Batra +4VLM Evaluation Video-Language Model Evaluation
Simon Fraser University, Canada
MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
Jun 19, 2026Han Jang, Junhyeok Lee, Songsoo Kim +4VLM Evaluation Medical VLMs
1Seoul National University · 2Seoul National University Hospital · 3Seoul National University College of Medicine +1
CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays
Jun 19, 2026Geon Choi, Hangyul Yoon, Nalee Kim +5VLM Evaluation Medical Imaging
1KAIST · 2Samsung Medical Center · 3Konkuk University Medical Center
Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City
Jun 18, 2026Adrian Cespedes, Marcelo Chincha, Dunant Cusipuma +3VLM Evaluation Autonomous Driving Datasets
Artificio Lima, Peru
Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation
Jun 18, 2026Ali Asaria, Tony Salomone, Deep GandhiVLM Evaluation 3D Asset Generation
Transformer Lab
Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR
Jun 18, 2026Prabhjot Singh, Bhushan Pawar, Madhu ReddiboinaVLM Evaluation Vision-Language Models
RediMinds Inc., USA
LaViSA: A Language and Vision Structural Ambiguity Benchmark
Jun 17, 2026Lee Sangmyeong, Shun Inadumi, Koichiro YoshinoVLM Evaluation Vision-Language Models
1Nara Institute of Science and Technology · 2Guardian Robot Project RIKEN · 3The University of Osaka
Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
Jun 17, 2026Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin +10VLM Evaluation Vision-Language-Action Models
1CogAI Lab · 2FusionBrain Lab · 3IAI MSU +8
REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation
Jun 17, 2026Tengjie Lin, Yutao Sun, Jingwei Ni +7VLM Evaluation Visual Question Answering
Zhejiang University · ETH Zürich · Nanjing University of Science and Technology +3
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
Jun 17, 2026Hong-Tao Yu, Chen-Wei Xie, Yuxin Peng +2VLM Evaluation VLM Robustness
School of Computer Science and Engineering, Southeast University, China · Alibaba Group · Wangxuan Institute of Computer Technology, National Key Laboratory for Multimedia Information Processing, Peking University, China +2
How Well Can Your Video Model Remember? Measuring Memory-Budget Trade-offs in Long Video Understanding
Jun 17, 2026Yixian TianVLM Evaluation Streaming Video Understanding
1Fudan University · 2Beijing University of Posts and Telecommunications
Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images
Jun 16, 2026Yunzhe Xue, Mohammed Saim Ahmed Quadri, Neal Panse +2VLM Evaluation Medical VQA
Department of Data Science, New Jersey Institute of Technology, Newark, NJ, USA · Department of Computer Science, New Jersey Institute of Technology, Newark, NJ, USA · Vascular and Endovascular Surgery, Robert Wood Johnson Hospital, New Brunswick, NJ, USA
A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality (and Why Cheap Proxies Fall Short)
Jun 16, 2026Ali Asaria, Tony Salomone, Deep GandhiVLM Evaluation 3D Mesh Generation
Transformer Lab
CaVe-VLM-CoT: An Interpretable Vision-Language Model Framework
Jun 16, 2026Sneha Rao, Shaina Raza, Dhanesh RamachandramVLM Evaluation Vision-Language Grounding
Vector Institute, Toronto, ON M5G 0C6, Canada
LongWebBench: Evaluating Structural and Functional Webpage Generation in Long-Horizon Settings
Jun 16, 2026Yi Zhao, Zhen Yang, Mengpan Chen +5VLM Evaluation Web Application Generation
Tsinghua University · Yanshan University · University of Waterloo +1
Vision-language models for chest radiography do not always need the image
Jun 16, 2026Mahshad Lotfinia, Sebastian Ziegelmayer, Lisa Adams +4VLM Evaluation Medical VQA
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany +1
LADBench: A Benchmark for Logical Fault Detection in Images
Jun 16, 2026Sahasra Kondapalli, Lara Radovanovic, Aadi Palnitkar +2VLM Evaluation Multimodal Reasoning
Embodied Robotics and Automation Lab, University of South Florida, Tampa, FL 33610, USA.
Attention Alignment Between Humans and Vision-Language Models
Jun 16, 2026Isaac R. Christian, Udith Haputhanthrige, Hanna Hornfeld +4VLM Evaluation Visual Attention
Princeton Neuroscience Institute, Princeton University · Department of Psychology, Princeton University · Department of Computer Science, Princeton University +2
Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models
Jun 16, 2026Logan Mann, Yi Xia, Ajit Saravanan +6VLM Evaluation VLM Robustness
Algoverse AI Research · University of California, Santa Barbara · University of California, Berkeley +1
Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
Jun 15, 2026Prabhjot Singh, Bhushan Pawar, Madhu Reddiboina +1VLM Evaluation Multilingual Language Models
RediMinds Inc., USA · The University of Texas at Austin, USA · Indian AI Research Organization (IAIRO), India
Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment
Jun 15, 2026Zijie MengVLM Evaluation Image Quality Assessment
Peking University
Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?
Jun 15, 2026Arnisa Fazla, Alberto Testoni, Ameen Abu-Hanna +2VLM Evaluation VLM Robustness
Department of Medical Informatics, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, The Netherlands. · Amsterdam Public Health, Methodology, Amsterdam, The Netherlands. · MaiNLP, Center for Information and Language Processing, LMU Munich, Germany. +1
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
Jun 15, 2026Jieyuan Liu, Jianyang Gu, Shijie Chen +2VLM Evaluation Multimodal RAG
University of California, San Diego · The Ohio State University
Chronological Blindness: Benchmarking Temporal Reasoning in Vision-Language Models with CHRONOSIGHT
Jun 15, 2026Parthaw Goswami, Jaynto Goswami DeepVLM Evaluation Temporal Reasoning
Department of Computer Science, University of Missouri, Columbia, 65201, MO, USA · SAP, Prague, Czech Republic
TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting
Jun 15, 2026Zhi Chen, Yuxuan Wang, Jialong Wu +5VLM Evaluation Forecasting Benchmarks
School of Software, BNRist, Tsinghua University, Beijing 100084, China
VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
Jun 15, 2026Young Rok Jang, Hyesoo Kong, Kyunghwan An +3VLM Evaluation Multimodal Large Language Models
LG AI Research
Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models
Jun 14, 2026Reza Khanmohammadi, Kundan Thind, Mohammad M. GhassemiVLM Evaluation Selective Prediction
Michigan State University East Lansing, MI, USA · Henry Ford Health Detroit, MI, USA
Comparing Human Gaze and Vision-Language Model Attention in Safety-Relevant Environments
Jun 13, 2026Marta Vallejo, Siwen WangVLM Evaluation Large Vision-Language Models
School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, EH14 4AS, United Kingdom

Jun 21, 2026, cs.CVJ/K move · Enter open · S save

MMGist: A Comprehensive Multimodal Benchmark for 2027

Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong, Chengping Zhao+2

1Shanghai Jiao Tong University · 2Sichuan University

VLM Evaluation Large Vision-Language Models Multimodal Foundation Models Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and therefore fail to effectively measure multimodal understanding; 2) many items are already close to performance saturation for current LVLMs, which limits their discriminative power; 3) a small number of anomalous items affect the reliability of evaluation results. To this end, we propose MMGist, a curated benchmark that covers seven capability dimensions and contains 7,262 items. MMGist is constructed through a three-stage pipeline, which sequentially combines text-ablation filtering, cross-model saturation filtering, and anomaly detection filtering. We conduct extensive experiments on 27 leading LVLMs and compare MMGist with the raw pool of 23,250 items. The results show that MMGist preserves model rankings with high fidelity, with Spearman

ρ= 0.98

, while reducing evaluation items by 69% and improving cross-model discrimination by 78%. Further results indicate that Visual Logic remains a systematic weakness of current LVLMs, while knowledge-intensive dimensions such as Expert Knowledge dimensions remain important factors for distinguishing closed-source models from open-source models. These findings suggest that high-quality evaluation should prioritize visual dependency, discriminative power, and reliability, rather than simply pursuing benchmark scale.

Momentum

Related topics

Latest papers 658