VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

71 papers in the last four weeks, up 69% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 658

All topics

A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision
Jun 1, 2026Stefano Samele, Eugenio Lomurno, Teodora Jovanovic +3VLM Evaluation Industrial Anomaly Detection
Politecnico di Milano, AIRLab, Milan, Italy · S&H – Software & Hardware, Milan, Italy
The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue
Jun 1, 2026Sherzod Hakimov, Mattia D'Agostini, Ivan Samodelkin +1VLM Evaluation Multimodal Grounding
Computational Linguistics, Department of Linguistics University of Potsdam, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
What to Test Next: Interpretable Coverage Gap Discovery in Driving VLMs
Jun 1, 2026Abhishek Aich, Sparsh Garg, Vijay Kumar BG +2VLM Evaluation VLMs for Autonomous Driving
NEC Laboratories, America, USA · University of California, San Diego, USA
Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing
May 31, 2026Minglai Yang, Xinyan Velocity Yu, Pengyuan Li +22VLM Evaluation Document Parsing
2077AI · University of Arizona · University of Southern California +8
ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats
May 31, 2026Shangpin Peng, Gengluo Li, Xingyu Wan +10VLM Evaluation Data Visualization
1Large Language Model Department, Tencent · 3Shenzhen Loop Area Institute · 5Hong Kong University of Science and Technology +2
Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning
May 31, 2026Garvin Guo, Yu Chen, Xiang Wang +4VLM Evaluation VLM Interpretability
Amap, Alibaba Group · Shanghai Innovation Institute
TECCI: Tricky Edits of Collected and Curated Images
May 31, 2026Aishwarya Agrawal, Roy Hirsch, Yasumasa Onoe +2VLM Evaluation Image Editing Evaluation
Google Research · Google DeepMind
HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers
May 31, 2026Issa Sugiura, Shuhei Kurita, Yusuke Oda +1VLM Evaluation Data Visualization
Institute of Science Tokyo · NII LLMC · NII
3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code
May 31, 2026Yipeng Gao, Lei Shu, Genzhi Ye +5VLM Evaluation 3D Asset Generation
Google DeepMind · University of Southern California · Google Research
Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated
May 30, 2026Rashid MushkaniVLM Evaluation Inter-Rater Reliability
Université de Montréal, Montréal, Québec, Canada · Mila – Québec AI Institute, Montréal, Québec, Canada
RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenes
May 30, 2026Leyi Wu, Yifan Zhao, Jinjie Zhang +11VLM Evaluation VLM Robustness
HKUST(GZ) · Knowin · HKUST
An Attribute-Based Measure of Video Complexity
May 30, 2026Aditya Sarkar, Yi Li, Zihao Wang +6VLM Evaluation Video QA
UMIACS-University of Maryland College Park · University of California San Diego · Qualcomm +2
Through the PRISM: Principle-Aware, Interpretable, and Multi-Scale Evaluation of Visual Designs
May 30, 2026Mona Gandhi, KJ Joseph, Srinivasan Parthasarathy +1VLM Evaluation VLM Interpretability
Ohio State University · Adobe Research
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models
May 29, 2026Olaf Dünkel, Basavaraj Sunagad, Haoran Wang +3VLM Evaluation Semantic Correspondence
Max Planck Institute for Informatics, SIC · CISPA Helmholtz Center for Information Security · University of Freiburg
EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision
May 29, 2026Rosario Forte, Giuseppe Lando, Antonino FurnariVLM Evaluation Streaming Video Understanding
Department of Mathematics and Computer Science University of Catania
Vision-Language Models Suppress Female Representations Under Ambiguous Input
May 29, 2026Arnau Marin-Llobet, Simon Henniger, Mahzarin R. BanajiVLM Evaluation VLM Interpretability
School of Engineering and Applied Sciences · Department of Psychology
Can You Trust What You See? Human and AI Detection of Synthetic Legal Evidence
May 29, 2026Jinzhe Tan, Ali Ekber Cinar, Karim BenyekhlefVLM Evaluation AI-Generated Image Detection
Cyberjustice Laboratory, Faculty of Law, Universit´e de Montr´eal · Faculty of Law, McGill University
"Înţelegi Româneşte?'' A Recipe for Romanian Vision-Language Models
May 29, 2026Mihai Masala, Marius Leordeanu, Mihai Dascalu +1VLM Evaluation Vision-Language Models
National University of Science and Technology POLITEHNICA Bucharest, Romania
A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation
May 29, 2026Yi Zhao, Siqi Wang, Zhe Hu +2VLM Evaluation LLM-as-a-Judge
Department of Computing, The Hong Kong Polytechnic University
FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection
May 29, 2026Paramananda Bhaskar, Naquee Rizwan, Daksh Jogchand +2VLM Evaluation VLM Adaptation
1Indian Institute of Technology (IIT), Kharagpur · 2Microsoft
Can Aerial VLA Models Cooperate? Evaluating Closed-Loop Air-Ground Coordination with CARLA-Air
May 29, 2026Tianle Zeng, Yanci Wen, Xueang Yu +1VLM Evaluation Aerial Robotics
Southern University of Science and Technology · Fudan University
Does Visual Information Play a Decisive Role in Vision-Language-Action Model Driving Behavior?
May 29, 2026Jingtao He, Hongliang Lu, Xiaoyun Qiu +2VLM Evaluation Autonomous Driving
Intelligent Transportation Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustness
May 29, 2026Yusheng He, Jizhe Zhou, Xia Du +3VLM Evaluation VLM Robustness
School of Computer Science, Engineering Research Center of Machine Learning and Industry Intelligence, Sichuan University, Chengdu, 610020, China · School of Computer and Information Engineering, Xiamen University of Technology, Xiamen, 361000, China · Department of Electrical and Computer Engineering, University of Hong Kong, Pok Fu Lam, Hong Kong, China +1
StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning
May 29, 2026Xixiang He, Baiqi Wu, Xingming Li +4VLM Evaluation Visual Reasoning
1National University of Defense Technology · 2Zhejiang University · 3Intelligent Game and Decision Lab
Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models
May 29, 2026Yijie Tong, Yifan Hou, Shaobo Cui +2VLM Evaluation Vision-Language Models
ETH Zürich · Shanghai Jiao Tong University · EPFL
Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?
May 28, 2026Yue Zhang, Zun Wang, Han Lin +3VLM Evaluation Spatial Reasoning Benchmarks
UNC Chapel Hill · Google Research
Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
May 28, 2026Cheolhong Min, Jaeyun Jung, Daeun Lee +5VLM Evaluation Spatial Reasoning Benchmarks
Seoul National University · The Ohio State University · NVIDIA
DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?
May 28, 2026Linhao Zhang, Aiwei Liu, Yuan Liu +1VLM Evaluation Fine-Grained Visual Perception
WeChat AI, Tencent Inc.
World Models in Words: Auditing Physical State-Transition Commitments in Vision-Language Models
May 28, 2026Emmanuelle BourigaultVLM Evaluation VLM Reasoning
University of Oxford
Comparative Evaluation of Machine Translation Systems on Images with Text
May 28, 2026Blai Puchol, Sergio Gómez González, Miguel Domingo +1VLM Evaluation Machine Translation
PRHLT Research Center - Universitat Polit`ecnica de Val`encia, Camino de Vera, s/n - Valencia · ValgrAI - Valencian Graduate School and Research Network for Artificial Intelligence, Camino de Vera s/n

Jun 1, 2026, cs.CVJ/K move · Enter open · S save

A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

Stefano Samele, Eugenio Lomurno, Teodora Jovanovic, Sanjay Shivakumar Manohar+2

Politecnico di Milano, AIRLab, Milan, Italy · S&H – Software & Hardware, Milan, Italy

VLM Evaluation Industrial Anomaly Detection Multimodal Anomaly Detection Zero-Shot Anomaly Detection Anomaly Detection

Open paper AskPDF arXiv

Industrial anomaly detection has historically been a unimodal task. Recent multimodal vision-language models have produced systems that admit textual input alongside the image and are presented as enabling text-guided zero- and few-shot inspection. Yet these methods are evaluated with protocols inherited from unimodal benchmarks that hold the textual condition constant and therefore cannot measure whether language conditions the decision; whether reported gains reflect text guidance or strong pretrained visual features remains open. We introduce Text-Guided Anomaly Detection (TGAD), a structured benchmark that progressively increases the functional role of language across three scenarios: a controlled prompt-sensitivity setting on MVTec AD; a component-tagged extension of MVTec AD that requires the model to restrict its assessment to an instructed part; and the new Assembled Panel Dataset (APD), a realistic industrial setting that requires both defect-type and component-location knowledge. We evaluate one representative model per paradigm: generative large vision-language, training-free discriminative, and embedding-adaptive discriminative. In all three, the textual interface conditions the decision only superficially: prompt content is absorbed unless the object noun is removed (the generative model's I-AUROC drops from 97.4 to 82.6); component-level instructions do not constrain the decision once defects outside the instructed part are admitted as normal (from 90.3 to 66.3); and when both combine on APD, image-level discrimination collapses below the MVTec level, in one case below chance (71.2, 50.5, 31.5). These results suggest that standard benchmarks overstate the text-guided capabilities of current multimodal anomaly detection systems, and that a protocol of this kind is a prerequisite for models that can be reliably controlled through language for industrial deployment.

Momentum

Related topics

Latest papers 658