VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

71 papers in the last four weeks, up 69% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 649

All topics

ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making
Aug 5, 2026Ningxin Pan, Hanyu Li, Yehui TangVLM Evaluation Visually Grounded Reasoning
Peking University, Beijing, China.
BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models
Aug 4, 2026Sadab Shiper, Tawsif Tashwar Dipto, Mir Md Inzamam +1VLM Evaluation Scene Text Recognition
Islamic University of Technology, Bangladesh · BRAC University, Bangladesh
Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation
Aug 4, 2026Chunlin Liu, Junnian Chen, Haitong Jiang +7VLM Evaluation VLM Unlearning
1Shenzhen University · 2Hebei University of Technology · 3Ludwig Maximilian University of Munich +1
COMEX: A Composition-Grounded Benchmark and Learning Framework for Explainable Aesthetic Image Cropping
Aug 4, 2026Rui Yang, Wei Zhou, Dingyong Gou +5VLM Evaluation Image Editing
State Key Laboratory of Mobile Network and Mobile Multimedia Technology, ZTE Corporation · School of Data Science and Institute of Artificial Intelligence, Chang’an University
Confident but Unreliable: A Behavioral Safety Audit of Vision-Language Models on Brain MRI
Aug 3, 2026Amir Sabbaghziarani, Mohammadsajad Abavisani, Sergey PlisVLM Evaluation VLM Hallucination
TReNDS Center Atlanta, GA, USA
Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search
Aug 3, 2026Han Wang, Alex Whitworth, Pak Ming Cheung +5VLM Evaluation Automated Evaluation
Pinterest · San Francisco, CA, USA
A General-Purpose VLM Can Teach an Astronomy Foundation Model to Better Recognize Galaxy Morphology
Aug 3, 2026Dichang Zhang, Jiaqi Deng, Yixuan Shao +7VLM Evaluation AI-Assisted Scientific Research
Department of Computer Science Stony Brook University Stony Brook, New York, USA · School of Computer Science University of Technology Sydney Sydney, New South Wales, Australia · Futurewei Technologies San Jose, California, USA
Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction
Aug 3, 2026Priyashree Roy, Sujitha Martin, Mohammad Rostami +6VLM Evaluation Document Understanding
Amazon Web Services
Can Urban Blight Be Accessed with Vision-language Models: A Case Study in Detroit
Aug 3, 2026Xiaohao Yang, Aohua Tian, Derek Van Berkel +2VLM Evaluation Vision-Language Models
School for Environment and Sustainability; School of Information University of Michigan, Ann Arbor, USA
Evaluating VLMs on Multimodal Aristotelian Persuasion Tasks
Aug 2, 2026Khondoker Ittehadul IslamVLM Evaluation Vision-Language Models
Saarland University · Department of Language Science & Technology
It's the Decoding Format, Not the Perturbation: Auditing Consistency-Based Selection for Vision-Language Test-Time Scaling
Aug 2, 2026Puzhuo Zheng, Hasan KurbanVLM Evaluation Self-Consistency Decoding
College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking
Aug 2, 2026Timothee Mickus, Claudio Savelli, Eduardo Calò +10VLM Evaluation Hallucination Detection in VLMs
University of Helsinki
Cross-Task Dissociation in Frontier Vision-Language Model Theory of Mind
Jul 31, 2026Kejia Zhang, Youran Sun, Chugang Yi +1VLM Evaluation Theory of Mind
University of Maryland, College Park
Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks
Jul 31, 2026Rupak Sarkar, Neha Srikanth, Saloni Gupta +3VLM Evaluation Vision-Language Models
University of Maryland, College Park · Army Research Lab
Is It Time for the Renaissance of Salient Object Detection in the Era of MLLMs?
Jul 31, 2026Wenzhuo Zhao, Xiuzhi Li, Zhongkuan Mao +6VLM Evaluation Salient Object Detection
College of Computer Science, Sichuan University, China · National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, China · School of Information and Communication Engineering, University of ElectronicScience and Technology of China, China
SciFigPlag-Bench: A Benchmark for Provenance-Aware Scientific Figure Plagiarism Detection
Jul 31, 2026Zhiying Cui, Minghao Yang, Linlin Gao +2VLM Evaluation Image Forgery Detection
1Ningbo University · 2Hokkaido University · 3IBM Research
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
Jul 30, 2026Qiushi Sun, Kanzhi Cheng, Yian Wang +20VLM Evaluation Computer-Use Agent Benchmarks
The University of Hong Kong · Nanjing University · National University of Singapore +4
Scaling Vision-Language Models Is Not Enough to Mitigate Bias
Jul 30, 2026Ioannis Sarridis, Ioannis Kompatsiaris, Symeon PapadopoulosVLM Evaluation VLM Robustness
Information Technologies Institute, CERTH Thessaloniki, Greece
MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes
Jul 30, 2026Weihang Wang, Kainan Tu, Jielei Zhang +9VLM Evaluation Vision-Language Models
1Bilibili · 2Fudan University · 3Peking University
Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
Jul 30, 2026Liangjie Zhao, Jiaqing Lyu, Kexin Tang +5VLM Evaluation Visual Reasoning
Adelaide University · Tsinghua University · Amap, Alibaba Group +3
JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles
Jul 30, 2026Shawn Li, Wei Yang, Jike Zhong +11VLM Evaluation Spatial Reasoning Benchmarks
University of Southern California · National University of Singapore · Adobe Research +3
Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers
Jul 30, 2026Fexiang Liu, Shiye Wang, Qiang Qiu +1VLM Evaluation Multimodal Large Language Models
1State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences
HumanCLAW: Can Vision-Language Models Act Through a Body?
Jul 29, 2026Li Siyao, Jiawei Gu, Shuai Liu +15VLM Evaluation Embodied AI
Meta · University of Washington · Nanyang Technological University +2
Visual Credit Audit for Multimodal Spatial Reasoning
Jul 29, 2026Feixiang Liu, Qiang Qiu, Lanbo Sun +3VLM Evaluation Spatial Reasoning Benchmarks
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference
Jul 29, 2026Feixiang Liu, Qiang Qiu, Hao Zhang +1VLM Evaluation Efficient VLM Inference
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences
OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation
Jul 29, 2026Kaiyu Li, Zepeng Xin, Zixuan Jiang +4VLM Evaluation Open-Vocabulary Object Detection
Xi’an Jiaotong University
Hearsay: Vision-Language Medical Diagnoses Without an Image
Jul 29, 2026Siddharth VohraVLM Evaluation Medical Image Analysis
Carnegie Mellon University Pittsburgh, PA, USA · Amazon Web Services AI Native Pittsburgh, PA, USA
Level, Sharpness, and Corpus: Why Zero-Shot OOD Detector Rankings Do Not Transfer
Jul 29, 2026Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Stephen Gould +1VLM Evaluation Distribution Shift Detection
University of Adelaide · 3Naval Group Pacific · 2Australian National University
An evolutionary model of animats with VLM-based subjective evaluation
Jul 29, 2026Shota Miyazaki, Takaya Arita, Reiji SuzukiVLM Evaluation Robotics
Graduate School of Informatics, Nagoya University, Japan
Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
Jul 28, 2026Jiaang Li, Chengzu Li, Zhaochong An +4VLM Evaluation Multimodal Large Language Models
University of Copenhagen · University of Cambridge · 3ETH Zürich +1

Aug 5, 2026, cs.CVJ/K move · Enter open · S save

ComplexityWorld: Benchmarking Vision-Language Models on Verifiable Visual Decision Making

Ningxin Pan, Hanyu Li, Yehui Tang

Peking University, Beijing, China.

VLM Evaluation Visually Grounded Reasoning Constraint Satisfaction

Open paper AskPDF arXiv

Vision-language models (VLMs) have made rapid progress in visual perception and increasingly support real-world tasks that depend on images. Many such tasks, however, require more than rec- ognizing what an image contains: a model must use visual evidence to make a complete decision whose parts jointly satisfy global constraints. We introduce COMPLEXITYWORLD, a benchmark of 390 tasks across 39 domain-inspired visual worlds and 29 decision categories. Each task is generated from a hidden structured specification, rendered as a visual scene, and scored by an exe- cutable verifier that accepts any feasible solution. Under direct inference, all evaluated models ex- cept GPT-5.6-Sol remain below 40% verifier ac- ceptance rate (VAR), while GPT-5.6-Sol reaches 75.6%. Performance improves substantially when the same decision information is made explicit in structured form, yet varies sharply across equiva- lent visual presentations. Agent scaffolds provide smaller, model-dependent gains. Together, these results reveal a persistent visual-to-decision bot- tleneck that additional inference alone does not remove.

Momentum

Related topics

Latest papers 649