VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

70 papers in the last four weeks, up 67% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 649

All topics

DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents
May 10, 2026Yixiong Chen, Wenjie Xiao, Pedro R. A. S. Bassi +7VLM Evaluation Tool Use in VLMs
Johns Hopkins University · University of Bologna · Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia +3
SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy
May 10, 2026Ismael Elsharkawi, Ahmed Sait, Silvio Giancola +3VLM Evaluation Sports Video Analysis
Department of Computer Science and Engineering, The American University in Cairo · Image And Visual Understanding Lab (IVUL), KAUST
From Pixels to Concepts: Do Segmentation Models Understand What They Segment?
May 10, 2026Shuang Liang, Zeqing Wang, Yuxian Li +2VLM Evaluation Image Segmentation
Department of Electrical and Computer Engineering, The University of Hong Kong · CASIC · School of Computer Science and Engineering, Sun Yat-sen University +1
Language-Conditioned Visual Grounding with CLIP Multilingual
May 9, 2026J. de Curtò, Mauro Liz, I. de ZarzàVLM Evaluation Multilingual VLM Evaluation
Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, Barcelona, Spain · Estudis d’Informàtica, Multimèdia i Telecomunicació, Universitat Oberta de Catalunya, Barcelona, Spain · Escuela T´ecnica Superior de Ingenier´ıa (ICAI), Universidad Pontificia Comillas, Madrid, Spain +2
Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models
May 9, 2026Tri Cao, Khoi Le, Thong Nguyen +7VLM Evaluation Object Hallucination in VLMs
National University of Singapore · VinUniversity · Nanyang Technological University
Illusion-Aware Visual Preprocessing and Anti-Illusion Prompting for Classic Illusion Understanding in Vision-Language Models
May 9, 2026Junli Zha, Jiahui Wang, Xinkai Lu +1VLM Evaluation VLM Robustness
1SF Technology Co., Ltd.
PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models
May 9, 2026Jiahui Guang, Zexun Zhan, Zhenlin Xu +5VLM Evaluation VLM Unlearning
Harbin Institute of Technology, Shenzhen, China · Sichuan University, Chengdu, China · Harbin Institute of Technology, Weihai, China +3
Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models
May 9, 2026Mashrafi Monon, Umaima Rahman, Asif Hanif +2VLM Evaluation 3D Spatial Reasoning
Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · New York University Abu Dhabi, Abu Dhabi, UAE
NICE FACT: Diagnosing and Calibrating VLMs in Quantitative Reasoning for Kinematic Physics
May 8, 2026Jian Lan, Zhicheng Liu, Xinpeng Wang +5VLM Evaluation Physical Reasoning
University of Munich (LMU), Germany 2Munich Center of Machine Learning3 Sichuan University
MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence
May 8, 2026Hanqi Jiang, Junhao Chen, Mingyu Kang +12VLM Evaluation VLM Robustness
University of Georgia · Chungbuk National University · Chungnam National University Hospital +5
RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderation
May 8, 2026Zhifeng Lu, Dianyuan Wang, Yuhu Shang +1VLM Evaluation Logical Reasoning
Beijing University of Posts and Telecommunications, Beijing, China
Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric
May 7, 2026Ying Gu, Mei Chee Leong, Hui Li Tan +3VLM Evaluation Large Vision-Language Models
Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR), Singapore
CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs
May 7, 2026Zhengru Fang, Yanan Ma, Yu Guo +5VLM Evaluation Chest X-Ray Classification
City University of Hong Kong · Tsinghua Shenzhen International Graduate School
iTRIALSPACE: Programmable Virtual Lesion Trials for Controlled Evaluation of Lung CT Models
May 7, 2026Fakrul Islam Tushar, Umme Hafsa Momy, Joseph Y. Lo +1VLM Evaluation Radiology VLMs
Department of Radiology and Imaging Sciences, University of Arizona · Department of Biomedical Engineering, Florida International University · Center for Virtual Imaging Trials, Department of Radiology, Duke University Medical Center
How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study
May 6, 2026Junran Wang, Xinjie Shen, Zehao Jin +1VLM Evaluation Vision-Language Models
Georgia Institute of Technology
StableI2I: Spotting Unintended Changes in Image-to-Image Transition
May 6, 2026Jiayang Li, Shuo Cao, Xiaohui Li +6VLM Evaluation Image Editing Evaluation
Peking University · Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China +1
Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits
May 5, 2026Logan Mann, Ajit Saravanan, Ishan Dave +4VLM Evaluation VLM Interpretability
UC Santa Barbara · UC Berkeley · NVIDIA +2
Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks
May 5, 2026JuneHyoung Kwon, MiHyeon Kim, Eunju Lee +3VLM Evaluation VLM Unlearning
Department of Artificial Intelligence, Chung-Ang University · KT Corporation · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models
May 5, 2026JuneHyoung Kwon, JungMin Yun, YoungBin KimVLM Evaluation VLM Unlearning
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
MHPR: Multidimensional Human Perception and Reasoning Benchmark for Large Vision-Languate Models
May 5, 2026Kangkang Wang, Qinting Jiang, Wanping Zhang +2VLM Evaluation Vision-Language Models
AI Compute Group, Baidu · Tsinghua University
VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition
May 4, 2026Tanush Yadav, Mohammadreza Salehi, Jae Sung Park +6VLM Evaluation VLM Adaptation
University of Washington · Stanford University · Allen Institute for AI
Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting
May 4, 2026Giacomo Pacini, Luca Ciampi, Nicola Messina +3VLM Evaluation VLM Robustness
Institute of Information Science and Technologies of the National Research Council (ISTI-CNR), Via G. Moruzzi 1, 56124, Pisa, Italy · University of Pisa - Department of Information Engineering, Via G. Caruso 16, 56122 Pisa, Italy
Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score
May 4, 2026Abdullah Ahmad Khan, Hamid Laga, Ferdous SohelVLM Evaluation VLM Unlearning
Murdoch University
SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?
May 3, 2026Jongmin Shin, Ka Young Kim, Eunki Cho +2VLM Evaluation Medical VQA
Department of Surgery, Samsung Medical Center, Seoul 06351, Republic of Korea · Kyung Hee University, Yongin 17104, Republic of Korea
SF20K Competition 2025: Summary and findings
May 2, 2026Ridouane Ghermi, Xi Wang, Vicky Kalogeiton +1VLM Evaluation Video Understanding
VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models
May 2, 2026Pang Liu, Yingjie LaoVLM Evaluation VLM Robustness
Department of Electrical and Computer Engineering, Tufts University
VISTA: Video Interaction Spatio-Temporal Analysis Benchmark
May 2, 2026Alejandro Aparcedo, Akash Kumar, Aaryan Garg +5VLM Evaluation Video Understanding
University of Central Florida · BITS Pilani · Ho Chi Minh City University of Science +1
Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification
May 1, 2026Charles Weng, Dingwen Li, Alexander MartinVLM Evaluation Zero-Shot Learning
Johns Hopkins University · Independent
Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception
Apr 30, 2026Neemias B da Silva, Rodrigo Minetto, Daniel Silver +1VLM Evaluation Personality Modeling in Language Models
Universidade Tecnol´ogica Federal do Paran´a (UTFPR), Curitiba, Brazil · University of Toronto, Toronto, Canada
ClimateVID -- Social Media Videos Analysis and Challenges Involved
Apr 30, 2026Shiqi Xu, Moritz Burmester, Katharina Prasse +3VLM Evaluation Social Media Analysis
University of Mannheim · Technical University of Munich · University of Mannheim, Max-Planck-Institute for Informatics, Saarland Informatics Campus

May 10, 2026, cs.CVJ/K move · Enter open · S save

DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents

Yixiong Chen, Wenjie Xiao, Pedro R. A. S. Bassi, Boyan Wang+6

Johns Hopkins University · University of Bologna · Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia+3

VLM Evaluation Tool Use in VLMs Medical VQA AI Agent Benchmarks Medical Image Benchmarks

Open paper AskPDF arXiv

Medical vision-language models (VLMs) and AI agents have made significant progress in learning to analyze and reason about clinical images. However, existing medical visual question answering (VQA) benchmarks collapse model capabilities into a single accuracy score, obscuring where and why models fail. We propose DeepTumorVQA, a hierarchical benchmark that follows the multi-stage evidence chain in tumor diagnosis and decomposes 3D CT reasoning into four stages: recognition, measurement, visual reasoning, and medical reasoning. Higher-level questions remain independently scorable, while their ground-truth evidence chains are defined over lower-level primitives. The benchmark contains 476K questions across 42 clinical subtypes on 9,262 3D CT volumes. In addition to a direct reasoning mode for VLMs, DeepTumorVQA provides tool-interaction environments for agent evaluation, where a model can call external tools, including segmentation models, measurement programs, and medical knowledge modules, before answering the question. Evaluating over 30 model configurations, we find that reliable quantitative measurement is the primary bottleneck, making later-stage visual and medical reasoning harder for VLMs, while tool augmentation substantially mitigates this issue. When tools are available, leveraging medical knowledge and tools to reason about medical images becomes a new challenge. We further show that ground-truth step-by-step tool-use traces from DeepTumorVQA can supervise agents and reduce tool-use and reasoning failures. This stage-wise progression from recognition to measurement to visual and medical reasoning provides a concrete roadmap for future medical VLM and AI agent studies. All data and code are released at https://github.com/Schuture/DeepTumorVQA.

Momentum

Related topics

Latest papers 649