VLM Evaluation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

71 papers in the last four weeks, up 69% on the four weeks before. 0.7% of all new papers.

Jul 13Week of Sep 28

Latest papers 658

All topics

When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Training
Jun 12, 2026Jianzhe LinVLM Evaluation VLM Adaptation
MetaAI · Work done at Meta
What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective
Jun 12, 2026Jiazhen Huang, Xiao Chen, Zhiming Liu +3VLM Evaluation Test-Time Adaptation
Tsinghua University · Shenzhen Technology University
Position: The Systemic Lack of Agency in Visual Reasoning
Jun 11, 2026Yizhao Huang, Haoyang Chen, Shiqin Wang +6VLM Evaluation VLM Reasoning
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. +2
Iterative Visual Thinking and the Self-Correction Mirage in VLM Grounding
Jun 11, 2026Animesh Tripathy, Aswanth KrishnanVLM Evaluation Vision-Language Grounding
QpiAI India Pvt. Ltd.
RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation
Jun 11, 2026Dayu Xia, Yue Shi, Yao Mu +7VLM Evaluation Robotic Manipulation
Shanghai AI Laboratory · Zhejiang University · Shanghai Jiao Tong University +2
SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images
Jun 10, 2026Ruotian Zhang, Xiaoxiao Sun, Junzhe Huang +2VLM Evaluation VLM Hallucination
Zhejiang University · The University of Queensland
Task-Aligned Stability Analysis of Vision-Language Models for Autonomous Driving Hazard Detection
Jun 10, 2026Everett RichardsVLM Evaluation VLM Robustness
Department of Computer Science, San Diego State University.
4DP-QA: Scalable QA for 4D Perception in Vision Language Models
Jun 10, 2026Seokju Cho, Abhishek Badki, Hang Su +5VLM Evaluation VLM Reasoning
1NVIDIA · 2Yale University · *Work done during internship at NVIDIA. +1
Towards Fully Automated Exam Grading: Fairness-Aware Recognition of Handwritten Answers with Foundation Models
Jun 9, 2026Hartwig GrabowskiVLM Evaluation Algorithmic Fairness
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany.
Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models
Jun 9, 2026Peiqi Jia, Haonan Jia, Ziqi Miao +3VLM Evaluation Personality Modeling in Language Models
Xi’an Jiaotong University · Beihang University
Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation
Jun 9, 2026Syed Wasiq, Syed Mohamad Tawseeq, Yashwant Pravinrao Bangde +1VLM Evaluation Visual Question Answering
Indian Institute of Technology Kharagpur
KCSAT-ML: Probing Reasoning Models with Nationwide-Cohort Human Difficulty
Jun 9, 2026Sanghee Park, Geewook Kim, Kee-Eung KimVLM Evaluation Mathematical Reasoning Benchmarks
NAVER Cloud AI · KAIST AI
Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
Jun 9, 2026Pratham Singla, Shivank Garg, Vihan Singh +1VLM Evaluation Vision-Language Models
Lossfunk · Indian Institute of Technology Roorkee · Raeth AI
DB-3DME: From Dataset to Benchmark for Human-aligned Automatic 3D Mesh Evaluation
Jun 8, 2026Nanshan Jia, Zhenyu Zhao, Junbo Peng +3VLM Evaluation 3D VLMs
University of California, Berkeley · Roblox Corporation
A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks
Jun 8, 2026Bruce Changlong Xu, Lan Wu, Alexander RyuVLM Evaluation Medical VQA
Stanford University Stanford, CA, USA · University of California, Berkeley Berkeley, CA, USA · Mayo Clinic Rochester, MN, USA
OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics
Jun 8, 2026Mingxian Lin, Shengju Qian, Yuqi Liu +9VLM Evaluation Game-Playing Agents
1The University of Hong Kong · 2LIGHTSPEED · 3The Chinese University of Hong Kong +1
Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Driving
Jun 8, 2026Yimu Wang, Yee Man Choi, Barry Zhang +3VLM Evaluation Autonomous Driving Benchmarks
University of Waterloo
Guide Me Out: A Framework to Benchmark VLM Operators Communication in Crisis Scenarios
Jun 8, 2026Giacomo Gonella, Stefano Menini, Marco GueriniVLM Evaluation Disaster Response
Fondazione Bruno Kessler, Italy · University of Trento, Italy
Zero-Shot Semantic Re-Identification for Autonomous Driving: A VLM Baseline Study
Jun 8, 2026Eduardo Borges, Manuel Abreu, Luís Garrote +1VLM Evaluation Vision-Language Models
Institute of Systems and Robotics, University of Coimbra, Coimbra, Portugal
ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China
Jun 8, 2026Yi Zhang, Bolei Ma, Yong Cao +3VLM Evaluation Visual Question Answering
1LMU Munich · 2FAU Erlangen-Nuremberg · 3Munich Center for Machine Learning +4
Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?
Jun 8, 2026Yizheng Sun, Mochuan Zhan, Yanan Ma +10VLM Evaluation VLM Robustness
University of Manchester · Marex · Imperial College London
Harnessing Streaming Video in the Wild
Jun 7, 2026Dingyu Yao, Shuhuan Gu, Qingyi Si +8VLM Evaluation Streaming Video Understanding
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problems
Jun 6, 2026Muhammad Falensi Azmi, Ikhlasul Akmal Hanif, Vallerie Alexandra Putra +3VLM Evaluation VLM Robustness
†Independent Researcher · ‡MBZUAI · §Binus University +1
VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
Jun 5, 2026Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1VLM Evaluation Multimodal Grounding
Imperial College London
The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
Jun 5, 2026Lujun Li, Lama Sleem, Niccolo Gentile +4VLM Evaluation Large Vision-Language Models
University of Luxembourg · Foyer S.A. · Université Paris-Saclay
MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models
Jun 5, 2026Rishabh Makwana, Mamta, Deeksha Varshney +1VLM Evaluation VLM Robustness
Dwarkadas Jivanlal Sanghvi College of Engineering, Mumbai, India · King’s College London, London, UK · Indian Institute of Technology Jodhpur, Jodhpur, India
Diagnosing Visual Ignorance in Vision-Language Models
Jun 5, 2026Runyu Zhou, Qi Zhang, Yisen WangVLM Evaluation VLM Robustness
Peking University
Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception
Jun 4, 2026Qian Zhang, Michal Golovanevsky, Fulvio Domini +1VLM Evaluation Vision-Language Models
Brown University Computer Science · Harvard University · Brown University Cognitive Science
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Jun 4, 2026Tianyi Tang, Zhuoyi Lin, Zeyu Feng +4VLM Evaluation VLM Reasoning
CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore · I2R, Agency for Science, Technology and Research (A*STAR) Singapore · Nanyang Technological University Singapore
Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
Jun 4, 2026XiuYu Zhang, Junfeng Fang, Zhenkai LiangVLM Evaluation Latent Visual Reasoning
National University of Singapore

Jun 12, 2026, cs.CRJ/K move · Enter open · S save

When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Training

Jianzhe Lin

MetaAI · Work done at Meta

VLM Evaluation VLM Adaptation Direct Preference Optimization VLM Reasoning Negative Transfer

Open paper AskPDF arXiv

Verifier reliability is not portable across tasks. A verifier-guided self-DPO pipeline with genuine held-out gains on MathVista (+9.6 points on self-training data, +8.0 held out) can be harmful on MMMU. The failure is invisible from the target-task self-training signal: over six learner-verifier configurations, MMMU self-training performance still improves by +3.52 points while held-out performance drops by 1.42 points. We call this silent negative transfer: a verifier validated as useful before can still show signatures of successful self-training after its induced update stops transferring to unseen data. Downstream failure alone cannot diagnose this: poor generalization may mean the verifier induces a wrong learning direction, or that the correct direction itself does not generalize beyond the self-training distribution. We separate these effects via gradient fidelity F, the alignment between verifier-induced and correct training directions, and gradient transferability T, the alignment between correct and held-out directions, giving a conservative Safe-Transfer Margin: positive first-order held-out alignment is certified when arccos(F) + arccos(T) < pi/2. We then introduce Asymmetric Acceptance Gating (AAG), a precision-first filter selecting preference pairs by the verifier's absolute confidence in the response receiving the positive update. On MMMU, AAG raises fidelity from 0.29 to 0.42. In a same-cell analysis, the correct direction stays positively aligned with the held-out direction (T = 0.31), yet raw verification rotates the induced update to negative held-out alignment (-0.13); AAG restores positive alignment (+0.22). Across all six MMMU configurations, AAG beats raw verifier-guided training. Verifier reliability thus means not that it worked before, but that it induces the right update on the task at hand and that this update stays useful beyond the data that generated it.

Momentum

Related topics

Latest papers 658