Counterfactual Evaluation of VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 21

All topics

It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bank
Oct 7, 2026Samuel Tetteh, Cody FlemingVLM Evaluation VLM Robustness
Iowa State University Ames, Iowa, USA
Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency
Oct 6, 2026M. Moein Esfahani, Sepehr Salem, Mohammed Alser +1Physical Reasoning Counterfactual Evaluation of VLMs
Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS) Georgia State University, Georgia Institute of Technology, and Emory University, Atlanta, GA, USA · Georgia State University, Atlanta, GA, USA
Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
Oct 1, 2026Yuan Huang, Zirui Song, Xiuying ChenVLM Robustness LLM-as-a-Judge
Northeastern University · Mohamed bin Zayed University of Artificial Intelligence
Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
Sep 30, 2026Xia Hu, Brian Potetz, Chun-Ta Lu +6LLM Evaluation Counterfactual Evaluation of VLMs
Google DeepMind · Google Research · Stanford University
Distill the Visual Evidence, Not Just the Answer: Cross-World On-Policy Distillation for Vision-Language Models
Sep 30, 2026Yuanhao Sun, Huawei Ji, Jiaxin Ding +2Counterfactual Evaluation of VLMs VLM Distillation
Shanghai Jiao Tong University, Shanghai, China
It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them
Sep 29, 2026Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem +1VLM Evaluation Figurative Language Understanding
Faculty of Data and Decision Sciences, Technion
Visual sensitivity is not claim retractability: persistence-aware credit assignment for multimodal reinforcement learning
Sep 29, 2026Zhongan Bi, Kepeng Lin, Xuanang Gao +2Counterfactual Evaluation of VLMs Credit Assignment in RL
Zhejiang University · Huazhong University of Science and Technology · Shanghai Jiao Tong University
When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context
Sep 28, 2026Yuxing Cheng, Yuan Wu, Yi ChangVLM Evaluation Counterfactual Evaluation of VLMs
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University
Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning
Sep 23, 2026Yeonsung Jung, Joonhyun Jeong, Hoang Pham +4VLM Robustness Counterfactual Evaluation of VLMs
Graduate School of AI, KAIST · NAVER Cloud · Department of Computer Science and Engineering, The Ohio State University +3
LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventions
Sep 21, 2026Xinnuo Zhang, Zhike Tang, Jing Xu +2VLM Evaluation Counterfactual Evaluation of VLMs
The Hong Kong University of Science and Technology (Guangzhou)
EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing
Sep 16, 2026Sihao Ding, Santosh Vasa, Aditi Ramadwar +1VLM Evaluation VLM Robustness
Mercedes-Benz Research & Development North America
When Do Frozen VLMs Respond to Image-Free Object-Token Edits? An Answer-Key-Free Protocol and What It Reveals
Sep 3, 2026Wonbin Son, Gyumun Choi, Junil Seo +3VLM Evaluation Counterfactual Evaluation of VLMs
Changwon National University · Chung-Ang University
CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models
Aug 5, 2026De Jiang, Zhengyang Zhang, Kehong Yuan +1VLM Robustness Counterfactual Evaluation of VLMs
MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos
Jul 1, 2026Leyuan Yu, Xiao Tang, Minghao Liu +6VLM Evaluation Spatial Reasoning Benchmarks
ZODA · Tongji University · Zhejiang University +1
Ill-Posed by Design: Probing Evidence Use in VLMs
Jun 23, 2026Boaz Meivar, Shaked Perek, Shani Shvartzman +2VLM Evaluation Counterfactual Evaluation of VLMs
1Tel Aviv University · 2IBM Research
How Many Counterfactuals Does It Take? Probing VLM Hallucinations Through Circuits and Causal Effects
Jun 7, 2026Abhivansh Gupta, Simardeep Singh, Advika Sinha +2Hallucination Detection in VLMs VLM Robustness
Data Science Group, IIT Roor- kee, Roorkee, Uttarakhand, India.
VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?
Jun 5, 2026Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1VLM Evaluation Multimodal Grounding
Imperial College London
Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM Evaluation Data Visualization
University of Waterloo · Vector Institute · Layer 6 AI
CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models
May 18, 2026Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid +3VLM Evaluation Vision-Language Models
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh · KAUST
C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Driving
May 11, 2026Kefei Tian, Yuansheng Lian, Kai Yang +2VLM Robustness Counterfactual Evaluation of VLMs
College of Transportation, Tongji University, Shanghai, China · Department of Civil Engineering, Tsinghua University, Beijing 100084, China · School of Vehicle and Mobility, Tsinghua University, Beijing, China +1
Cross-Cultural Value Attribution in Large Vision-Language Models
Apr 10, 2026Phillip Howard, Xin Su, Kathleen C. FraserVLM Evaluation Counterfactual Evaluation of VLMs
Thoughtworks · University of Ottawa
Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples
Mar 2, 2026Phillip Howard, Xin Su, Kathleen C. FraserVLM Evaluation Counterfactual Evaluation of VLMs
Thoughtworks · University of Ottawa
When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs
Feb 19, 2026Yu Fang, Yuchun Feng, Dong Jing +5VLM Robustness Counterfactual Evaluation of VLMs
University of North Carolina at Chapel Hill

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bank

Samuel Tetteh, Cody Fleming

Iowa State University Ames, Iowa, USA

VLM Evaluation VLM Robustness Counterfactual Evaluation of VLMs Vision-Language Alignment

Open paper AskPDF arXiv

Frozen vision-language models increasingly provide safety signals for reinforcement learning. Their use assumes that similarity to language describing danger indicates the hazard itself. Yet policy return and collision rate cannot reveal whether a score detects hazards or responds to correlated features of the scene. VLM-based methods have reported gains in driving and safe-RL benchmarks by converting image-text similarity into rewards, costs, or confidence weights. Such signals promise to reduce reliance on manually designed feedback. They may also reflect prompt structure, embedding geometry, or camera viewpoint, leaving their safety meaning unverified. To address this gap, we present a controlled evaluation of a frozen CLIP prompt-margin safety score. We apply the score to trajectories generated by policies that never receive it, match pre-contact observations to contact-free observations with comparable hazard geometry, and vary the captions, encoder, and camera view. Across three policies, 180 episodes, and 130 isolated contact onsets, the score decreases for about twenty steps before contact. Mechanism controls indicate that the score mainly tracks resemblance to the scene shared by its captions and changes with caption separation and camera view. A constant-confidence control retains the lower catastrophe-rate point estimate, so policy gains do not establish hazard perception.

Momentum

Related topics

Latest papers 21