Counterfactual Evaluation of VLMs

VLM: Vision-Language Model

Ask about this topic

Overview

Counterfactual evaluation studies whether vision-language models use visual evidence by changing an image, video, or context in a controlled way and checking whether the answer responds as expected. Common approaches pair factual and edited examples, use image ablations or misleading text, and measure answer flips, consistency, grounding, and confidence; some also trace internal activations or train models on evidence-dependent changes. These tests are used to diagnose hallucination, spatial and causal reasoning, chart interpretation, medical grounding, and safety decisions. A central challenge is separating genuine evidence use from correct guesses, language shortcuts, and unperceived edits, so robust evaluation needs multiple perturbations and controls for perceptibility and blind baselines.

Latest papers

CardsList

61 papers20 papers in the last four weeks, up 54% on the four weeks before. 0.1% of all new papers.

  1. Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models

    Sep 6, 2026Yangyang Xie, Ke Hao, Jiaqi Liu +2Confidence Estimation for VLMsVLM Calibration

    Shanghai Jiao Tong University, Shanghai, China · Medical Image Insights Co. Ltd., Shanghai, China

  2. One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation

    Sep 5, 2026Rasul Khanbayov, Mariam Sohail, Ahmed Abdala +1Counterfactual Evaluation of VLMs

    College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar

  3. When Do Frozen VLMs Respond to Image-Free Object-Token Edits? An Answer-Key-Free Protocol and What It Reveals

    Sep 3, 2026Wonbin Son, Gyumun Choi, Junil Seo +3VLM EvaluationCounterfactual Evaluation of VLMs

    Changwon National University · Chung-Ang University

  4. Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails

    Sep 2, 2026Suyoung Lee, Myungsub ChoiCultural Safety Evaluation of VLMsVLM Evaluation

    Tynapse, Seoul, Republic of Korea

  5. Don't Just Look, Intervene: Perturbation Based Region Labeling for VQA Images

    Sep 1, 2026Marko Jojic, Zhaonan Li, Ben ZhouVision-Language GroundingVisual Question Answering

    Arizona State University

  6. GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models

    Aug 26, 2026Yiqun Sun, Junyu Chen, Pengfei Wei +1Activation SteeringCounterfactual Evaluation of VLMs

    Magellan Technology Research Institute (MTRI) · National University of Singapore

  7. EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

    Aug 24, 2026Xuetong Li, Gaofeng LiuCultural Safety Evaluation of VLMsVLM Safety

    School of Mathematical Sciences, MOE Key Lab of Scientific and Engineering Computing, Shanghai Center for Applied Mathematics, Shanghai Jiao Tong University · Department of Automation, Shanghai Jiao Tong University

  8. Fidelity Preference, Not Demographic Preference: A Pixel-Level Attribute-Sensitivity Audit of Image Aesthetic/Preference Scorers

    Aug 17, 2026Mingyang XuAlgorithmic BiasAlgorithmic Fairness

    Peking University Beijing, China

  9. Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

    Aug 12, 2026Mizanur Rahman, Huan Wu, Arash Asgari +2Cultural Bias in VLMsVLM Evaluation

    York University · Vector Institute · Connected Minds

  10. CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models

    Aug 5, 2026De Jiang, Zhengyang Zhang, Kehong Yuan +1VLM RobustnessCounterfactual Evaluation of VLMs

  11. MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos

    Jul 1, 2026Leyuan Yu, Xiao Tang, Minghao Liu +6VLM EvaluationSpatial Reasoning Benchmarks

    ZODA · Tongji University · Zhejiang University +1

  12. Ill-Posed by Design: Probing Evidence Use in VLMs

    Jun 23, 2026Boaz Meivar, Shaked Perek, Shani Shvartzman +2VLM EvaluationCounterfactual Evaluation of VLMs

    1Tel Aviv University · 2IBM Research

  13. How Many Counterfactuals Does It Take? Probing VLM Hallucinations Through Circuits and Causal Effects

    Jun 7, 2026Abhivansh Gupta, Simardeep Singh, Advika Sinha +2Hallucination Detection in VLMsVLM Robustness

    Data Science Group, IIT Roor- kee, Roorkee, Uttarakhand, India.

  14. VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?

    Jun 5, 2026Didi Zhu, Changrui Chen, Stefanos Zafeiriou +1VLM EvaluationMultimodal Grounding

    Imperial College London

  15. Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

    May 26, 2026Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1VLM EvaluationData Visualization

    University of Waterloo · Vector Institute · Layer 6 AI

  16. CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

    May 18, 2026Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti +4VLM EvaluationVision-Language Models

    King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh

  17. C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Driving

    May 11, 2026Kefei Tian, Yuansheng Lian, Kai Yang +2VLM RobustnessCounterfactual Evaluation of VLMs

    College of Transportation, Tongji University, Shanghai, China · Department of Civil Engineering, Tsinghua University, Beijing 100084, China · School of Vehicle and Mobility, Tsinghua University, Beijing, China +1

  18. Cross-Cultural Value Attribution in Large Vision-Language Models

    Apr 10, 2026Phillip Howard, Xin Su, Kathleen C. FraserVLM EvaluationCounterfactual Evaluation of VLMs

    Thoughtworks · University of Ottawa

  19. Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions

    Mar 31, 2026Xuesong Wang, Harry WangVLM RobustnessReasoning for VLMs

    Wayne State University · University of Michigan

  20. Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions

    Mar 30, 2026Doan Nam Long Vu, Simone BalloccuMedical VLMsPrompt Sensitivity

    Natural Language Processing for Expert Domains (ExpNLP), Technical University of Darmstadt

  21. CDH-Bench: A Commonsense-Driven Hallucination Benchmark for Evaluating Visual Fidelity in Vision-Language Models

    Mar 30, 2026Kesheng Chen, Yamin Hu, Qi Zhou +2Emotional Hallucination in VLMsVLM Hallucination

    Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China

  22. CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations

    Mar 22, 2026Shanmukha Vellamcheti, Uday Kiran Kothapalli, Disharee Bhowmick +1Spatial Reasoning BenchmarksMulti-View Consistency

    Department of Computer Science and Software Engineering, Auburn University Auburn, AL 36849, USA

  23. FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs

    Mar 20, 2026Zhihan Yin, Jianxin Liang, Yueqian Wang +3VLM HallucinationVLM Evaluation

    Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence

  24. To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

    Mar 19, 2026Rui Hong, Shuxue QuanSelective Prediction for VLMsSycophancy in VLMs

    George Mason University, Fairfax, VA, USA · Independent Researcher

  25. Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

    Mar 8, 2026Kaihua Tang, Jiaxin Qi, Jinli Ou +2VLM RobustnessTest-Time Scaling for VLMs

    Tongji University, China · Computer Network Information Center, CAS, China · University of Chinese Academy of Sciences, China +1

  26. VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images

    Mar 3, 2026Neil TripathiVLM EvaluationVLM Reasoning

    New York University

  27. Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning

    Mar 3, 2026Anas Zafar, Leema Krishna Murali, Ashish VashistVisual GroundingMedical VQA

    The University of Texas MD Anderson Cancer Center · Cohere Labs Community · Eisai Inc. +1

  28. Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples

    Mar 2, 2026Phillip Howard, Xin Su, Kathleen C. FraserVLM EvaluationCounterfactual Evaluation of VLMs

    Thoughtworks · University of Ottawa

  29. Half-Truths Break Similarity-Based Retrieval

    Feb 27, 2026Bora Kargi, Arnas Uselis, Seong Joon OhContrastive Learning for VLMsVision-Language Models

    University of Tübingen, Tübingen AI Center · Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)

  30. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

    Feb 19, 2026Yu Fang, Yuchun Feng, Dong Jing +5VLM RobustnessCounterfactual Evaluation of VLMs

    University of North Carolina at Chapel Hill