Counterfactual Evaluation of VLMs

VLM: Vision-Language Model

Overview

Counterfactual evaluation studies whether vision-language models use visual evidence by changing an image, video, or context in a controlled way and checking whether the answer responds as expected. Common approaches pair factual and edited examples, use image ablations or misleading text, and measure answer flips, consistency, grounding, and confidence; some also trace internal activations or train models on evidence-dependent changes. These tests are used to diagnose hallucination, spatial and causal reasoning, chart interpretation, medical grounding, and safety decisions. A central challenge is separating genuine evidence use from correct guesses, language shortcuts, and unperceived edits, so robust evaluation needs multiple perturbations and controls for perceptibility and blind baselines.

Latest papers

CardsList

76 papers17 papers in the last four weeks, down 11% on the four weeks before. 0.1% of all new papers.

  1. SpatialCast: Reading the 3D World via Cast Shadows

    Nov 17, 2026Enhan Li, Jinghao Zhou, Jiaze Zhang +1Spatial Reasoning in Language Models3D Spatial Reasoning

  2. Words Speak Louder? Exploring the Language Shortcut of LLM-Driven Incomplete Multimodal Learning

    Nov 16, 2026Qiangchang Wang, Chong Xiao, Xiaokun Bai +5Multimodal LearningMissing-Modality Learning

  3. ReactBench-AD: Do Driving VLAs and World-Action Models React to Imminent Hazards in Their Path?

    Nov 15, 2026Nikos Theodoridis, Reenu Mohandas, Sushil Sharma +5Autonomous DrivingLong-Tail Autonomous Driving

  4. Do VLMs Really Look at the Road? A Diagnostic Evaluation for Autonomous Driving

    Nov 14, 2026Shankar Gangisetty, Muskan Singh, C V JawaharVLMs for Autonomous DrivingAutonomous Driving Safety Evaluation

  5. Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics

    Nov 10, 2026Qinan Zhang, Xinyu Wang, Qihang Jin +6VLM EvaluationVision-Language Model-as-a-Judge

  6. Do Embodied VLMs Actually Use Their Scene-Graph Memory?

    Nov 1, 2026Shubh Garg, Divyansh Pandey, Vedant Khandelwal +2Graph-Based Agent MemoryVLM Evaluation

  7. See, Think, Use: Diagnosing Perception and Reasoning Failures in VLMs

    Nov 1, 2026Jiacheng Ding, Yanliang Qi, Cong Guo +1VLM EvaluationReasoning for VLMs

  8. Visually Irrelevant Hallucinations in Medical Vision-Language Models

    Oct 28, 2026Ke Zou, Nan Zhou, Tao Yang +9VLM HallucinationHallucination Detection in VLMs

  9. Diagnosing and Correcting Bias in MLLM for Long Video Understanding

    Oct 26, 2026Xusheng Liang, Jianqiao Sun, Hao Zhang +3Long-Video UnderstandingVisual Evidence Sensitivity in VLMs

  10. When Pointing Becomes Profiling: Social Bias and Refusal in Text-Steerable Visual Encoders

    Oct 26, 2026Seyedarmin Azizi, Parsa Razmara, Sajjad Ghiasvand +2Demographic Bias in VLMsVLM Safety

  11. Conviction Without Evidence: Verdict Stability Is Independent of Visual Grounding in Medical Vision--Language Models

    Oct 25, 2026Federico Felizzi, Francesco Andrea Causio, Bianca Destro Castaniti +17Medical VLMsVisual Evidence Sensitivity in VLMs

  12. Grounding on Request: Learning Modality Control in Vision-Language Models

    Oct 24, 2026Maxime Dassen, Alexander Martin, Andrew Yates +2Vision-Language GroundingVisual Evidence Sensitivity in VLMs

  13. InfiniteWorld: Separating Visual Evidence from Familiar Physics

    Oct 24, 2026Tong Zhang, Yuxuan Cheng, Yun PengPhysical ReasoningVisual Evidence Sensitivity in VLMs

  14. VLTest: Finding Joint Image-Text Failures in Vision-Language Models

    Oct 22, 2026Bin Duan, Jintao Lin, Guowei YangVLM EvaluationMutation Testing

  15. VisualTrust: Video Reasoning under On-Screen Claim Framing

    Oct 15, 2026Huachao Zhu, Junjielong Xu, Sicheng Li +5Video-Language Model EvaluationVLM Robustness

  16. Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning

    Oct 14, 2026Chunxi Luo, Qishuai Ding, Xingchen Peng +6Spatial Reasoning BenchmarksSpatial Reasoning

  17. Teaching Vision-Language Models to Decline: Support Completion Against Mirage

    Oct 12, 2026Alfaj Uddin Ahmed, Md. Shaown Miah, Md Arid Hasan +2Abstention in VLMsVLM Hallucination Mitigation

  18. The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs

    Oct 9, 2026Qiji Zhou, Junyi Hu, Peiyu Wang +1VLM Hallucination MitigationMultimodal Large Language Models

  19. Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency

    Oct 6, 2026M. Moein Esfahani, Sepehr Salem, Mohammed Alser +1Physical ReasoningCounterfactual Evaluation of VLMs