Counterfactual Evaluation of VLMs

Counterfactual evaluation studies whether vision-language models use visual evidence by changing an image, video, or context in a controlled way and checking whether the answer responds as expected. Common approaches pair factual and edited examples, use image ablations or misleading text, and measure answer flips, consistency, grounding, and confidence; some also trace internal activations or train models on evidence-dependent changes. These tests are used to diagnose hallucination, spatial and causal reasoning, chart interpretation, medical grounding, and safety decisions. A central challenge is separating genuine evidence use from correct guesses, language shortcuts, and unperceived edits, so robust evaluation needs multiple perturbations and controls for perceptibility and blind baselines.