Counterfactual Evaluation of VLMs

VLM: Vision-Language Model

Counterfactual evaluation studies whether vision-language models use visual evidence by changing an image, video, or context in a controlled way and checking whether the answer responds as expected. Common approaches pair factual and edited examples, use image ablations or misleading text, and measure answer flips, consistency, grounding, and confidence; some also trace internal activations or train models on evidence-dependent changes. These tests are used to diagnose hallucination, spatial and causal reasoning, chart interpretation, medical grounding, and safety decisions. A central challenge is separating genuine evidence use from correct guesses, language shortcuts, and unperceived edits, so robust evaluation needs multiple perturbations and controls for perceptibility and blind baselines.

Latest papers

61 papers20 papers in the last four weeks, up 54% on the four weeks before. 0.1% of all new papers.

CARE Drive A Framework for Evaluating Reason-Responsiveness of Vision Language Models in Automated Driving
Feb 17, 2026Lucas Elbert Suryana, Farah Bierenga, Sanne van Buuren +6Reasoning for VLMs VLM Reasoning
Department of Transport & Planning, Faculty of Civil Engineering and Geosciences, Delft University of Technology · Centre for Meaningful Human Control, Delft University of Technology · Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands +1

Feb 17, 2026J/K move · Enter open · S save

CARE Drive A Framework for Evaluating Reason-Responsiveness of Vision Language Models in Automated Driving

Lucas Elbert Suryana, Farah Bierenga, Sanne van Buuren, Pepijn Kooij+5

Department of Transport & Planning, Faculty of Civil Engineering and Geosciences, Delft University of Technology · Centre for Meaningful Human Control, Delft University of Technology · Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands+1

Reasoning for VLMs VLM Reasoning VLMs for Autonomous Driving Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

Foundation models, including vision language models, are increasingly used in automated driving to interpret scenes, recommend actions, and generate natural language explanations. However, existing evaluation methods primarily assess outcome based performance, such as safety and trajectory accuracy, without determining whether model decisions reflect human relevant considerations. As a result, it remains unclear whether explanations produced by such models correspond to genuine reason responsive decision making or merely post hoc rationalizations. This limitation is especially significant in safety critical domains because it can create false confidence. To address this gap, we propose CARE Drive, Context Aware Reasons Evaluation for Driving, a model agnostic framework for evaluating reason responsiveness in vision language models applied to automated driving. CARE Drive compares baseline and reason augmented model decisions under controlled contextual variation to assess whether human reasons causally influence decision behavior. The framework employs a two stage evaluation process. Prompt calibration ensures stable outputs. Systematic contextual perturbation then measures decision sensitivity to human reasons such as safety margins, social pressure, and efficiency constraints. We demonstrate CARE Drive in a cyclist overtaking scenario involving competing normative considerations. Results show that explicit human reasons significantly influence model decisions, improving alignment with expert recommended behavior. However, responsiveness varies across contextual factors, indicating uneven sensitivity to different types of reasons. These findings provide empirical evidence that reason responsiveness in foundation models can be systematically evaluated without modifying model parameters.