CARE Drive A Framework for Evaluating Reason-Responsiveness of Vision Language Models in Automated Driving

Lucas Elbert Suryana; Farah Bierenga; Sanne van Buuren; Pepijn Kooij; Elsefien Tulleners; Federico Scari; Simeon Calvert; Bart van Arem

cs.AIFeb 17, 2026

CARE Drive A Framework for Evaluating Reason-Responsiveness of Vision Language Models in Automated Driving

Authors: Lucas Elbert Suryana, Farah Bierenga, Sanne van Buuren, Pepijn Kooij, Elsefien Tulleners, Federico Scari, Simeon Calvert, Bart van Arem,

Organizations: Department of Transport & Planning, Faculty of Civil Engineering and Geosciences, Delft University of Technology · Centre for Meaningful Human Control, Delft University of Technology · Faculty of Mechanical Engineering, Delft University of Technology, Delft, The Netherlands · Department of Cognitive Robotics, Faculty of Mechanical Engineering, Delft University of Technology

Abstract

Foundation models, including vision language models, are increasingly used in automated driving to interpret scenes, recommend actions, and generate natural language explanations. However, existing evaluation methods primarily assess outcome based performance, such as safety and trajectory accuracy, without determining whether model decisions reflect human relevant considerations. As a result, it remains unclear whether explanations produced by such models correspond to genuine reason responsive decision making or merely post hoc rationalizations. This limitation is especially significant in safety critical domains because it can create false confidence. To address this gap, we propose CARE Drive, Context Aware Reasons Evaluation for Driving, a model agnostic framework for evaluating reason responsiveness in vision language models applied to automated driving. CARE Drive compares baseline and reason augmented model decisions under controlled contextual variation to assess whether human reasons causally influence decision behavior. The framework employs a two stage evaluation process. Prompt calibration ensures stable outputs. Systematic contextual perturbation then measures decision sensitivity to human reasons such as safety margins, social pressure, and efficiency constraints. We demonstrate CARE Drive in a cyclist overtaking scenario involving competing normative considerations. Results show that explicit human reasons significantly influence model decisions, improving alignment with expert recommended behavior. However, responsiveness varies across contextual factors, indicating uneven sensitivity to different types of reasons. These findings provide empirical evidence that reason responsiveness in foundation models can be systematically evaluated without modifying model parameters.

Figures & tables

Figure 1: Overview of the CARE-Drive framework for evaluating reason-responsive VLM decision-making in ethically ambiguous driving scenarios. The left panel illustrates the motivating setting: similar driving scenes (e.g., cyclist on a no-passing road) can give rise to conflicting reasons (safety, legality, comfort, efficiency), leading to ambiguity whether to overtake. The middle panel shows how a vision-language model (VLM) receives a scene representation $S=\{V,O\}$ , consisting of a visual input $V$ and observable context $O$ , together with a decision instruction $I(R,T,L)$ that specifies injected human reasons $R$ , a thought strategy $T$ , and an explanation-length regime $L$ . The VLM produces a decision $D_{VLM}$ with or without human reasons $(R=\varnothing)$ , enabling comparison between baseline and reason-augmented behaviour. The right panel depicts the two-stage CARE-Drive evaluation pipeline. In Stage 1 (Prompt Calibration) , the model $M$ and thought strategy $T$ are treated as calibration variables. The goal is to identify an optimal configuration $(M^{*},T^{*})$ that maximises alignment between the reason-augmented VLM decision $V_{VLM}^{(+R)}$ and the expert reference decision $D_{AV}$ . In Stage 2 (Contextual Evaluation) , the calibrated configuration $(M^{*},T^{*})$ is held fixed, and the observable context $O$ (e.g., time-to-collision with oncoming vehicles, presence of vehicle behind, passenger urgency) is systematically varied to measure how sensitively VLM augmented with human reasons casually influences decisions across situations. The bottom arrow in blue illustrates CARE-Drive’s core outcome: quantifying the alignment between baseline VLM decisions, reason-augmented decisions, and expert judgments.

Figure 2: Visual scenes used in CARE-Drive. The scenes are taken from CARLA simulator. (a) Scenario 1 (baseline) and Scenario 3 (vehicle behind) share the same dashboard image, since rear vehicles are not visible in $V$ . (b) Scenario 2 includes an oncoming vehicle visible in the image. The vehicle behind in Scenario 3 is represented only in the observable context $O$ .

Figure 3: Illustration of the overtaking scenario with an oncoming vehicle. The passing phase, defined as the moment when the ego vehicle is in the opposing lane while overtaking the cyclist, is used as the reference point for calculating the time-to-collision (TTC) with the oncoming vehicle. The longitudinal distance to the cyclist is defined when the ego vehicle is directly behind the cyclist prior to initiating the overtaking maneuver.

Component	Scenario 1	Scenario 2	Scenario 3
Visual input $V_{s}$	$\{V_{s1}\}$	$\{V_{s1},V_{s2}\}$	$\{V_{s1}\}$
$TTC_{o}$ (s)	$\varnothing$	$8.5$	$\varnothing$
Vehicle behind $B$	$0$	$0$	$1$
Following time $F$ (s)	$24$	$24$	$24$

Table 1: Fixed scene instances used in Step 2 robustness evaluation. Each scenario corresponds to a predefined scene

(V_{s},O_{s})

rather than a parametric variation of context variables.

#	Varied variable	Symbolic definition
1	Time-to-collision with oncoming vehicle	$TTC_{o}\in\{1.7,\,3.4,\,5.1,\,6.8,\,8.5\}\;\text{s}$
2	Vehicle behind indicator	$B\in\{0,1\}$ , where $B=1$ indicates a vehicle following the AV
3	Passenger urgency indicator	$U\in\{0,1\}$ , where $U=1$ denotes a passenger in a hurry
4	Following time behind cyclist	$F\in\{12,\,18,\,24\}\;\text{s}$
5	Explanation-length regime	$L\in\{0,1\}$ , where $L=1$ indicates limit on explanation length

Table 2: Varied variables used in Stage 2 to evaluate

D^{(+R)}_{VLM}

under variations of the baseline oncoming-vehicle scenario (Scenario 2).

Prompt configuration	Model $M$	Overtake count
Role	gpt-4.1	0
Role	gpt-4.1-mini	0
Role	gpt-4.1-nano	0
Role + R	gpt-4.1	20
Role + R	gpt-4.1-mini	15
Role + R	gpt-4.1-nano	0

Table 3: Overtaking counts in the baseline calibration scenario (24 s following time) for different prompt configurations and models. Each condition was run 30 times.

Scenario	Prompt	No-limit		Few-sentences
		Time (s)	Overtake (%)	Time (s)	Overtake (%)
Scenario 1 (None)	Role + HR + CoT	18.037	50.00	3.950	96.67
	Role + HR + ToT	16.600	76.67	4.468	96.67
Scenario 2 (Oncoming)	Role + HR + CoT	18.000	30.00	5.221	0.00
	Role + HR + ToT	19.568	93.33	5.630	0.00
Scenario 3 (Follow)	Role + HR + CoT	19.900	16.67	4.212	10.00

Table 4: Stage 1 (Step 2) calibration refinement: robustness across scenarios and explanation-length regimes for gpt-4.1 . Values report average generation time (s) and overtaking rate (%) over 30 runs per condition.

Figure 4: Full-factorial empirical overtaking probability $P(Y=1)$ of the calibrated CARE-Drive configuration, where $(M^{*},T^{*})=(\texttt{gpt-4.1},\text{Tree-of-Thought})$ , under systematic variation of observable context and explanation length. Each subplot shows the empirical overtaking probability $P(Y=1)$ as a function of time-to-collision $TTC_{o}$ . Columns correspond to following time $F\in\{12,18,24\}\,$ s, and rows correspond to the explanation-length regime (top row: $L=\text{No-Limit}$ ; bottom row: $L=\text{Few-Sentences}$ ). Within each subplot, curves represent combinations of vehicle-behind indicator $B\in\{0,1\}$ and passenger urgency $U\in\{0,1\}$ . Each point represents the proportion of overtaking decisions over 30 stochastic runs per condition.

	Coefficient ( $\beta$ )	Significance	Odds ratio	95% CI ( $\beta$ )	p-value
Intercept	$-1.953$		0.142	[-2.238, -1.669]	$<0.001$
Time-to-collision ( $TTC_{o}$ )	$3.015$		20.389	[2.687, 3.343]	$<0.001$
Vehicle behind ( $B$ )	$1.330$		3.781	[1.116, 1.545]	$<0.001$
Passenger urgency ( $U$ )	$-0.872$		0.418	[-1.081, -0.663]	$<0.001$
Following time ( $F$ )	$-0.049$		0.952	[-0.299, 0.201]	0.702
Explanation length ( $L$ )	$-4.184$		0.015	[-4.529, -3.840]	$<0.001$

Table 5: Binary logit regression results showing the effect of observable context and explanation length on overtaking decisions.

Predictor condition	Predicted probability	Change from baseline
Baseline ( $TTC_{o}=0,B=0,U=0,F=0,L=0$ )	12.4%	—
Time-to-collision increased ( $TTC_{o}=1\text{, normalized}$ )	74.3%	${\color[rgb]{0,0.5,0.5}+61.9\%}$
Vehicle behind present ( $B=1$ )	34.9%	${\color[rgb]{0,0.5,0.5}+22.5\%}$
Passenger urgency present ( $U=1$ )	5.6%	${\color[rgb]{0.75,0,0.25}-6.8\%}$
Following time increased ( $F=1\text{, normalized}$ )	11.9%	${\color[rgb]{0.75,0,0.25}-0.5\%}$
Explanation present ( $L=1$ )	0.2%	${\color[rgb]{0.75,0,0.25}-12.2\%}$

Table 6: Predicted overtaking probability under baseline conditions and individual predictor changes

Reasoning for VLMs

See all 39 papers

ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving
May 27, 2026Mohammadreza Teymoorianfard, Jean-Philippe Monteuuis, Jonathan Petit +1VLMs for Autonomous Driving Autonomous Driving Safety Evaluation
University of Massachusetts Amherst · Qualcomm
CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving
Sep 26, 2026Narendiran Chembu, Navvrat Rao, Shreedhar Shreeshail Kodate +10Reasoning for VLA Models Vision-Language-Action Models
Fastcode AI · Indian Institue of Technology Delhi
More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models
Oct 6, 2025Xurui Song, Shuo Huai, JingJing Jiang +2VLMs for Autonomous Driving Reasoning for VLA Models
S-Lab, Nanyang Technological University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore

May 27, 2026J/K move · Enter open · S save

ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving

Mohammadreza Teymoorianfard, Jean-Philippe Monteuuis, Jonathan Petit, Amir Houmansadr

University of Massachusetts Amherst · Qualcomm

VLMs for Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving Adversarial Attacks on VLA Models VLM Reasoning Vision-Language-Action Models+2

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models with integrated reasoning have been proposed for end-to-end autonomous driving, assuming a tight coupling between reasoning and trajectory generation. However, the robustness of such systems under realistic input perturbations remains largely unexplored. We show that these models are highly vulnerable to realistic input perturbations, achieving up to 89% attack success rate (ASR) on reasoning and up to 72% on trajectory manipulation in closed-loop simulation, leading to increased collision rates and degraded safety metrics. Using NVIDIA's recent Alpamayo models as representative industry-developed VLAs, we conduct the first systematic black-box study of reasoning-enabled VLA models under realistic textual input corruptions, evaluating their impact on reasoning and driving behavior. We introduce a reasoning-aware evaluation framework capturing both semantic and structural aspects of reasoning, along with safety-centric measures. We also introduce a benchmark for evaluating attacks and defenses on reasoning-trajectory interactions in autonomous driving. Our results highlight the need for rigorous evaluation and improved defenses to ensure the safety of reasoning-enabled VLA systems in autonomous driving.

Abstract

Reasoning for VLMs

Explore similar work

ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving

CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving

More Than Meets the Eye? Uncovering the Reasoning-Planning Disconnect in Training Vision-Language Driving Models