Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset

Royden Wagner; Omer Sahin Tas; Jaime Villa; Felix Hauser; Yinzhe Shen; Marlon Steiner; Dominik Strutz; Carlos Fernandez

cs.CVMar 24, 2026

Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset

Authors: Royden Wagner, Omer Sahin Tas, Jaime Villa, Felix Hauser, Yinzhe Shen, Marlon Steiner, Dominik Strutz, Carlos Fernandez,

Organizations: Karlsruhe Institute of Technology (KIT) · FZI Research Center for Information Technology · Carlos III University of Madrid · Technical University of Darmstadt ·

Abstract

Handling rare events is the central open challenge in autonomous driving. Reasoning models, which generate explicit chains of reasoning before acting, promise to generalize to such events. Here we show that these models frequently do not follow their own reasoning: the actions they state in their reasoning often diverge from the actions they ultimately execute. We introduce KITScenes LongTail, a curated dataset of rare driving scenarios to quantify this divergence through a measure of semantic reasoning-action coherence. We find that incoherence is widespread across current general-purpose and domain-specific models. Strikingly, when reasoned actions and executed actions disagree, the reasoning is usually right: extracting actions from the reasoning trace and executing them through a kinematic model enforces coherence and typically improves motion planning. These results suggest that the reasoning capabilities of current models exceed what their actions reveal, and that coherently acting on stated reasoning is a prerequisite for trustworthy autonomous driving. Our dataset and evaluation space are available at: https://hf.co/datasets/kit-mrt/kitscenes-longtail

Figures & tables

Panel of Figure 1 — Figure 1: When reasoned actions and executed actions disagree, the reasoning is usually right. The actions described in the reasoning trace (left) provide a better answer to the long-tail scenario than the executed actions (right). Therefore, executing them with a kinematic model improves motion planning.

Lang.	Metric	Acceleration	Steering	Avg
English	BLEU [ 48 ]	0.48 $\pm$ 0.38	0.80 $\pm$ 0.17	0.64 $\pm$ 0.33
	ROUGE-L [ 38 ]	0.48 $\pm$ 0.38	1.00 $\pm$ 0.00	0.74 $\pm$ 0.37
	LingoJudge [ 43 ]	1.00 $\pm$ 0.00	0.95 $\pm$ 0.07	0.98 $\pm$ 0.05
	RocchioEmb (ours)	0.98 $\pm$ 0.06	1.00 $\pm$ 0.00	0.99 $\pm$ 0.04
Spanish	BLEU [ 48 ]	0.38 $\pm$ 0.29	0.90 $\pm$ 0.14	0.64 $\pm$ 0.35
	ROUGE-L [ 38 ]	0.40 $\pm$ 0.35	0.90 $\pm$ 0.14	0.65 $\pm$ 0.36

Table 1: Synonym robustness: Our light-weight Rocchio classifier outperforms BLEU and ROUGE-L and an LLM-as-a-judge metric. We measure the robustness w.r.t. synonyms as a classification task. Specifically, we classify synonyms and paraphrased versions of the set of high-level actions shown in the left column in Tab. 9 . RocchioEmb refers to our approach using an EmbeddingGemma model [ 62 ] with 300M parameters. Action labels, synonyms, and paraphrased versions are included in the extended data section. Best average values are bold , second best are underlined .

Figure 2: Our multi-maneuver scores (MMS) are a closer approximation of closed-loop simulation than L2 errors. Showing the relationship between MMS and

L_{2}

vs. closed-loop DrivingScore (DS) [ 28 ] , with linear fits and Pearson

r

values (

0.59

and

-0.45

Model	MMS $\uparrow$		$L_{2}$ $\downarrow$
Model	mean	median	mean	median
UniAD [ 25 ]	3.29	3.50	10.90	6.79
DMAD [ 54 ]	3.52	3.50	10.04	5.51
Epona [ 72 ]	2.74	2.50	17.67	12.57
AutoVLA [ 73 ]	3.13	2.50	22.12	7.92
OneVL [ 40 ]	3.91	3.50	3.70	2.72

Table 2: Reasoning models (lower block) generalize better to our long-tail data. We evaluate on our test split and the metrics are averaged over our scenario types (see Sec. 4.5 ). In addition, the lower mean-to-median ratio demonstrates that the distribution of our proposed multi-maneuver score (MMS, see Sec. 4.2 ) is less skewed by outliers than that of standard

L_{2}

errors.

Model type	Model	Full coherence $\uparrow$	SRAC $\uparrow$	SRAC $\uparrow$	MMS $\uparrow$	$L_{2}\downarrow$
	Model	(%)	SRAC $\uparrow$	(top2)	MMS $\uparrow$	( $\mathrm{m}$ )
Driving-specific reasoning models	AutoVLA [ 73 ]	28.6 $\pm$ 0.5	0.59 $\pm$ 0.01	0.69 $\pm$ 0.00	3.13 $\pm$ 0.06	22.12 $\pm$ 0.09
	OneVL [ 40 ]	39.1 $\pm$ 4.6	0.77 $\pm$ 0.01	0.83 $\pm$ 0.02	3.91 $\pm$ 0.07	3.70 $\pm$ 0.26
	Alpamayo 1.5 [ 64 ]	15.9 $\pm$ 1.8	0.61 $\pm$ 0.02	0.72 $\pm$ 0.03	4.69 $\pm$ 0.04	3.30 $\pm$ 0.26
	Alpamayo 2 [ 46 ]	14.0 $\pm$ 1.1	0.56 $\pm$ 0.01	0.67 $\pm$ 0.02	5.06 $\pm$ 0.09	2.51 $\pm$ 0.03
Embodied reasoning models	Cosmos-Reason2 [ 45 ]	50.3 $\pm$ 2.4	0.67 $\pm$ 0.01	0.68 $\pm$ 0.01	3.65 $\pm$ 0.07	8.58 $\pm$ 1.57

Table 3: Reasoned and executed actions are frequently incoherent. Full coherence gives the percentage of scenarios where all reasoned actions are correctly executed (i.e., where SRAC = 1.0). Semantic reasoning-action coherence (SRAC) gives the average coherence rate across the 4 steering and acceleration actions expressed in reasoning traces (see Sec. 4.1 ). For open-weight models, we compute the mean and standard deviation using 3 global seeds. For API-gated models, we call the API 3 times to compute means and standard deviations.

Language	Model	Full coherence $\uparrow$	SRAC $\uparrow$	SRAC $\uparrow$	MMS $\uparrow$	$L_{2}\downarrow$
Language	Model	(%)	SRAC $\uparrow$	(top2)	MMS $\uparrow$	( $\mathrm{m}$ )
Spanish	GPT-5.6 Luna [ 56 ]	35.3 $\pm$ 0.7	0.71 $\pm$ 0.01	0.75 $\pm$ 0.01	3.98 $\pm$ 0.10	4.53 $\pm$ 0.32
	Gemini 3.6 Flash [ 21 ]	34.1 $\pm$ 0.7	0.70 $\pm$ 0.01	0.74 $\pm$ 0.01	4.13 $\pm$ 0.09	4.62 $\pm$ 0.29
	Claude Haiku 4.5 [ 2 ]	23.7 $\pm$ 0.8	0.57 $\pm$ 0.01	0.64 $\pm$ 0.01	3.39 $\pm$ 0.07	8.42 $\pm$ 0.33
Chinese	GPT-5.6 Luna [ 56 ]	35.3 $\pm$ 1.8	0.74 $\pm$ 0.01	0.76 $\pm$ 0.01	3.92 $\pm$ 0.12	4.09 $\pm$ 0.25
	Gemini 3.6 Flash [ 21 ]	42.0 $\pm$ 1.9	0.78 $\pm$ 0.00	0.81 $\pm$ 0.00	4.20 $\pm$ 0.07	4.41 $\pm$ 0.12

Table 4: When reasoning in Spanish or Chinese, reasoned and executed actions are frequently incoherent as well. Compared to reasoning in English (see lower block in Tab. 3 ), most scores worsen slightly for GPT-5.6 Luna and Claude Haiku 4.5, while Gemini 3.6 Flash seems to be less affected by the change in language.

Model type	Model	Win rate $\uparrow$	MMS $\uparrow$	$L_{2}$ $\downarrow$	SRAC $\uparrow$	SRAC $\uparrow$
	Model	(%)	MMS $\uparrow$	( $\mathrm{m}$ )	SRAC $\uparrow$	(top2)
Driving-specific reasoning models	AutoVLA [ 73 ] (kinematic)	86.0 $\pm$ 1.4	3.90 $\pm$ 0.04	8.95 $\pm$ 0.25	0.88 $\pm$ 0.01	0.88 $\pm$ 0.00
	OneVL [ 40 ] (kinematic)	80.0 $\pm$ 2.8	4.42 $\pm$ 0.01	3.63 $\pm$ 0.26	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00
	Alpamayo 1.5 [ 64 ] (kinematic)	46.2 $\pm$ 3.0	4.30 $\pm$ 0.07	3.45 $\pm$ 0.05	0.98 $\pm$ 0.01	0.98 $\pm$ 0.01
	Alpamayo 2 [ 46 ] (kinematic)	37.9 $\pm$ 1.5	4.69 $\pm$ 0.09	3.27 $\pm$ 0.08	0.96 $\pm$ 0.02	0.97 $\pm$ 0.02
Embodied reasoning models	Cosmos-Reason2 [ 45 ] (kinematic)	75.8 $\pm$ 1.8	4.01 $\pm$ 0.05	6.40 $\pm$ 0.67	0.93 $\pm$ 0.00	0.94 $\pm$ 0.00

Table 5: When reasoned and executed actions disagree, executing the reasoned actions through a kinematic model typically improves results. Win rate measures how often MMS improves when executing reasoned actions through the kinematic model compared to the originally executed actions, after filtering ties (the MMS values of the originally executed actions are shown in Tab. 3 ). Furthermore, this hybrid inference setting enforces semantic reasoning-action coherence by design, as shown by high SRAC scores. As reference for the upper bound, the last row shows the scores obtained by executing the ground truth actions (extracted from the expert trajectory) through our kinematic model (see Sec. 4.4 ).

Model	MMS $\uparrow$	$\Delta$ MMS	$L_{2}$ $\downarrow$	$\Delta$ $L_{2}$
Alpamayo 1.5 [ 64 ]	4.69 $\pm$ 0.04		3.30 $\pm$ 0.26
Alpamayo 1.5 (masked CoT)	4.81 $\pm$ 0.08	+0.12	2.95 $\pm$ 0.15	$-$ 0.35
AutoVLA [ 73 ]	3.13 $\pm$ 0.06		22.12 $\pm$ 0.09
AutoVLA (no CoT)	3.30 $\pm$ 0.01	+0.16	21.28 $\pm$ 0.02	$-$ 0.84
OneVL [ 40 ]	3.91 $\pm$ 0.07		3.70 $\pm$ 0.26
OneVL (no latent reasoning)	3.85 $\pm$ 0.12	$-$ 0.06	3.75 $\pm$ 0.09	+0.05

Table 6: Removing reasoning only slightly affects executed actions. We remove reasoning by masking thinking tokens from the trajectory decoder (Alpamayo 1.5, Alpamayo 2) or truncating their generation (AutoVLA, OneVL). Overall, the changes in MMS and

L_{2}

are rather marginal and close to the standard deviations between runs.

Figure 3: Raising sampling temperature until reasoning traces become nonsensical only moderately affects executed actions. Specifically, semantic reasoning-action coherence (SRAC) degrades significantly more with sampling temperature than the multi-maneuver scores (MMS) of executed actions.

Category	Comfort penalty (CP)	Score
Expert-like trajectory	none	$10$
	jerk XOR tortuosity	$9$
	jerk AND tortuosity	$8$
Wrong speed	none	$7$
	jerk XOR tortuosity	$6$
	jerk AND tortuosity	$5$

Table 7: Reference multi-maneuver scores (MMS). Our metric ranks planned trajectories based on similarity to reference trajectories of 5 categories. For the first three categories, we apply comfort penalties if the jerk or tortuosity significantly exceeds that of the reference trajectory.

Figure 4: Our pseudo-simulation via MMS covers different maneuvers and speed variations. Specifically, we provide three reference trajectories. The expert trajectory describes the intended left turn. The wrong speed trajectory executes the same maneuver, but too slowly. The neglect instruction trajectory turns right instead of left. The matching areas at the end of each reference trajectory are based on the miss rate metric [ 15 ] . Inside an area, sim = 1 . Here, the predicted trajectory ends close to the wrong-speed reference’s matching area (sim = 0.75). Thus, we compute the MMS by multiplying the reference MMS by the similarity. In this case, no comfort penalty is subtracted since jerk and tortuosity are similar to the reference trajectory.

Figure 5: Distribution of scenario types. Numbers are percentages.

Panel of Figure 6 — Figure 6: We provide multi-view video data and past motion trajectories, and prompt reasoning models to reason before planning a future trajectory. The reasoning covers situational awareness and driving actions (i.e., steering and acceleration). Our reasoning evaluation (see Sec. 2.3 ) is split into two segments: 4 s to 7 s (first 3 s of the future trajectory) and 7 s to 9 s (last 2 s of the future trajectory). We adapt this structure from well-established motion forecasting benchmarks [ 15 , 67 ] .

Panel of Table 8 — Table 8: Our dataset and evaluation fill a benchmark gap and target reasoning-capable end-to-end driving methods. A half-filled circle indicates that a feature is partially available. For example, regarding long-tail scenarios, related work selects interesting scenarios based on variations in trajectories instead of scenario classes such as navigating a construction zone. As high-level instructions, related work provides only {right, left, straight}.

Panel of Figure 7 — Figure 7: Qualitative results of driving-specific (Alpamayo 2), embodied (Gemini Robotics ER 1.6) and general-purpose (GPT-5.6 Luna) reasoning models. (a) to (c): Show examples where the predicted trajectory (blue) is perfectly matched to the expert trajectory (green). (d) : Shows an example where the predicted trajectory is matched to a crash trajectory (red, representing a rear-end crash). (e) and (f): Examples where the predicted trajectory is not matched to any reference trajectory. Additionally, neglect instruction (i.e., different maneuver) and wrong speed trajectories are shown in pink and yellow.

Panel of Figure 8 — Figure 8: Front-view images of specifically selected, heavy rain, and snow scenarios. In addition to rare events like protesting climate activists (shown in the main paper), crashes, or road closures we also specifically select combinations of other long-tail classes. For example: (a) Specifically selected because of wintry mix and during the night. (b) and (c) Specifically selected because of heavy rain and during the night. (d) Specifically selected because of a construction zone and during the night.

Action label	Synonyms
maintain the current speed	maintaining the current speed; keep the current speed; hold the current speed; keep going at the current speed; stay at the same speed; staying at the same speed; maintain my current speed; maintain the current speed
accelerate slightly	accelerating slightly; speed up slightly; speeding up slightly; speed up a little; speeding up a little; increase speed slightly; increasing speed slightly; accelerate slightly
accelerate strongly	accelerating strongly; speed up quickly; speed up a lot; speeding up a lot; increase speed significantly; increasing speed significantly; accelerate hard; accelerate strongly
decelerate slightly	decelerating slightly; decelerate a little; decelerating a little; slow down slightly; slowing down slightly; reduce speed slightly; reducing speed slightly; decelerate slightly
decelerate strongly	decelerating strongly; decelerate significantly; decelerating significantly; slow down significantly; slowing down significantly; reduce speed significantly; reducing speed significantly; decelerate strongly
steer straight	steering straight; go straight; going straight; keep straight; stay straight; continue straight ahead; drive straight ahead; steer straight

Table 9: Action labels and their surface-form synonyms. We use these for measuring synonym robustness and lookup in parsing.

Model	MMS $\uparrow$		$L_{2}$ $\downarrow$		SRAC $\uparrow$
Model	mean	median	mean	median	top1	top2
AutoVLA [ 73 ]	2.51	2.50	18.19	5.84	0.48	0.60
OneVL [ 40 ]	3.20	3.50	4.53	3.44	0.72	0.81
Alpamayo 1.5 [ 64 ]	3.69	3.50	4.13	3.30	0.57	0.71
Alpamayo 2 [ 46 ]	4.49	3.50	2.45	2.11	0.53	0.71

Table 10: Validation results. We evaluate on our val split and the metrics are averaged over our scenario types (see Sec. 4.5 ). Semantic reasoning-action coherence (SRAC) scores and

L_{2}

errors are close to the values on the test split, while multi-maneuver scores (MMS) are moderately lower.

Figure 9: Image stitching using gradual warping.

Action type	Command	Low speed	High speed
Acceleration $\mathrm{m}\text{/}{\mathrm{s}}^{2}$	Decelerate strongly	$-3.000$	$-5.500$
	Decelerate slightly	$-1.286$	$-1.705$
	Maintain current speed	$-0.088$	$-0.251$
	Accelerate slightly	$1.018$	$1.469$
	Accelerate strongly	$3.000$	$5.050$
Steering ( $\mathrm{m}$ )	Steer left	$3.290$	$1.386$

Table 11: Command-to-control mapping. Controller parameters obtained with the selected configuration. Low and high speed correspond to velocities below and above 60 km/h, respectively.

Long-Tail Autonomous Driving

See all 11 papers

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving
May 29, 2026Zhiyu Huang, Johnson Liu, Rui Song +13Autonomous Driving Datasets Autonomous Driving Benchmarks
Xuewei · University of California, Los Angeles · 2Motional
Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving
Aug 31, 2026Zhengxu Tang, Xiaozhou Zhang, Guofeng Cui +10Autonomous Driving Reasoning for VLA Models
NVIDIA
CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving
Sep 7, 2026Jenny Schmalfuss, Despoina Paschalidou, Simon Gerstenecker +2Spatiotemporal Reasoning Autonomous Driving Datasets
NVIDIA · ETH Zürich · Work done during NVIDIA Internship

May 29, 2026J/K move · Enter open · S save

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

Zhiyu Huang, Johnson Liu, Rui Song, Zewei Zhou+12

Xuewei · University of California, Los Angeles · 2Motional

Autonomous Driving Datasets Autonomous Driving Benchmarks Long-Tail Autonomous Driving Autonomous Driving VLM Reasoning Autonomous Driving Planning

Open paper AskPDF arXiv

Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer agent interactions, and make safe decisions. However, existing AD datasets and benchmarks mainly target perception, prediction, or planning, and provide limited supervision for reasoning over realistic long-tail driving scenes. We introduce nuReasoning, a large-scale real-world dataset and benchmark for reasoning-centric AD. Following the lineage of nuScenes and nuPlan, nuReasoning advances real-world AD datasets and benchmarks toward reasoning in long-tail driving scenarios. The dataset contains 20,000 clips, each 20 seconds long, collected across multiple cities, with synchronized multi-camera images, LiDAR data, HD maps, object annotations, and human-verified reasoning annotations spanning Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning. Unlike prior datasets that focus primarily on visual question answering, nuReasoning supports both reasoning evaluation and planning evaluation, enabling a direct study of how reasoning supervision affects driving performance. Experiments show that fine-tuning VLMs on nuReasoning substantially improves driving-specific question answering, while incorporating reasoning supervision into VLA training improves planning performance even when textual reasoning outputs are disabled at inference time. These results establish nuReasoning as a foundation for evaluating and improving robust, interpretable, reasoning-driven AD systems in realistic long-tail settings.

Abstract

Long-Tail Autonomous Driving

Explore similar work

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving

CASCADE: A Spatio-Temporal-Causal Reasoning Representation and Dataset for Driving