Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset
Authors: Royden Wagner, Omer Sahin Tas, Jaime Villa, Felix Hauser, Yinzhe Shen, Marlon Steiner, Dominik Strutz, Carlos Fernandez, Quentin Delfosse, Christoph Weinhuber, Christian Kinzig, Guillermo S. Gutierrez-Cabello, Hendrik Königshof, Fabian Immel, Richard Schwarzkopf, Nils Alexander Rack, Kevin Rösch, Kaiwen Wang, Jan-Hendrik Pauls, Martin Lauer, Igor Gilitschenski, Holger Caesar, Christoph Stiller
Organizations: Karlsruhe Institute of Technology (KIT) · FZI Research Center for Information Technology · Carlos III University of Madrid · Technical University of Darmstadt · University of Oxford · Technical University of Madrid · University of Toronto · Delft University of Technology
Handling rare events is the central open challenge in autonomous driving. Reasoning models, which generate explicit chains of reasoning before acting, promise to generalize to such events. Here we show that these models frequently do not follow their own reasoning: the actions they state in their reasoning often diverge from the actions they ultimately execute. We introduce KITScenes LongTail, a curated dataset of rare driving scenarios to quantify this divergence through a measure of semantic reasoning-action coherence. We find that incoherence is widespread across current general-purpose and domain-specific models. Strikingly, when reasoned actions and executed actions disagree, the reasoning is usually right: extracting actions from the reasoning trace and executing them through a kinematic model enforces coherence and typically improves motion planning. These results suggest that the reasoning capabilities of current models exceed what their actions reveal, and that coherently acting on stated reasoning is a prerequisite for trustworthy autonomous driving. Our dataset and evaluation space are available at: https://hf.co/datasets/kit-mrt/kitscenes-longtail
Figures & tables
Figure 1: When reasoned actions and executed actions disagree, the reasoning is usually right. The actions described in the reasoning trace (left) provide a better answer to the long-tail scenario than the executed actions (right). Therefore, executing them with a kinematic model improves motion planning.
Lang.
Metric
Acceleration
Steering
Avg
English
BLEU [ 48 ]
0.48 ± 0.38
0.80 ± 0.17
0.64 ± 0.33
ROUGE-L [ 38 ]
0.48 ± 0.38
1.00 ± 0.00
0.74 ± 0.37
LingoJudge [ 43 ]
1.00 ± 0.00
0.95 ± 0.07
0.98 ± 0.05
RocchioEmb (ours)
0.98 ± 0.06
1.00 ± 0.00
0.99 ± 0.04
Spanish
BLEU [ 48 ]
0.38 ± 0.29
0.90 ± 0.14
0.64 ± 0.35
ROUGE-L [ 38 ]
0.40 ± 0.35
0.90 ± 0.14
0.65 ± 0.36
Table 1: Synonym robustness: Our light-weight Rocchio classifier outperforms BLEU and ROUGE-L and an LLM-as-a-judge metric. We measure the robustness w.r.t. synonyms as a classification task. Specifically, we classify synonyms and paraphrased versions of the set of high-level actions shown in the left column in Tab. 9 . RocchioEmb refers to our approach using an EmbeddingGemma model [ 62 ] with 300M parameters. Action labels, synonyms, and paraphrased versions are included in the extended data section. Best average values are bold , second best are underlined .
Figure 2: Our multi-maneuver scores (MMS) are a closer approximation of closed-loop simulation than L2 errors. Showing the relationship between MMS and L2 vs. closed-loop DrivingScore (DS) [ 28 ] , with linear fits and Pearson r values ( 0.59 and −0.45 ).
Model
MMS ↑
L2↓
mean
median
mean
median
UniAD [ 25 ]
3.29
3.50
10.90
6.79
DMAD [ 54 ]
3.52
3.50
10.04
5.51
Epona [ 72 ]
2.74
2.50
17.67
12.57
AutoVLA [ 73 ]
3.13
2.50
22.12
7.92
OneVL [ 40 ]
3.91
3.50
3.70
2.72
Table 2: Reasoning models (lower block) generalize better to our long-tail data. We evaluate on our test split and the metrics are averaged over our scenario types (see Sec. 4.5 ). In addition, the lower mean-to-median ratio demonstrates that the distribution of our proposed multi-maneuver score (MMS, see Sec. 4.2 ) is less skewed by outliers than that of standard L2 errors.
Model type
Model
Full coherence ↑
SRAC ↑
SRAC ↑
MMS ↑
L2↓
(%)
(top2)
( m )
Driving-specific reasoning models
AutoVLA [ 73 ]
28.6 ± 0.5
0.59 ± 0.01
0.69 ± 0.00
3.13 ± 0.06
22.12 ± 0.09
OneVL [ 40 ]
39.1 ± 4.6
0.77 ± 0.01
0.83 ± 0.02
3.91 ± 0.07
3.70 ± 0.26
Alpamayo 1.5 [ 64 ]
15.9 ± 1.8
0.61 ± 0.02
0.72 ± 0.03
4.69 ± 0.04
3.30 ± 0.26
Alpamayo 2 [ 46 ]
14.0 ± 1.1
0.56 ± 0.01
0.67 ± 0.02
5.06 ± 0.09
2.51 ± 0.03
Embodied reasoning models
Cosmos-Reason2 [ 45 ]
50.3 ± 2.4
0.67 ± 0.01
0.68 ± 0.01
3.65 ± 0.07
8.58 ± 1.57
Table 3: Reasoned and executed actions are frequently incoherent. Full coherence gives the percentage of scenarios where all reasoned actions are correctly executed (i.e., where SRAC = 1.0). Semantic reasoning-action coherence (SRAC) gives the average coherence rate across the 4 steering and acceleration actions expressed in reasoning traces (see Sec. 4.1 ). For open-weight models, we compute the mean and standard deviation using 3 global seeds. For API-gated models, we call the API 3 times to compute means and standard deviations.
Language
Model
Full coherence ↑
SRAC ↑
SRAC ↑
MMS ↑
L2↓
(%)
(top2)
( m )
Spanish
GPT-5.6 Luna [ 56 ]
35.3 ± 0.7
0.71 ± 0.01
0.75 ± 0.01
3.98 ± 0.10
4.53 ± 0.32
Gemini 3.6 Flash [ 21 ]
34.1 ± 0.7
0.70 ± 0.01
0.74 ± 0.01
4.13 ± 0.09
4.62 ± 0.29
Claude Haiku 4.5 [ 2 ]
23.7 ± 0.8
0.57 ± 0.01
0.64 ± 0.01
3.39 ± 0.07
8.42 ± 0.33
Chinese
GPT-5.6 Luna [ 56 ]
35.3 ± 1.8
0.74 ± 0.01
0.76 ± 0.01
3.92 ± 0.12
4.09 ± 0.25
Gemini 3.6 Flash [ 21 ]
42.0 ± 1.9
0.78 ± 0.00
0.81 ± 0.00
4.20 ± 0.07
4.41 ± 0.12
Table 4: When reasoning in Spanish or Chinese, reasoned and executed actions are frequently incoherent as well. Compared to reasoning in English (see lower block in Tab. 3 ), most scores worsen slightly for GPT-5.6 Luna and Claude Haiku 4.5, while Gemini 3.6 Flash seems to be less affected by the change in language.
Model type
Model
Win rate ↑
MMS ↑
L2↓
SRAC ↑
SRAC ↑
(%)
( m )
(top2)
Driving-specific reasoning models
AutoVLA [ 73 ] (kinematic)
86.0 ± 1.4
3.90 ± 0.04
8.95 ± 0.25
0.88 ± 0.01
0.88 ± 0.00
OneVL [ 40 ] (kinematic)
80.0 ± 2.8
4.42 ± 0.01
3.63 ± 0.26
1.00 ± 0.00
1.00 ± 0.00
Alpamayo 1.5 [ 64 ] (kinematic)
46.2 ± 3.0
4.30 ± 0.07
3.45 ± 0.05
0.98 ± 0.01
0.98 ± 0.01
Alpamayo 2 [ 46 ] (kinematic)
37.9 ± 1.5
4.69 ± 0.09
3.27 ± 0.08
0.96 ± 0.02
0.97 ± 0.02
Embodied reasoning models
Cosmos-Reason2 [ 45 ] (kinematic)
75.8 ± 1.8
4.01 ± 0.05
6.40 ± 0.67
0.93 ± 0.00
0.94 ± 0.00
Table 5: When reasoned and executed actions disagree, executing the reasoned actions through a kinematic model typically improves results. Win rate measures how often MMS improves when executing reasoned actions through the kinematic model compared to the originally executed actions, after filtering ties (the MMS values of the originally executed actions are shown in Tab. 3 ). Furthermore, this hybrid inference setting enforces semantic reasoning-action coherence by design, as shown by high SRAC scores. As reference for the upper bound, the last row shows the scores obtained by executing the ground truth actions (extracted from the expert trajectory) through our kinematic model (see Sec. 4.4 ).
Model
MMS ↑
Δ MMS
L2↓
ΔL2
Alpamayo 1.5 [ 64 ]
4.69 ± 0.04
3.30 ± 0.26
Alpamayo 1.5 (masked CoT)
4.81 ± 0.08
+0.12
2.95 ± 0.15
− 0.35
AutoVLA [ 73 ]
3.13 ± 0.06
22.12 ± 0.09
AutoVLA (no CoT)
3.30 ± 0.01
+0.16
21.28 ± 0.02
− 0.84
OneVL [ 40 ]
3.91 ± 0.07
3.70 ± 0.26
OneVL (no latent reasoning)
3.85 ± 0.12
− 0.06
3.75 ± 0.09
+0.05
Table 6: Removing reasoning only slightly affects executed actions. We remove reasoning by masking thinking tokens from the trajectory decoder (Alpamayo 1.5, Alpamayo 2) or truncating their generation (AutoVLA, OneVL). Overall, the changes in MMS and L2 are rather marginal and close to the standard deviations between runs.
Figure 3: Raising sampling temperature until reasoning traces become nonsensical only moderately affects executed actions. Specifically, semantic reasoning-action coherence (SRAC) degrades significantly more with sampling temperature than the multi-maneuver scores (MMS) of executed actions.
Category
Comfort penalty (CP)
Score
Expert-like trajectory
none
10
jerk XOR tortuosity
9
jerk AND tortuosity
8
Wrong speed
none
7
jerk XOR tortuosity
6
jerk AND tortuosity
5
Table 7: Reference multi-maneuver scores (MMS). Our metric ranks planned trajectories based on similarity to reference trajectories of 5 categories. For the first three categories, we apply comfort penalties if the jerk or tortuosity significantly exceeds that of the reference trajectory.
Figure 4: Our pseudo-simulation via MMS covers different maneuvers and speed variations. Specifically, we provide three reference trajectories. The expert trajectory describes the intended left turn. The wrong speed trajectory executes the same maneuver, but too slowly. The neglect instruction trajectory turns right instead of left. The matching areas at the end of each reference trajectory are based on the miss rate metric [ 15 ] . Inside an area, sim = 1 . Here, the predicted trajectory ends close to the wrong-speed reference’s matching area (sim = 0.75). Thus, we compute the MMS by multiplying the reference MMS by the similarity. In this case, no comfort penalty is subtracted since jerk and tortuosity are similar to the reference trajectory.
Figure 5: Distribution of scenario types. Numbers are percentages.
Figure 6: We provide multi-view video data and past motion trajectories, and prompt reasoning models to reason before planning a future trajectory. The reasoning covers situational awareness and driving actions (i.e., steering and acceleration). Our reasoning evaluation (see Sec. 2.3 ) is split into two segments: 4 s to 7 s (first 3 s of the future trajectory) and 7 s to 9 s (last 2 s of the future trajectory). We adapt this structure from well-established motion forecasting benchmarks [ 15 , 67 ] .
Table 8: Our dataset and evaluation fill a benchmark gap and target reasoning-capable end-to-end driving methods. A half-filled circle indicates that a feature is partially available. For example, regarding long-tail scenarios, related work selects interesting scenarios based on variations in trajectories instead of scenario classes such as navigating a construction zone. As high-level instructions, related work provides only {right, left, straight}.
Figure 7: Qualitative results of driving-specific (Alpamayo 2), embodied (Gemini Robotics ER 1.6) and general-purpose (GPT-5.6 Luna) reasoning models. (a) to (c): Show examples where the predicted trajectory (blue) is perfectly matched to the expert trajectory (green). (d) : Shows an example where the predicted trajectory is matched to a crash trajectory (red, representing a rear-end crash). (e) and (f): Examples where the predicted trajectory is not matched to any reference trajectory. Additionally, neglect instruction (i.e., different maneuver) and wrong speed trajectories are shown in pink and yellow.
Figure 8: Front-view images of specifically selected, heavy rain, and snow scenarios. In addition to rare events like protesting climate activists (shown in the main paper), crashes, or road closures we also specifically select combinations of other long-tail classes. For example: (a) Specifically selected because of wintry mix and during the night. (b) and (c) Specifically selected because of heavy rain and during the night. (d) Specifically selected because of a construction zone and during the night.
Action label
Synonyms
maintain the current speed
maintaining the current speed; keep the current speed; hold the current speed; keep going at the current speed; stay at the same speed; staying at the same speed; maintain my current speed; maintain the current speed
accelerate slightly
accelerating slightly; speed up slightly; speeding up slightly; speed up a little; speeding up a little; increase speed slightly; increasing speed slightly; accelerate slightly
accelerate strongly
accelerating strongly; speed up quickly; speed up a lot; speeding up a lot; increase speed significantly; increasing speed significantly; accelerate hard; accelerate strongly
decelerate slightly
decelerating slightly; decelerate a little; decelerating a little; slow down slightly; slowing down slightly; reduce speed slightly; reducing speed slightly; decelerate slightly
decelerate strongly
decelerating strongly; decelerate significantly; decelerating significantly; slow down significantly; slowing down significantly; reduce speed significantly; reducing speed significantly; decelerate strongly
Table 9: Action labels and their surface-form synonyms. We use these for measuring synonym robustness and lookup in parsing.
Model
MMS ↑
L2↓
SRAC ↑
mean
median
mean
median
top1
top2
AutoVLA [ 73 ]
2.51
2.50
18.19
5.84
0.48
0.60
OneVL [ 40 ]
3.20
3.50
4.53
3.44
0.72
0.81
Alpamayo 1.5 [ 64 ]
3.69
3.50
4.13
3.30
0.57
0.71
Alpamayo 2 [ 46 ]
4.49
3.50
2.45
2.11
0.53
0.71
Table 10: Validation results. We evaluate on our val split and the metrics are averaged over our scenario types (see Sec. 4.5 ). Semantic reasoning-action coherence (SRAC) scores and L2 errors are close to the values on the test split, while multi-maneuver scores (MMS) are moderately lower.
Figure 9: Image stitching using gradual warping.
Action type
Command
Low speed
High speed
Acceleration m/s2
Decelerate strongly
−3.000
−5.500
Decelerate slightly
−1.286
−1.705
Maintain current speed
−0.088
−0.251
Accelerate slightly
1.018
1.469
Accelerate strongly
3.000
5.050
Steering ( m )
Steer left
3.290
1.386
Table 11: Command-to-control mapping. Controller parameters obtained with the selected configuration. Low and high speed correspond to velocities below and above 60 km/h, respectively.
Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer agent interactions, and make safe decisions. However, existing AD datasets and benchmarks mainly target perception, prediction, or planning, and provide limited supervision for reasoning over realistic long-tail driving scenes. We introduce nuReasoning, a large-scale real-world dataset and benchmark for reasoning-centric AD. Following the lineage of nuScenes and nuPlan, nuReasoning advances real-world AD datasets and benchmarks toward reasoning in long-tail driving scenarios. The dataset contains 20,000 clips, each 20 seconds long, collected across multiple cities, with synchronized multi-camera images, LiDAR data, HD maps, object annotations, and human-verified reasoning annotations spanning Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning. Unlike prior datasets that focus primarily on visual question answering, nuReasoning supports both reasoning evaluation and planning evaluation, enabling a direct study of how reasoning supervision affects driving performance. Experiments show that fine-tuning VLMs on nuReasoning substantially improves driving-specific question answering, while incorporating reasoning supervision into VLA training improves planning performance even when textual reasoning outputs are disabled at inference time. These results establish nuReasoning as a foundation for evaluating and improving robust, interpretable, reasoning-driven AD systems in realistic long-tail settings.
Zhiyu Huang, Johnson Liu, Rui Song +13
Xuewei · University of California, Los Angeles · 2Motional
Chain-of-thought (CoT) reasoning powers generative models by eliciting intermediate steps before producing an answer. In autonomous driving, the answer is a continuous action. Thus its reasoning must share the same spatiotemporal structure as the physical world. This survey studies the resulting shift from textual CoT to action-grounded reasoning. Surveying 171 papers, including 130 method papers and 41 benchmarks, datasets, surveys, and analysis papers, we propose a representation-centered taxonomy that treats the form of the intermediate state as the organizing axis. We systematize the 130 methods into four categories: language-based, visual-spatial, latent-dynamic, and externalized reasoning, further divided into 13 subtypes tied to distinct regions of interests. Our synthesis shows that the open frontier of reasoning in driving agents lies in intermediate representations that can be grounded in the real world, coupled to real-time action, and verified under safety-critical systems. Project page: https://github.com/tangzhengxu/awesome-av-cot.
Reasoning is a promising route to the generalization that autonomous driving requires in the long tail, as it can infer how the elements of a scene depend on one another and traverse those dependencies to conclusions beyond what is observed. Yet it is hard to tell whether a model's conclusions follow the scene's dependencies, because no driving representation makes them explicit enough to test against. Text-based reasoning traces lack spatio-temporal grounding, spatio-temporal scene graphs lack causal links, and reasoning annotations at scale are increasingly model-generated and hard to verify. To this end, we introduce CASCADE (Causal Spatio-Temporal Analysis of Driving Environments), which encompasses two components: (1) a structured scene representation for reasoning in driving scenes and (2) a human-annotated dataset built on it. For every actor that interacts with the ego vehicle, the CASCADE representation records frame-by-frame, for as long as the actor is visible, what action is taken, where it occurs, and how it depends on the actions and states of others. The resulting structure makes reasoning predictions machine-verifiable: they can be scored against it element by element, without relying on (M)LLM judges. The CASCADE dataset provides comprehensive human annotations for 2,066 driving clips of the PhysicalAI dataset, with over 34K elements that establish the spatio-temporal and causal context of each scene, including 8.6K time-stamped ego and agent actions, 3.7K causal links and 2.9K potential influences, and 6.1K annotations for agents, objects, traffic lights, and environments. Being entirely human-annotated, CASCADE provides the reference for this comparison: benchmarking the reasoning abilities of Physical AI models, and verifying the quality of automatically generated reasoning labels. The CASCADE dataset is available at https://huggingface.co/datasets/nvidia/cascade.
Jenny Schmalfuss, Despoina Paschalidou, Simon Gerstenecker +2
NVIDIA · ETH Zürich · Work done during NVIDIA Internship