Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoning reflects the causal structure of the scene remains untested. We introduce CausalDriveBench, an evaluation framework grounded in Pearl's Causal Hierarchy (PCH) that tests causal reasoning in driving-specific VLAs through structured visual question answering (QA) and alternative-trajectory prediction. To this end, we construct causal scene graphs over nuScenes that distinguish causally active, dormant, and distractor entities, separating perceptual salience from causal relevance. The benchmark spans all four rungs of PCH (association, intervention, and counterfactual along with causal discovery) for QA generation. For the higher rungs, we additionally provide reference trajectories under specified scene modifications, enabling action-level verification that complements reasoning-level evaluation. In total, the benchmark contains 7,285 verified causal QA pairs and 1,000 counterfactual trajectories derived from nuScenes. We evaluate 10 driving-specific VLAs and 3 general-purpose VLMs, and report three findings. First, the best model reaches only 70.6% QA accuracy, and 4 of 13 models score below random chance. Second, comparing each driving VLA to the general-purpose VLM that shares its language backbone, the cost of driving fine-tuning ranges from 2 to 34 percentage points on causal QA, with post-training design explaining the spread. Third, causal QA and trajectory accuracy are statistically uncorrelated across models: under counterfactual prompts, predicted trajectories either over-react or collapse onto the observed-scene baseline. Taken together, these results show that neither fluent rationales nor accurate observed-scene trajectories constitute evidence of causal understanding.
Figures & tables
Benchmark
Graph
Causal Status
PCH Layers
CF Semantics
Alt-traj GT
nuScenes-QA [ 30 ]
SG
✗
R1
✗
✗
BDD-X [ 20 ]
✗
✗
R1
✗
✗
DriveLM [ 34 ]
QA-Dep
✗
R1–R2
✗
✗
Reason2Drive [ 26 ]
✗
✗
R1–R2
✗
✗
DriveLMM-o1 [ 17 ]
✗
✗
R1–R2
✗
✗
DriveAction [ 15 ]
Tree
✗
R1–R2
✗
✗
Table 1: Comparison with driving VQA and trajectory benchmarks. Graph types: spatial scene graph (SG), QA-dependency graph (QA-Dep), action-rooted tree (Tree), causal scene graph (Causal). Causal Status indicates whether active, dormant, and distractor entities are separated. PCH Layers : Coverage of QA across rungs. CF Semantics : forward simulation (Sim) versus Pearl-form counterfactuals (Pearl). Alt-traj GT : ground-truth trajectories under counterfactual antecedents.
Figure 1: CausalDriveBench Overview with an example: Each scene is encoded as a causal scene graph extracted from multi-view camera observations and BEV HD map context. The graph distinguishes three causal statuses among visible entities: active (warm), dormant (green), and distractor (blue). From this graph, we derive structured QA pairs spanning all four PCH layers and counterfactual ego trajectory targets under controlled scene modification.
Figure 2: Distribution of QA pairs in the diverse-sampled subset of CausalDriveBench across three aspects: question type grouped by entity family (top), active subgraph structure within active-element QA (middle), and PCH layer (bottom). Slice colours encode entity status (red: active, sage: dormant, blue: distractor). Rungs use purple gradients that are visually orthogonal to the entity-family hues. See appendix C for full per-type counts.
Figure 3: Construction pipeline and validation. Stages interleave deterministic logic, LLM extraction, and human review to produce 7,285 QA pairs and 1,000 counterfactual trajectories from 3,189 candidate samples. Three validation gates certify the released benchmark: graph review at Stage 1, an independent 5% QA audit in Stage-3, and a trajectory audit at Stage 4. See section 4 for more details.
PCH layer
Entity type
LLM-judge axis
Model
n
Overall
R0
R1
R2
R3
Active
Dormant
Distractor
Comp.
Spat.
Mech.
Cond.
Cons.
Driving-specific VLAs
ImpromptuVLA-7B [ 11 ]
7285
69.27%
66.43%
59.41%
85.10%
66.93%
69.07%
63.92%
71.69%
1.06
1.08
1.12
0.63
1.39
OpenREAD [ 43 ]
7274
63.76%
70.10%
48.95%
78.89%
61.03%
69.81%
60.62%
60.80%
1.18
1.14
1.30
0.83
1.44
Orion [ 14 ]
7285
57.25%
25.44%
76.03%
66.40%
52.58%
30.27%
53.26%
78.08%
0.19
0.14
0.07
0.06
0.51
UniDrive-VLA [ 24 ]
7285
54.87%
63.69%
29.71%
68.24%
60.08%
65.88%
53.75%
47.54%
1.12
1.21
1.24
0.74
1.31
Table 2: VQA accuracy (%) and LLM-judge reasoning scores (0–2 per dimension, higher is better). Best results in each column are in bold. See section 5.1 for more details.
BTE (m)
CTE-R2 (m)
CTE-R3 (m)
Model
n
Mean
Std
Min
Max
n
Mean
Std
Min
Max
n
Mean
Std
Min
Max
Driving-specific VLAs
UniDrive-VLA [ 24 ]
815
0.68
0.60
0.02
4.51
209
2.90
1.74
0.14
8.06
791
1.96
1.60
0.04
10.49
RecogDrive [ 23 ]
815
1.19
0.65
0.14
6.63
209
4.30
2.21
0.44
11.66
791
1.97
1.41
0.22
9.77
ImpromptuVLA-7B [ 11 ]
814
1.27
1.58
0.00
18.45
203
5.69
4.58
0.00
19.63
781
2.97
2.49
0.00
15.40
Alpamayo-1.5-10B [ 36 ]
815
1.31
1.25
0.00
6.79
209
2.70
2.16
0.01
11.90
791
1.70
1.50
0.00
10.25
Table 3: Trajectory error comparison of models (ADE in metres). Best mean ADE results are in bold. See section 5.1 for more details.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Examples
EGO
The ego vehicle itself (unique, exactly one per scene)
OTHER_AGENT
Cars, trucks, buses, pedestrians, cyclists, active emergency vehicles - any entity whose future state can change by its own volition
TRAFFIC_CONTROL
Signals, signs, stop lines, crosswalk markings - entities that impose a codified traffic regulation
ROAD_OBSTACLE
Construction cones, temporary barriers, debris - inert, localised entities requiring only spatial avoidance
Appendix
Table 4: Node categories in the causal scene graph, derived from the decision tree in Figure 4 a.
Effect type
Description
D(e)
GRANTS_PROCEED
Signal or sign permitting ego to enter a region (discrete, permits)
s,a
INHIBITS_PROCEED
Entity blocking or prohibiting ego from entering a region: yielding, obstruction, gap waiting (discrete, prohibits)
s,a
CONSTRAINS_LONGITUDINAL
Entity modulating ego’s speed or acceleration within its current trajectory: lead-vehicle following, speed limiting (continuous, speed axis)
s,a
CONSTRAINS_LATERAL
Entity modulating ego’s heading or lane position: lateral bounding, evasive steering (continuous, heading axis)
h
Appendix
Table 5: Ego edge effect types, derived from the decision tree in Figure 4 b.
Figure 4: Both taxonomies are derived from binary decision trees over objectively testable properties. Node types split on agency and rule imposition (after the unique ego split). Edge types split on discrete vs. continuous control and axis or polarity. Each tree is exhaustive and produces mutually exclusive leaves, ensuring both taxonomies are closed by construction , not by enumeration of driving scenarios.
Type
Rung
Format
Description
Active-element types: CaLM causal scenarios over active subgraphs
CaI
0
M
Causal Identification. Identify which visible element is currently constraining ego.
CA
0
M+B
Causal Attribution. Trace a chain of causal influence from a distal element through a mediator to ego.
CB
1
B
Collider Bias. Apply explaining-away: when two independent causes share ego as a collider, learning one reduces the necessity of the other.
CoI
2
M
Confounder Identification. Identify a shared cause that links multiple active influences on ego.
BAS
2
M
Backdoor Adjustment Set. Identify the complete set of variables to control for when isolating one element’s effect on ego.
Appendix
Table 6: Question types in CausalDriveBench. Types marked † are novel contributions introduced in this work; the remaining 12 instantiate established CaLM causal scenarios in the embodied driving setting. Format codes: B (binary), M (multiple-choice), M+B (both formats supported).
Structure
Discovery (R0)
Association (R1)
Intervention (R2)
Counterfactual (R3)
Direct ( A→ ego)
CaI
–
–
CR, SC
Chain ( A→B→ ego)
CA
–
CDE
NIE, NDE
Confounding ( A→ { B→ ego})
CaI
–
CoI, BAS, CDE
CR, NIE, NDE, SC
Collider ( A→ ego ←B )
–
CB
–
NC
Appendix
Table 7: Active question type taxonomy in CausalDriveBench mapped to causal graph structures and Pearl’s rungs of inference. Empty cells indicate the graph structure does not produce a non-trivial question at that rung. Example, a direct graph has no mediator for Rung-2 mediation
Figure 5: Distribution of QA pairs in CausalDriveBench across the full unfiltered set of 3,189 verified scenes (22,952 QA pairs), shown by (a) question type, (b) PCH layer, (c) difficulty, and (d) entity status. In (a), warm hues mark active-element types, sages mark dormant types, and blues/purples mark distractor types; (d) gives the per-family aggregate using each family’s anchor hue. Compared to the diversity-sampled subset ( fig. 2 ), the per-category and per-rung shares are nearly identical, indicating that the proportional sampling at rate 0.5 within Stage 2 clusters preserves the question-type composition of the source pool while reducing it to 1,887 representative scenes.
Figure 6: Effect of Stage 2 diversity sampling on benchmark composition.
Graph Structure
QA pairs
% of total
Direct ( A→ ego)
1,646
22.6
Collider ( A→ ego ←B )
359
4.9
Confounding ( C→ ego, C→B→ ego)
286
3.9
Chain ( A→B→ ego)
127
1.7
No active structure (dormant + distractor)
4,867
66.8
Total
7,285
100.0
Appendix
Table 8: Distribution of QA pairs by underlying active-element graph structure. Dormant and distractor questions are aggregated as “no active structure” since they are generated outside the active-edge enumeration.
Property
Count
%
Format
Binary
6,072
83.3
Multiple choice
1,213
16.7
Difficulty
Easy
1,145
15.7
Medium
3,146
43.2
Appendix
Table 9: Format and difficulty distribution.
Figure 7: Level-1 clustering kernels visualised on the joint embedding. Two-dimensional t-SNE of the combined hierarchical-clustering kernel Kcombined=0.5,K1+0.5,K2 over n=3,189 verified scene samples, recoloured by each of the four Level-1 categorical axes that the K1 kernels hash on. (top-left) dominant active causal-source category (signal / vehicle / vulnerable road user / etc.); (top-right) dormant-edge link status – whether the sample has any dormant ego-edge and, if so, whether it connects to an active cause via an inter-node edge ( linked ) or not ( isolated ); (bottom-left) dominant distractor category from the graph’s rejected-cause list; (bottom-right) the dominant active-subgraph topology (empty / direct / chain / collider / confounding). The visible structure within and across panels indicates that the Level-1 axes capture complementary aspects of causal-graph diversity rather than redundant ones.
Figure 8: Level-2 clustering kernels visualised on the joint embedding. Two-dimensional t-SNE of the combined hierarchical-clustering kernel Kcombined=0.5K1+0.5K2 over n=3,189 verified scene samples, recoloured by each of the four Level-2 categorical axes that the K2 kernels embed. (top-left) ego speed-trajectory class derived from the per-sample state.json future trajectory (stopped / decelerating / cruising / accelerating); (top-right) weather condition parsed from the nuScenes scene.description field; (bottom-left) the ego’s navigation intent at T=0 ( go straight , turn left , turn right ); (bottom-right) road-geometry context with junction temporal phase as the primary axis (approaching / at / past) and lane structure as the secondary axis. Together with Figure 7 , the panels show that scene-context diversity is largely orthogonal to the causal-graph structural diversity captured by the Level-1 kernels.
Edit operation
Count
Node → distractor demotion
1,739
Ego-edge removed
2,054
Inter-node edge removed
1,446
Ego-edge added
527
Ego-edge causal-status string changed
338
Ego-edge active flag toggled
327
Appendix
Table 10: Per-operation edit counts across the 3,189 reviewed samples. Operations are not mutually exclusive: a single review session may apply multiple edits to one graph.
Decision
Metric
Value
Causal status (3-class, overall)
κ
0.81
Active vs rest
κ
0.87
Dormant vs rest
κ
0.63
Distractor vs rest
κ
0.91
Ego edge effect type (4-class)
κ
0.85
Inter-node edge type (4-class)
κ
0.75
Appendix
Table 11: Inter-annotator agreement on causal labels over 50 random samples. Cohen’s κ reported per decision type. Landis-Koch interpretation: 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect.
Model
CaI
CA
CB
CoI
BAS
CDE
CR
SC
NC
NDE
NIE
DQ
EL
AB
DR
CI
NI
WC
Driving-specific VLAs
ImpromptuVLA-7B
87.5
71.4
26.9
91.5
90.9
69.4
67.6
59.0
70.9
78.3
52.0
51.4
72.4
69.8
21.7
58.6
93.0
70.1
OpenREAD
87.9
87.8
45.6
93.2
90.9
67.7
61.7
64.7
71.4
52.2
58.0
47.9
72.9
63.2
76.3
41.0
86.6
56.7
Orion
0.0
0.0
12.1
0.0
0.0
43.6
65.1
14.7
55.9
54.4
36.0
59.9
46.3
52.4
0.0
94.1
79.9
63.0
UniDrive-VLA
82.4
89.8
58.4
98.3
95.5
61.3
49.4
66.5
65.7
58.7
60.0
37.1
60.6
65.8
83.5
15.4
68.0
63.2
RecogDrive
87.6
100.0
44.0
88.7
90.5
62.7
55.2
78.1
58.6
65.9
66.7
24.1
61.7
63.2
48.4
29.1
54.7
57.8
Appendix
Table 12: Per-question-type accuracy (%) across the 18 canonical CausalDriveBench types. Bold = best per column. Driving VLAs upper block, general VLMs lower block.
Model
Active overall
Direct
Chain
Confounding
Collider
Driving-specialist VLAs
RecogDrive
69.9
73.2
74.0
70.9
52.5
OpenREAD
69.8
71.0
68.5
74.8
60.7
ImpromptuVLA-7B
69.1
71.1
72.4
76.2
52.7
UniDrive-VLA
65.9
65.0
68.5
74.5
62.1
SafeAuto
56.6
55.9
68.5
69.9
45.4
Appendix
Table 13: Active-element accuracy (%) by graph topology. Dormant and distractor questions have empty graph structure by construction. Bold = best per column.
Model
Composite
Spatial
Mechanism
Conditionality
Consistency
Driving-specialist VLAs
RecogDrive
1.25
1.36
1.40
0.75
1.48
OpenREAD
1.18
1.14
1.30
0.83
1.44
UniDrive-VLA
1.12
1.21
1.24
0.74
1.31
ImpromptuVLA-7B
1.06
1.08
1.12
0.63
1.39
SimLingo
0.71
0.91
0.70
0.30
0.93
Appendix
Table 14: LLM-judge reasoning scores (0–2 per dimension; higher is better), averaged over Claude Sonnet 4.6 and ChatGPT 4o. Composite is the mean of the four dimensions. Bold = best per column. Driving VLAs upper block, general VLMs lower block.
Dimension
Pearson r
Spearman ρ
Composite
0.97
0.92
Spatial grounding
0.96
0.93
Mechanism identification
0.99
0.97
Conditionality awareness
0.94
0.94
Answer consistency
0.83
0.81
Appendix
Table 15: Inter-judge agreement on per-model reasoning scores between Claude Sonnet 4.6 and ChatGPT 4o across the 13 evaluated models, computed per dimension over the 13 model-level scores.
Model
CB (Active)
EL (Dormant)
CI (Distractor)
Overall
Driving-specialist VLAs
ImpromptuVLA-7B
26.85%
72.38%
58.64%
69.27%
OpenREAD
45.64%
72.93%
40.95%
63.76%
Orion
12.08%
46.33%
94.08%
57.25%
UniDrive-VLA
58.39%
60.58%
15.39%
54.87%
RecogDrive
43.97%
61.69%
29.14%
54.25%
Appendix
Table 16: Rung 1 accuracy decomposed by question source (%). CB targets active-graph collider structures; EL targets dormant entities and probes structural decoupling, shielding, or shared cause; CI targets perceptually salient distractors. Driving VLAs upper block, general VLMs lower block; each block sorted by overall accuracy.
Model
Total CTE
GOOD_CF (%)
INCONCLUSIVE (%)
FLAGGED (%)
TRAJ_SAME_BTE (%)
Driving-specialist VLAs
WiseAD
972
48.66
33.85
17.49
17.49
UniDrive-VLA
1000
20.10
37.90
42.00
42.00
Alpamayo-1.5-10B
1000
16.80
2.50
63.50
34.00
ImpromptuVLA-7B
984
15.04
45.53
42.28
22.97
Omnidrive
999
10.71
42.64
46.65
46.65
Appendix
Table 17: CTE grade and flag distribution. GOOD_CF fires when the counterfactual trajectory differs meaningfully from BTE in the prescribed direction. TRAJ_SAME_BTE fires when the counterfactual is essentially unchanged from BTE. Percentages are relative to total CTE count. Flags can co-occur, so columns need not sum to 100%. Driving VLAs upper block, general VLMs lower block; each block sorted by GOOD_CF . Bold = best per column ( GOOD_CF higher is better, others lower). SafeAuto’s grade and flag fields were not retained in its trajectory report. Alpamayo-1.5-10B’s high FLAGGED share is dominated by an ANSWER_MISMATCH flag (57.40%). Cosmos-Reason-2’s rerun pipeline emits a GOOD / EXCELLENT grade taxonomy rather than INCONCLUSIVE / FLAGGED , so the four columns shown read 0%; the full grade breakdown for Cosmos is GOOD =6.60%, EXCELLENT =8.10%.
Model
BTE (m)
CTE-R2 (m)
CTE-R3 (m)
CTE-R2/BTE
CTE-R3/BTE
Regime
Driving-specialist VLAs
WiseAD
2.91
1.97
1.57
0.68
0.54
C (positive)
UniDrive-VLA
0.68
2.90
1.96
4.26
2.88
B
Alpamayo-1.5-10B
1.31
2.70
1.70
2.06
1.30
B+C
ImpromptuVLA-7B
1.27
5.69
2.97
4.48
2.34
B
Omnidrive
5.39
4.51
2.48
0.84
0.46
C
Appendix
Table 18: CTE failure-mode regime classification. A = frozen, B = wild-swerving, C = default-to-stop. Some models exhibit characteristics of more than one regime (B+C: wild on R2 and stop-collapsing on R3; A+C: frozen on R2 and stop-collapsing on R3). WiseAD is the single positive outlier with proportionate counterfactual differentiation in the prescribed direction.
Model
Language backbone
Source
Driving-specific VLAs
ImpromptuVLA-7B
Qwen2.5-VL-7B
[ 11 ] §2
OpenREAD
Qwen3-VL-8B ∗
[ 43 ] §3.1
UniDrive-VLA
Qwen3-VL-8B ∗
[ 24 ] Tab. 3
RecogDrive
InternVL3-8B ∗
[ 23 ] App. E
Alpamayo-1.5-10B
Cosmos-Reason ∗
[ 36 ] §3.1
Appendix
Table 19: Language backbone of each evaluated model. Driving VLAs in the upper block, general VLMs in the lower block. Models marked ∗ appear as the backbone of one or more driving VLAs in the upper block.
Backbone
Vanilla VLM (%)
Driving VLA
Driving VLA (%)
Δ (pp)
Qwen3-VL-8B
66.04
OpenREAD
63.76
−2.3
Qwen3-VL-8B
66.04
UniDrive-VLA
54.87
−11.2
InternVL3-8B
62.91
RecogDrive
54.25
−8.7
Cosmos-Reason
70.56
Alpamayo-1.5-10B
36.86
−33.7
Appendix
Table 20: Overall causal QA accuracy when the same language backbone is post-trained for general-purpose use (left) versus driving (right). Δ measures the cost of driving-specific post-training in percentage points.
NAVSIM PDMS (collisions, drivable area, comfort) only
Diffusion planner only (DiffGRPO; VLM frozen)
−8.7
OpenREAD
LLM-as-critic on QA semantic match + embedding similarity + trajectory ADE
Whole VLA (GRPO)
−2.3
Appendix
Table 21: RFT reward design across the three RFT-trained driving VLAs in our roster, alongside the resulting same-backbone causal-QA degradation. The reward composition column lists the components of the RFT reward signal; the policy column indicates which model parameters are updated during RFT.
Figure 9: Scene used for the attention-masking probe. The ego vehicle is accelerating straight through a green signal at an open intersection, with the multi-view camera frames and ego-history tokens encoding this factual configuration. This scene is used as the substrate for both the reasoning-on injection sweep ( fig. 10 ) and the reasoning-off ablation sweep ( fig. 11 ).
Inputs
Driving VLAs
General-purpose VLMs
V
E
R
OpenREAD
AlpaMayo-1.5
Qwen3-VL-8B-Thinking
Cosmos-Reason2-8B
Reasoning on with injected R′ : spread of T′ across counterfactual rationales
✓
✓
✓
No effect
No effect
No effect
No effect
✓
✗
✓
No effect
No effect
No effect
No effect
✗
✓
✓
Some effect
No effect
Hallucinates
Hallucinates
✗
✗
✓
Good effect
No effect
Hallucinates
Hallucinates
Appendix
Table 22: Qualitative summary of the attention-masking outcomes across the eight (V,E,R) configurations for the two driving VLAs and the two general-purpose VLMs. Reasoning-on rows report whether the five injected counterfactual rationales R′ cause the predicted trajectory T′ to fan out into qualitatively distinct curves (more ⇒ greater rationale influence). Reasoning-off rows report how far the predicted T′ deviates from ground-truth T when the rationale is ablated (less ⇒ closer to T ). Hallucinates marks configurations where the trajectory collapses to degenerate, near-zero, or unstructured output.
Figure 10: Reasoning-on regime with rationale injection. Predicted trajectories for the un-masked baseline (teal) and the five injected counterfactual rationales cautious , emergency , aggressive , turn-left , turn-right (coloured) versus nuScenes ground truth (dashed black) under the four (V,E) mask configurations across the four evaluated models. A faithfully-conditioned trajectory head should produce a visible fan of five distinct curves with emergency decelerating and turn-left and turn-right bending laterally. For AlpaMayo-1.5 the curves remain bunched in every configuration, including V : off , E : off ; for OpenREAD a visible fan emerges only once vision is masked, and grows further once ego history is also removed. Qwen3-VL and Cosmos-Reason collapse to degenerate outputs rather than fanning, indicating that the rationale never structurally drives their trajectory head either.
Figure 11: Reasoning-off regime with the rationale ablated. Predicted trajectories (teal) versus nuScenes ground truth (dashed black) under the four (V,E) mask configurations across the four evaluated models. Removing the rationale entirely while preserving vision and ego history leaves the prediction nearly aligned with ground truth for both driving VLAs, confirming that the rationale is not a load-bearing input for trajectory generation. Deviations escalate only when vision is also removed, and degenerate when all three modalities are masked.
R0
R1
R2
R3
Model
ER
L
DCKA
ER
L
DCKA
ER
L
DCKA
ER
L
DCKA
Cosmos-Reason-2 (VLM)
55.8
0.45
—
43.4
0.51
—
57.7
0.48
—
80.4
0.48
—
Alpamayo-1.5-10B (VLA)
1.41
0.029
0.124
2.25
0.026
0.047
3.75
0.026
0.113
78.1
0.026
0.411
Qwen3-VL-8B (VLM)
37.3
0.30
—
14.0
0.16
—
16.4
0.15
—
31.5
0.13
—
OpenREAD (VLA)
47.4
0.46
0.014
27.8
0.58
0.014
33.2
0.57
0.036
62.3
0.61
0.265
Appendix
Table 23: Terminal-block (layers 27 – 36 ) summary statistics for the two parent–child pairs across the four rungs of the benchmark. Each cell reports the mean over the ten terminal layers. ER is effective rank (higher = more directions used); L is Fisher–Rao length between consecutive layers (higher = more change in next-token prediction); DCKA is cross-model CKA distance (higher = more divergence from the parent). Alpamayo’s terminal block sits at the floor on R0 – R2 and recovers on R3 ; OpenREAD’s terminal block exceeds the parent on every metric, on every rung.
Figure 12: Layer-wise representational metrics across R0 – R3 for the two parent–child pairs. Rows in each 3×4 panel: per-layer Fisher–Rao length between consecutive layers, effective rank, cross-model DCKA . Columns: R0 (association), R1 , R2 (intervention), R3 (counterfactual). Shaded: terminal block, layers 27 – 36 . (a) On R0 – R2 Alpamayo’s terminal block is collapsed: ER and Fisher–Rao length sit at the floor, and DCKA is small. The terminal layers have stopped contributing to the next-token prediction and lie on a near- 1 D subspace inside the parent’s dominant directions. On R3 ER climbs back to parent levels and DCKA rises sharply, but Fisher–Rao length stays low: the terminal block has high-rank activations that no longer translate into changes in the predicted output. (b) OpenREAD’s terminal effective rank and Fisher–Rao length are at or above the parent on every rung, and DCKA grows monotonically with depth, with the magnitude of divergence increasing from R0 to R3 . The contrast between (a) and (b) shows that the collapse is recipe-specific.
Figure 13: Dataset-wide review dashboard. The page lists all 850 nuScenes scenes alongside their per-sample review status. Each scene row carries four sample chips (S0–S3), with chip colour encoding review state (green for Good, amber for Needs Review or in-progress, grey for filtered). The header surfaces aggregate stat cards (scenes, samples, QA pairs, counterfactual trajectories, alternative-cause trajectories) and a one-click bulk download of the entire reviewed dataset.
Figure 14: Per-sample editing interface in Edit Mode. Reviewers can change a node’s Type, Category, Label, State, Position, and per-camera visibility, or remove it entirely; node-level edits are tracked through an undo / redo stack and committed as a versioned graph.json . The Original / Pruned toggle compares the extractor’s raw output against the post-edit graph, the Active/Dormant and Distractors sub-tabs mirror the causal-status partition of section A.2 , and the Needs Review / Good buttons in the top bar set the sample’s dashboard-level status.
Figure 15: Per-sample read-only inspection view. The header lists the six JSON artefacts ( meta , frames , calib , state , graph , qa ) and node, ego-edge, inter-edge, and timestep counts for the sample. The middle panel renders the four-timestep BEV stack (top row) and the four ego-centric camera views (FRONT, FRONT_LEFT, FRONT_RIGHT, BACK) with all node bounding boxes overlaid. The bottom panel renders the extracted causal scene graph: solid arrows are directed ego edges colour-coded by effect type, and dashed bidirected arrows are CONFOUNDED_BY links between non-ego nodes encoding shared unobserved causes.
Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.
Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems.
Yun Li, Yidu Zhang, Simon Thompson +2
Graduate School of Information Science and Technology, The University of Tokyo · TIER IV, Inc.
We present the first systematic study of faithfulness in Vision-Language-Action (VLA) driving models, analyzing 300 Alpamayo-R1-10B inferences across 100 diverse PhysicalAI-AV scenarios. Our main finding is that output natural-language rationales with trajectories may be significantly unfaithful: (i) overall reasoning fidelity is only 42.5%, with Chain-of-Causation matching scene reality less than half the time; (ii) 94 missed pedestrians in one-third of pedestrian-relevant scenes; (iii) 97.7% trajectory fragility under mild visual perturbations; and (iv) only 48.3% mean reasoning-action consistency, with 53.3% of inferences exhibiting low consistency, including 37.9% of stop-claimed cases where the model continues instead. We formalize faithfulness information-theoretically, define entity and action fidelity with verification criteria, and outline a four-component safety architecture aligned with these results.
Nicanor Mayumu, Xiaoheng Deng, Patrick Mukala
School of Computer Science and Engineering Central South University 932 Lushan Road, Changsha, Hunan P. R. China · School of Computer Science University of Wollongong in Dubai UOWD Building, Dubai Knowledge Park Dubai, United Arab Emirates