Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Yun Li; Yidu Zhang; Simon Thompson; Ehsan Javanmardi; Manabu Tsukada

cs.ROApr 2, 2026

Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Authors: Yun Li, Yidu Zhang, Simon Thompson, Ehsan Javanmardi, Manabu Tsukada

Organizations: Graduate School of Information Science and Technology, The University of Tokyo · TIER IV, Inc.

Abstract

Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems.

Figures & tables

Figure 1: Three-stage framework overview. CSN (Stage 2) restructures driving-environment information into structured text, while the runtime safety supervisor (Stage 3) monitors VLA waypoints against $\mathcal{S}_{\text{sem}}$ . Both modules operate on CPU.

Figure 2: The text structure bottleneck. (a) Template text presents intent

\mathcal{I}

and constraints

\mathcal{C}

as independent fragments with no

\mathcal{I}\to\mathcal{C}

link. (b) CSN restores the causal link via explicit connectives ( BUT , BEFORE ).

Figure 3: CSN pipeline illustrated on a left-turn scenario. Each principle progressively transforms flat template text into quantitative, structured, and causally linked input.

Approach	Level	VRAM	Retrain	DS Gain	Key Mechanism
LMDrive template ( Shao et al., 2024 )	L0	0	–	baseline	Isolated instruction + notice
TLS-Assist ( Schmidt et al., 2025 )	L1	0	No	+14.1%	Structured signal messages
GraphPilot ( Schmidt et al., 2026 )	L2	0	No	+15.6%	Entity-relationship graph text
SteerVLA ( Gao et al., 2026 )	L2	0	Yes	+4.77 pts	Fine-grained meta-actions
DriveVLM CoT ( Tian et al., 2024 )	L3	High	Yes	–	3-stage causal chain
DriveLM graph QA ( Sima et al., 2024 )	L3	High	Yes	–	Graph dependency chains

Table 1: Taxonomy of VLA text approaches by causal structure level. DS Gain values are from each work’s own evaluation setup and are not directly comparable across rows due to differing routes, weather, and CARLA versions.

(a) Template (LMDrive original)
Instruction	Turn left at intersection.
Notice	Watch out for pedestrians.
(b) +Flat Text (same facts, no causal links)
Instruction	Turn left. Speed 25 km/h. Pedestrian 5m right crossing left. Sedan 12m ahead 30 km/h. RED light. Junction 20m.
(c) CSN (causal structure)
Instruction	Turn left at intersection, BUT yield to pedestrian crossing from right at 5m BEFORE executing turn. Maintain distance from sedan ahead.

Table 2: Text input comparison for a left-turn scenario. Causal connectives shown in bold.

Figure 4: Evaluation environments. (a) Town01, clear day. (b) Town03, heavy rain. (c) Town05, night. (d) Town07, dense fog.

Configuration	DS ( $\uparrow$ )	RC ( $\uparrow$ )	IS ( $\uparrow$ )	$\Delta$ DS ${}_{\text{orig}}$	$\Delta$ DS ${}_{\text{base}}$
LMDrive (original)	32.54 $\pm$ 3.00	48.3 $\pm$ 2.6%	0.729 $\pm$ 0.034	—	—
+ CSN	42.67 $\pm$ 2.74	56.5 $\pm$ 1.7%	0.787 $\pm$ 0.028	+31.1%	—
+ Flat Text	38.71 $\pm$ 1.44	48.7 $\pm$ 1.9%	0.828 $\pm$ 0.025	+18.9%	—
+ Semantic Safety	34.10 $\pm$ 2.25	45.5 $\pm$ 2.1%	0.785 $\pm$ 0.038	+4.8%	—
+ PL-DPO-NLL	32.49 $\pm$ 3.34	44.9 $\pm$ 2.7%	0.754 $\pm$ 0.021	$-$ 0.1%	—
+ TTC Safety	22.02 $\pm$ 4.07	33.7 $\pm$ 3.4%	0.658 $\pm$ 0.037	$-$ 32.3%	$-$ 32.2%

Table 3: Multi-town ablation (16 routes, 8 towns,

N{=}5

). Values are mean

\pm

95% bootstrap CI. Best bold , second underlined ; ties share marking.

Figure 5: Driving Score comparison across all ten configurations (

N{=}5

). Left and center: mean DS with 95% bootstrap CI. Right: raincloud plot showing KDE density (shaded), individual runs (dots), mean (diamond), and 95% CI (whisker). Dotted line: original LMDrive baseline.

Figure 6: Decomposition of CSN’s DS improvement into information content and causal structure contributions on both weight configurations.

Noise Level	Dist. $\sigma$	Speed	Miss Rate	DS	95% CI
Clean (privileged)	0 m	0%	0%	$42.7\pm 3.3$	$[39.9,\;45.4]$
Mild	$\pm$ 1 m	$\pm$ 10%	0%	$45.2\pm 2.9$	$[42.6,\;47.8]$
Moderate	$\pm$ 2 m	$\pm$ 20%	0%	$45.4\pm 2.7$	$[43.1,\;47.6]$
Severe	$\pm$ 5 m	$\pm$ 20%	10%	$46.0\pm 4.1$	$[42.5,\;49.6]$
Extreme	$\pm$ 5 m	$\pm$ 30%	20%	$45.1\pm 2.7$	$[42.7,\;47.4]$

Table 4: Perception noise ablation on Original LMDrive + CSN (

N{=}5

). We inject Gaussian noise on distance, multiplicative noise on speed, and random actor miss rates. All noise-level CIs overlap with the clean baseline, indicating no statistically significant degradation.

Explore similar work

CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving
Sep 26, 2026Narendiran Chembu, Navvrat Rao, Shreedhar Shreeshail Kodate +10Reasoning for VLA Models Vision-Language-Action Models
Fastcode AI · Indian Institue of Technology Delhi
SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
Oct 14, 2026Shaoyuan Xie, Zihan Zhang, Jingxuan Wang +6Autonomous Driving End-to-End Autonomous Driving
UC Irvine · UC San Diego · University of Cincinnati +1
Teaching Vision-Language-Action Models What to See and Where to Look
Jul 2, 2026Yuguang Yang, Canyu Chen, Zhewen Tan +10End-to-End Autonomous Driving VLM Distillation
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5

Sep 26, 2026J/K move · Enter open · S save

CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving

Narendiran Chembu, Navvrat Rao, Shreedhar Shreeshail Kodate, Gayatri Srujana Banda+9

Fastcode AI · Indian Institue of Technology Delhi

Reasoning for VLA Models Vision-Language-Action Models Autonomous Driving Benchmarks Counterfactual Evaluation of VLMs

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoning reflects the causal structure of the scene remains untested. We introduce CausalDriveBench, an evaluation framework grounded in Pearl's Causal Hierarchy (PCH) that tests causal reasoning in driving-specific VLAs through structured visual question answering (QA) and alternative-trajectory prediction. To this end, we construct causal scene graphs over nuScenes that distinguish causally active, dormant, and distractor entities, separating perceptual salience from causal relevance. The benchmark spans all four rungs of PCH (association, intervention, and counterfactual along with causal discovery) for QA generation. For the higher rungs, we additionally provide reference trajectories under specified scene modifications, enabling action-level verification that complements reasoning-level evaluation. In total, the benchmark contains 7,285 verified causal QA pairs and 1,000 counterfactual trajectories derived from nuScenes. We evaluate 10 driving-specific VLAs and 3 general-purpose VLMs, and report three findings. First, the best model reaches only 70.6% QA accuracy, and 4 of 13 models score below random chance. Second, comparing each driving VLA to the general-purpose VLM that shares its language backbone, the cost of driving fine-tuning ranges from 2 to 34 percentage points on causal QA, with post-training design explaining the spread. Third, causal QA and trajectory accuracy are statistically uncorrelated across models: under counterfactual prompts, predicted trajectories either over-react or collapse onto the observed-scene baseline. Taken together, these results show that neither fluent rationales nor accurate observed-scene trajectories constitute evidence of causal understanding.

Abstract

Explore similar work

CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving

SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving

Teaching Vision-Language-Action Models What to See and Where to Look