Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Yun Li; Yidu Zhang; Simon Thompson; Ehsan Javanmardi; Manabu Tsukada

cs.ROApr 2, 2026

Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Authors: Yun Li, Yidu Zhang, Simon Thompson, Ehsan Javanmardi, Manabu Tsukada

Organizations: Graduate School of Information Science and Technology, The University of Tokyo · TIER IV, Inc.

Abstract

Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems.

Figures & tables

Figure 1: Three-stage framework overview. CSN (Stage 2) restructures driving-environment information into structured text, while the runtime safety supervisor (Stage 3) monitors VLA waypoints against $\mathcal{S}_{\text{sem}}$ . Both modules operate on CPU.

Figure 2: The text structure bottleneck. (a) Template text presents intent

\mathcal{I}

and constraints

\mathcal{C}

as independent fragments with no

\mathcal{I}\to\mathcal{C}

link. (b) CSN restores the causal link via explicit connectives ( BUT , BEFORE ).

Figure 3: CSN pipeline illustrated on a left-turn scenario. Each principle progressively transforms flat template text into quantitative, structured, and causally linked input.

Approach	Level	VRAM	Retrain	DS Gain	Key Mechanism
LMDrive template ( Shao et al., 2024 )	L0	0	–	baseline	Isolated instruction + notice
TLS-Assist ( Schmidt et al., 2025 )	L1	0	No	+14.1%	Structured signal messages
GraphPilot ( Schmidt et al., 2026 )	L2	0	No	+15.6%	Entity-relationship graph text
SteerVLA ( Gao et al., 2026 )	L2	0	Yes	+4.77 pts	Fine-grained meta-actions
DriveVLM CoT ( Tian et al., 2024 )	L3	High	Yes	–	3-stage causal chain
DriveLM graph QA ( Sima et al., 2024 )	L3	High	Yes	–	Graph dependency chains

Table 1: Taxonomy of VLA text approaches by causal structure level. DS Gain values are from each work’s own evaluation setup and are not directly comparable across rows due to differing routes, weather, and CARLA versions.

(a) Template (LMDrive original)
Instruction	Turn left at intersection.
Notice	Watch out for pedestrians.
(b) +Flat Text (same facts, no causal links)
Instruction	Turn left. Speed 25 km/h. Pedestrian 5m right crossing left. Sedan 12m ahead 30 km/h. RED light. Junction 20m.
(c) CSN (causal structure)
Instruction	Turn left at intersection, BUT yield to pedestrian crossing from right at 5m BEFORE executing turn. Maintain distance from sedan ahead.

Table 2: Text input comparison for a left-turn scenario. Causal connectives shown in bold.

Figure 4: Evaluation environments. (a) Town01, clear day. (b) Town03, heavy rain. (c) Town05, night. (d) Town07, dense fog.

Configuration	DS ( $\uparrow$ )	RC ( $\uparrow$ )	IS ( $\uparrow$ )	$\Delta$ DS ${}_{\text{orig}}$	$\Delta$ DS ${}_{\text{base}}$
LMDrive (original)	32.54 $\pm$ 3.00	48.3 $\pm$ 2.6%	0.729 $\pm$ 0.034	—	—
+ CSN	42.67 $\pm$ 2.74	56.5 $\pm$ 1.7%	0.787 $\pm$ 0.028	+31.1%	—
+ Flat Text	38.71 $\pm$ 1.44	48.7 $\pm$ 1.9%	0.828 $\pm$ 0.025	+18.9%	—
+ Semantic Safety	34.10 $\pm$ 2.25	45.5 $\pm$ 2.1%	0.785 $\pm$ 0.038	+4.8%	—
+ PL-DPO-NLL	32.49 $\pm$ 3.34	44.9 $\pm$ 2.7%	0.754 $\pm$ 0.021	$-$ 0.1%	—
+ TTC Safety	22.02 $\pm$ 4.07	33.7 $\pm$ 3.4%	0.658 $\pm$ 0.037	$-$ 32.3%	$-$ 32.2%

Table 3: Multi-town ablation (16 routes, 8 towns,

N{=}5

). Values are mean

\pm

95% bootstrap CI. Best bold , second underlined ; ties share marking.

Figure 5: Driving Score comparison across all ten configurations (

N{=}5

). Left and center: mean DS with 95% bootstrap CI. Right: raincloud plot showing KDE density (shaded), individual runs (dots), mean (diamond), and 95% CI (whisker). Dotted line: original LMDrive baseline.

Figure 6: Decomposition of CSN’s DS improvement into information content and causal structure contributions on both weight configurations.

Noise Level	Dist. $\sigma$	Speed	Miss Rate	DS	95% CI
Clean (privileged)	0 m	0%	0%	$42.7\pm 3.3$	$[39.9,\;45.4]$
Mild	$\pm$ 1 m	$\pm$ 10%	0%	$45.2\pm 2.9$	$[42.6,\;47.8]$
Moderate	$\pm$ 2 m	$\pm$ 20%	0%	$45.4\pm 2.7$	$[43.1,\;47.6]$
Severe	$\pm$ 5 m	$\pm$ 20%	10%	$46.0\pm 4.1$	$[42.5,\;49.6]$
Extreme	$\pm$ 5 m	$\pm$ 30%	20%	$45.1\pm 2.7$	$[42.7,\;47.4]$

Table 4: Perception noise ablation on Original LMDrive + CSN (

N{=}5

). We inject Gaussian noise on distance, multiplicative noise on speed, and random actor miss rates. All noise-level CIs overlap with the clean baseline, indicating no statistically significant degradation.

Explore similar work

Sep 26, 2026

CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving

Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoning reflects the causal structure of the scene remains untested. We introduce CausalDriveBench, an evaluation framework grounded in Pearl's Causal Hierarchy (PCH) that tests causal reasoning in driving-specific VLAs through structured visual question answering (QA) and alternative-trajectory prediction. To this end, we construct causal scene graphs over nuScenes that distinguish causally active, dormant, and distractor entities, separating perceptual salience from causal relevance. The benchmark spans all four rungs of PCH (association, intervention, and counterfactual along with causal discovery) for QA generation. For the higher rungs, we additionally provide reference trajectories under specified scene modifications, enabling action-level verification that complements reasoning-level evaluation. In total, the benchmark contains 7,285 verified causal QA pairs and 1,000 counterfactual trajectories derived from nuScenes. We evaluate 10 driving-specific VLAs and 3 general-purpose VLMs, and report three findings. First, the best model reaches only 70.6% QA accuracy, and 4 of 13 models score below random chance. Second, comparing each driving VLA to the general-purpose VLM that shares its language backbone, the cost of driving fine-tuning ranges from 2 to 34 percentage points on causal QA, with post-training design explaining the spread. Third, causal QA and trajectory accuracy are statistically uncorrelated across models: under counterfactual prompts, predicted trajectories either over-react or collapse onto the observed-scene baseline. Taken together, these results show that neither fluent rationales nor accurate observed-scene trajectories constitute evidence of causal understanding.

Narendiran Chembu, Navvrat Rao, Shreedhar Shreeshail Kodate +10

Fastcode AI · Indian Institue of Technology Delhi

Reasoning for VLA Models Vision-Language-Action Models Autonomous Driving Benchmarks Counterfactual Evaluation of VLMs

Ask PDF

Oct 14, 2026

SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving

Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first benchmark suite that evaluates navigation following and conflict awareness together, along with adversarial robustness. Existing VLAs fall short on both: SimLingo completes no language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant. Video demonstrations are available on our project page.

Shaoyuan Xie, Zihan Zhang, Jingxuan Wang +6

UC Irvine · UC San Diego · University of Cincinnati +1

Autonomous Driving End-to-End Autonomous Driving Robustness of VLA Models Safety Evaluation of VLA Models World Models for Autonomous Driving Autonomous Driving Benchmarks

Ask PDF

Jul 2, 2026

Teaching Vision-Language-Action Models What to See and Where to Look

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Yuguang Yang, Canyu Chen, Zhewen Tan +10

School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5

End-to-End Autonomous Driving VLM Distillation Vision-Language-Action Models Autonomous Driving Planning Vehicle Trajectory Prediction

Ask PDF