Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Yun Li; Yidu Zhang; Simon Thompson; Ehsan Javanmardi; Manabu Tsukada

cs.ROApr 2, 2026

Causal Scene Narration with Runtime Safety Supervision for Vision-Language-Action Driving

Authors: Yun Li, Yidu Zhang, Simon Thompson, Ehsan Javanmardi, Manabu Tsukada

Organizations: Graduate School of Information Science and Technology, The University of Tokyo · TIER IV, Inc.

Abstract

Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems.

Figures & tables

Figure 1: Three-stage framework overview. CSN (Stage 2) restructures driving-environment information into structured text, while the runtime safety supervisor (Stage 3) monitors VLA waypoints against $\mathcal{S}_{\text{sem}}$ . Both modules operate on CPU.

Figure 2: The text structure bottleneck. (a) Template text presents intent

\mathcal{I}

and constraints

\mathcal{C}

as independent fragments with no

\mathcal{I}\to\mathcal{C}

link. (b) CSN restores the causal link via explicit connectives ( BUT , BEFORE ).

Figure 3: CSN pipeline illustrated on a left-turn scenario. Each principle progressively transforms flat template text into quantitative, structured, and causally linked input.

Approach	Level	VRAM	Retrain	DS Gain	Key Mechanism
LMDrive template ( Shao et al., 2024 )	L0	0	–	baseline	Isolated instruction + notice
TLS-Assist ( Schmidt et al., 2025 )	L1	0	No	+14.1%	Structured signal messages
GraphPilot ( Schmidt et al., 2026 )	L2	0	No	+15.6%	Entity-relationship graph text
SteerVLA ( Gao et al., 2026 )	L2	0	Yes	+4.77 pts	Fine-grained meta-actions
DriveVLM CoT ( Tian et al., 2024 )	L3	High	Yes	–	3-stage causal chain
DriveLM graph QA ( Sima et al., 2024 )	L3	High	Yes	–	Graph dependency chains

Table 1: Taxonomy of VLA text approaches by causal structure level. DS Gain values are from each work’s own evaluation setup and are not directly comparable across rows due to differing routes, weather, and CARLA versions.

(a) Template (LMDrive original)
Instruction	Turn left at intersection.
Notice	Watch out for pedestrians.
(b) +Flat Text (same facts, no causal links)
Instruction	Turn left. Speed 25 km/h. Pedestrian 5m right crossing left. Sedan 12m ahead 30 km/h. RED light. Junction 20m.
(c) CSN (causal structure)
Instruction	Turn left at intersection, BUT yield to pedestrian crossing from right at 5m BEFORE executing turn. Maintain distance from sedan ahead.

Table 2: Text input comparison for a left-turn scenario. Causal connectives shown in bold.

Figure 4: Evaluation environments. (a) Town01, clear day. (b) Town03, heavy rain. (c) Town05, night. (d) Town07, dense fog.

Configuration	DS ( $\uparrow$ )	RC ( $\uparrow$ )	IS ( $\uparrow$ )	$\Delta$ DS ${}_{\text{orig}}$	$\Delta$ DS ${}_{\text{base}}$
LMDrive (original)	32.54 $\pm$ 3.00	48.3 $\pm$ 2.6%	0.729 $\pm$ 0.034	—	—
+ CSN	42.67 $\pm$ 2.74	56.5 $\pm$ 1.7%	0.787 $\pm$ 0.028	+31.1%	—
+ Flat Text	38.71 $\pm$ 1.44	48.7 $\pm$ 1.9%	0.828 $\pm$ 0.025	+18.9%	—
+ Semantic Safety	34.10 $\pm$ 2.25	45.5 $\pm$ 2.1%	0.785 $\pm$ 0.038	+4.8%	—
+ PL-DPO-NLL	32.49 $\pm$ 3.34	44.9 $\pm$ 2.7%	0.754 $\pm$ 0.021	$-$ 0.1%	—
+ TTC Safety	22.02 $\pm$ 4.07	33.7 $\pm$ 3.4%	0.658 $\pm$ 0.037	$-$ 32.3%	$-$ 32.2%

Table 3: Multi-town ablation (16 routes, 8 towns,

N{=}5

). Values are mean

\pm

95% bootstrap CI. Best bold , second underlined ; ties share marking.

Figure 5: Driving Score comparison across all ten configurations (

N{=}5

). Left and center: mean DS with 95% bootstrap CI. Right: raincloud plot showing KDE density (shaded), individual runs (dots), mean (diamond), and 95% CI (whisker). Dotted line: original LMDrive baseline.

Figure 6: Decomposition of CSN’s DS improvement into information content and causal structure contributions on both weight configurations.

Noise Level	Dist. $\sigma$	Speed	Miss Rate	DS	95% CI
Clean (privileged)	0 m	0%	0%	$42.7\pm 3.3$	$[39.9,\;45.4]$
Mild	$\pm$ 1 m	$\pm$ 10%	0%	$45.2\pm 2.9$	$[42.6,\;47.8]$
Moderate	$\pm$ 2 m	$\pm$ 20%	0%	$45.4\pm 2.7$	$[43.1,\;47.6]$
Severe	$\pm$ 5 m	$\pm$ 20%	10%	$46.0\pm 4.1$	$[42.5,\;49.6]$
Extreme	$\pm$ 5 m	$\pm$ 30%	20%	$45.1\pm 2.7$	$[42.7,\;47.4]$

Table 4: Perception noise ablation on Original LMDrive + CSN (

N{=}5

). We inject Gaussian noise on distance, multiplicative noise on speed, and random actor miss rates. All noise-level CIs overlap with the clean baseline, indicating no statistically significant degradation.

Explore similar work

Sep 26, 2026

CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving

Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoning reflects the causal structure of the scene remains untested. We introduce CausalDriveBench, an evaluation framework grounded in Pearl's Causal Hierarchy (PCH) that tests causal reasoning in driving-specific VLAs through structured visual question answering (QA) and alternative-trajectory prediction. To this end, we construct causal scene graphs over nuScenes that distinguish causally active, dormant, and distractor entities, separating perceptual salience from causal relevance. The benchmark spans all four rungs of PCH (association, intervention, and counterfactual along with causal discovery) for QA generation. For the higher rungs, we additionally provide reference trajectories under specified scene modifications, enabling action-level verification that complements reasoning-level evaluation. In total, the benchmark contains 7,285 verified causal QA pairs and 1,000 counterfactual trajectories derived from nuScenes. We evaluate 10 driving-specific VLAs and 3 general-purpose VLMs, and report three findings. First, the best model reaches only 70.6% QA accuracy, and 4 of 13 models score below random chance. Second, comparing each driving VLA to the general-purpose VLM that shares its language backbone, the cost of driving fine-tuning ranges from 2 to 34 percentage points on causal QA, with post-training design explaining the spread. Third, causal QA and trajectory accuracy are statistically uncorrelated across models: under counterfactual prompts, predicted trajectories either over-react or collapse onto the observed-scene baseline. Taken together, these results show that neither fluent rationales nor accurate observed-scene trajectories constitute evidence of causal understanding.

Narendiran Chembu, Navvrat Rao, Shreedhar Shreeshail Kodate +10

Fastcode AI · Indian Institue of Technology Delhi

Reasoning for VLA Models Vision-Language-Action Models Autonomous Driving Benchmarks Counterfactual Evaluation of VLMs

Ask PDF

Jul 2, 2026

Teaching Vision-Language-Action Models What to See and Where to Look

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.

Yuguang Yang, Canyu Chen, Zhewen Tan +10

School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5

End-to-End Autonomous Driving VLM Distillation Vision-Language-Action Models Autonomous Driving Planning Vehicle Trajectory Prediction

Ask PDF

Sep 23, 2026

Less Language, More Latents: Annotation-Efficient VLAs for Driving

Vision-language-action models (VLA) promise human-steerable autonomous driving, but their training is bottlenecked by the scarcity of frames paired with natural-language instructions: while camera streams and expert trajectories are logged at scale, language annotations (e.g., turn left at the intersection) remain scarce and expensive to acquire. To address this challenge, we introduce Latent Action Driving Annotations (LADA), a three-stage pipeline that transforms abundant unlabelled observation-trajectory pairs into a substrate for language-conditioned control. First, we train a latent action model with a vector-quantised bottleneck, producing a compact codebook of high-level vehicle intents. Second, a small language-annotated subset is used to train a vision-language translator to map observations and language instructions into this codebook. Third, we train a driving VLA on observation-latent-action pairs over the full unlabelled corpus. Using fewer than 5% of language annotations and without leveraging any auxiliary chain-of-thought reasoning or visual question answering streams, LADA achieves a Driving Score of 87.98 and a Success Rate of 70.46% on the closed-loop Bench2Drive benchmark, matching or surpassing fully supervised baselines.

Alexey Zakharov, Kemal Oksuz, Puneet K. Dokania

Robert Bosch GmbH, Germany · Five AI Ltd., United Kingdom

Latent Action Learning Autonomous Driving Vision-Language-Action Models

Ask PDF