Vision-language-action (VLA) driving models couple a reasoning stage with a diffusion-based trajectory decoder, but do not give a direct way to redirect attention toward safety-critical actors at inference time without retraining. We studied a bounded additive pre-softmax attention bias on the visual tokens of detector localized traffic actors on Alpamayo-R1's Qwen3-VL backbone. It is applied as a fail open forward pre-hook with no weight changes. On 50 lane-change scenarios from the Physical AI World Model Synthetic dataset. The trajectory decoder shows a monotonic dose response in the bias magnitude, separate from a paired zero bias control at every tested magnitude. It reaches ≈17,cm mean displacement with lateral shifts up to ∼140\ cm at the clamp. A layer ablation places the action-relevant signal in late layers, where the effect increases with the number of hooked layers (2.0cm for the first 8 layers; 67.6cm for all 36). A per call injection audit explains why the Chain-of-Causation text never changes. The mask based bias never reaches the reasoning pathway in this serving stack, so the invariance is verified exposure, not robustness. Steered trajectories tend to shift toward the attended actor, suggesting the bias governs where the model looks rather than encoding a target behavior.
Figures & tables
Figure 1: Overview of the intervention. A detected lead actor is mapped by Eq. 4 onto the merged visual-token grid. Every index is verified against the tokenizer’s video-token spans. And a forward pre-hook adds the bounded bias of Eq. 2 to those tokens’ attention logits, with no weight changes. The two pathways are not equally reachable: the diffusion decoder carries an explicit 4D additive mask and receives the bias; the CoC path uses a fused causal kernel with no mask and is never perturbed.
Figure 2: Front-camera actor grounding. The YOLO11 detector localizes the lead vehicle ( red box , with confidence and pinhole range estimate); the box is mapped through Eq. 4 onto the 10×18 merged token grid ( thin overlay ); the resulting token patch ( orange cells ) receives the pre-softmax bias. The mapping is verified per run against the tokenizer’s video-token spans, so any resolution mismatch is detected before injection.
Figure 3: Trajectory response to pre-softmax attention bias on detector-grounded actor tokens, across 50 lane-change scenarios. Boxes show the IQR; the white line is the median, the diamond the mean, and individual points are per-scenario values. Left: ADE grows monotonically with bias magnitude, reaching ≈17 cm mean at head_scale =4 . Right: maximum lateral shift reaches ≈38 cm mean, with a heavy upper tail (single cases up to ∼140 cm). The zero-bias condition is exactly zero in every case by construction of the paired-seed gate.
Figure 4: Layer subset ablation at head_scale =4.0 on detector-grounded actor tokens ( N=5 clips). Early layers ( first_8 ) are nearly inert; late layers dominate; the effect accumulates with the number of hooked layers, and all_layers exceeds every contiguous subset on both metrics.
Layer subset
Hooked layers
Mean ADE (cm)
Mean max lat. shift (cm)
first_8
8
2.0
2.9
mid_8
8
9.7
11.5
last_4
4
11.5
13.0
last_8
8
16.7
57.8
last_16
16
42.0
109.2
all_layers
36
67.6
205.5
Table 1: Layer-subset ablation. Mean ADE and mean max lateral shift, at head_scale =4.0 across N=5 clips. Hooked count refers to the number of transformer layers in which the pre-hook is installed.
Figure 5: Bird’s-eye-view paired comparison at head_scale =4.0 . Left: baseline (blue) and steered (orange) ego trajectories over a 6.4 s horizon; the detected actor is shown in red on the ego’s left. Right: lateral delta (steered − baseline) over time. The steered trajectory deviates progressively toward the actor’s side as the rollout proceeds. CoC text is identical between the two runs.
Figure 6: Trajectory-deviation panel for the same clip. Left: lateral delta over time (filled) and longitudinal delta (dashed). Middle: deviation in (lateral, longitudinal) coordinates with time coloring. Right: lateral-zoom BEV emphasizing the small but reproducible lateral offset relative to the baseline. Maximum lateral shift is 73.5 cm at t≈6.3 s.
Vision-Language-Action (VLA) models for autonomous driving must integrate diverse textual inputs, including navigation commands, hazard warnings, and traffic state descriptions, yet current systems often present these as disconnected fragments, forcing the model to discover on its own which environmental constraints are relevant to the current maneuver. We introduce Causal Scene Narration (CSN), which restructures VLA text inputs through intent-constraint alignment, quantitative grounding, and structured separation, at inference time with zero GPU cost. We complement CSN with Simplex-based runtime safety supervision and training-time alignment via Plackett-Luce DPO with negative log-likelihood (NLL) regularization. A multi-town closed-loop CARLA evaluation shows that CSN improves Driving Score by +31.1% on original LMDrive and +24.5% on the preference-aligned variant. A controlled ablation reveals that causal structure accounts for 39.1% of this gain, with the remainder attributable to information content alone. A perception noise ablation confirms that CSN's benefit is robust to realistic sensing errors. Semantic safety supervision improves Infraction Score, while reactive Time-To-Collision monitoring degrades performance, demonstrating that intent-aware monitoring is needed for VLA systems.
Yun Li, Yidu Zhang, Simon Thompson +2
Graduate School of Information Science and Technology, The University of Tokyo · TIER IV, Inc.
Vision-Language-Action (VLA) models have shown strong potential for end-to-end autonomous driving by jointly modeling visual perception, language reasoning, explainability and action prediction. However, their large vision-language backbones and reasoning modules introduce substantial inference latency and thereby prevent their deployment in the unforgiving reality of the road networks. We propose RT-VLA, a lightweight, distilled VLA model that transfers the driving and reasoning capabilities of the state-of-the-art SimLingo model into a compact student through multi-level supervised distillation. RT-VLA preserves language-based reasoning and supports post-hoc explanation through offline language analysis of safety-critical driving moments without adding latency to real-time control. Compared to the SimLingo teacher, RT-VLA maintains competitive closed-loop driving and language reasoning performance while reducing inference time by 44.8X in vision-only mode and 7.9X in vision+language mode. These results suggest that supervised distillation is a practical approach for building real-time, explainable VLA-style autonomous driving models.
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.
Yuguang Yang, Canyu Chen, Zhewen Tan +10
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5