Driving vision-language-action (VLA) models can accept navigation instructions, yet predicting a driver's trajectory does not establish that a model follows the instruction. We study how driving-language supervision is collected and evaluated, distinguishing prospective instructions issued before an action from retrospective labels written after observing it. We develop ADVLAT-Engine to synchronize spoken instructions, road-scene video, and vehicle trajectories, and compare three navigation apps on 48 replayed routes. On 87 maneuvers announced by both Google Maps and a passenger, the passenger uses more scene-dependent language. We evaluate four open driving VLAs before and after fine-tuning on prospective navigation-app drives. Fine-tuning reduces passenger-drive trajectory error by 23-59%, but instruction effects remain inconsistent. At real intersections, counterfactual instructions reveal stronger responsiveness: fine-tuned Alpamayo2 Super gains 31 and 18 accuracy points at 15 and 5m, while Qwen-Drive follows alternative maneuvers more reliably with a discrete command than with a sentence. We additionally collected retrospective annotations from four people for the driving recordings and describe controlled comparisons of annotation content, timing, and downstream learning; those analyses are pending. Our findings separate maneuver prediction from instruction-following and establish a framework for comparing prospective and retrospective supervision.
Figures & tables
Fig. 1: Instruction sources differ in authorship (machine or human) and timing (retrospective or prospective). Retrospective labels are created after observing a completed drive; navigation apps and passengers issue prospective instructions before the relevant action. ADVLAT-Engine collects the latter alongside synchronized vision and trajectories. Temporal order motivates, but does not by itself identify, causal instruction effects.
Fig. 2: Illustrative causal graphs, with observed pre-instruction context S , latent common causes U , prospective instruction L , recorded action A , and retrospective label L′ . (a) A genuine request can influence the driver’s subsequent action, but U can confound the association. (b) A post-hoc label depends on the completed action and cannot affect that already-recorded action. (c) Our paired recordings provide both instruction types for the same action. Arrows depict possible dependencies, not an identified causal effect; some context and annotator-specific influences are omitted for clarity.
Fig. 3: ADVLAT-Engine synchronizes spoken navigation or passenger instructions, forward camera observations, and vehicle localization to form instruction–vision–trajectory training examples.
Apple
Google
Waze
Turn
95.9
92.9
97.6
Road name
91.8
94.7
96.5
Distance
48.2
48.2
78.8
Static object
64.7
67.1
38.2
Lane
18.8
20.6
25.3
Light
17.1
4.7
0.6
TABLE I: Navigation-app prompts on 170 shared maneuvers (%); median first-prompt leads are measured at replayed speed.
Fig. 4: Example navigation-app recording with a GNSS route, speech-completion positions, transcribed prompts, and associated camera frames. The original prompts were spoken during the drive, rather than written after viewing its outcome.
Passenger
Google Maps
Road name
1.1
93.1
Distance
0.0
34.5
Static object
52.9
67.8
Deictic (“here”, “now”)
43.7
1.1
Continue/follow road
36.8
1.1
None of eight app classes
20.7
0.0
TABLE II: Passenger and Google Maps prompts on the same 87 maneuvers (% unless indicated).
ADE ↓
Effect ↓
FDE ↓
Turn ↑
Model
Instr.
No instr.
Difference [95% CI]
Instr.
L / R
Ego extrap.
1.331
–
3.16
60 / 54
Alpamayo 1.5
1.887
1.863
+0.024 [ +0.001,+0.052 ]
4.38
30 / 6
fine-tuned
0.775
0.772
+0.003 [ −0.010,+0.018 ]
1.78
83 / 83
Alpamayo 2 Super
1.381
1.391
−0.010 [ −0.026,+0.008 ]
3.19
90 / 88
fine-tuned
0.693
0.686
+0.007 [ −0.010,+0.029 ]
1.56
92 / 84
TABLE III: Passenger-drive test, 3-s horizon. ADE and FDE in meters; Effect = instructed minus uninstructed ADE (negative indicates improvement), with 95% CI. Turn is left/right accuracy (%).
15 m
5 m
0 m driven
0 m parked
Model
Acc.
Gain
Acc.
Gain
Acc.
Gain
Steer
Acc.
Gain
Steer
Alpamayo 1.5
31
+2
27
+2
23
+1
1
27
0
0
fine-tuned
24
+4
13
+3
1
0
1
19
0
5
Alpamayo 2 Super
32
+8
10
+2
2
0
0
15
−1
0
fine-tuned
55
+31
25
+18
4
+3
7
34
+15
40
Qwen-Drive, sentence
43
+19
34
+21
93
+86
151
21
+15
42
TABLE IV: Counterfactual results at the hand-marked stop line. Accuracy in %, Gain in percentage points, Steer in degrees (signed left-minus-right). At 0 m, driven uses the recorded approach and parked uses repeated frames and stationary ego history. Bold highlights maximum accuracy/gain within a condition, including ties.
Fig. 5: Passenger-drive example: observed trajectory and pretrained/fine-tuned model predictions compared with fine-tuned predictions after removing the passenger instruction. A single trajectory illustration complements, but does not replace, paired error and counterfactual instruction tests.
Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.
Zihan You, Zehua Fan, Wenqi Lyu +19
Institute for AI Industry Research (AIR), Tsinghua University · Southeast University · Shanghai Jiao Tong University +16
Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving. However, existing VLAs' training relies heavily on text-centric visual question answering and chain-of-thought reasoning data, which emphasizes linguistic reasoning rather than action-grounded planning. As a result, the learned representations capture semantic knowledge but lack spatial dependencies crucial for reliable trajectory prediction. We propose DriveTeach-VLA, a framework that explicitly teaches VLAs what to see and where to look. Driving-aware Vision Distillation (DVD) injects driving-specific perceptual priors into the vision encoder, while 2D Trajectory-Guided Prompts (2D-TGP) provide spatial conditioning aligned with feasible driving trajectories. Together, they form a vision-guided learning pipeline: what to see (DVD pretraining) - where to look (TGP-guided SFT) - how to act (TGP-guided GRPO). DriveTeach-VLA achieves the state-of-the-art performance on NAVSIM and nuScenes. Our code is available at: https://github.com/ShivaTeam/DriveTeach-VLA.
Yuguang Yang, Canyu Chen, Zhewen Tan +10
School of Electronic Information Engineering, Beihang University · Institute for AI Industry Research (AIR), Tsinghua University · National College for Excellent Engineers, Beihang University +5
Driving Vision-Language-Action Models (Driving VLAs) aim to use language to improve end-to-end planning, but the language-action gap limits this promise. We propose DriveMA, a Driving VLA framework built on verifiable meta-actions, which summarize future ego motion into compact language-domain intentions and can be constructed from expert trajectories with a trajectory-grounded annotation pipeline and can be verified against generated trajectories through rule-based projection. DriveMA exploits this verifiability with action-centric supervised training and a data-efficient turn-level credit assignment reinforcement learning framework, explicitly aligning high-level decisions with low-level trajectory planning through dense rewards and precise credit assignment. DriveMA sets a new state of the art on the Waymo Open Dataset Vision-based E2E Driving, achieving a Rater Feedback Score of 8.060 with a 2B model and further improving it to 8.079 with a 4B model; it also obtains competitive closed-loop planning performance on NAVSIM. These results show that even a simple meta-action interface can achieve state-of-the-art planning when made verifiable and optimized for language-action alignment. Code, data, and models are available at https://tsinghua-mars-lab.github.io/DriveMA.
Weicheng Zheng, Yixin Huang, Qiao Sun +2
1Shanghai Qi Zhi Institute · 3Tongji University · 2IIIS Tsinghua University