Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
Authors: Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, +14 more
Organizations: Institute for AI Industry Research (AIR), Tsinghua University · Southeast University · Shanghai Jiao Tong University · University of Adelaide · Beijing Information Science and Technology University · Amazon · Beijing Jiaotong University · Wuhan University · Huazhong University of Science and Technology · University of Science and Technology of China · NIO · University of Texas at Austin · Chinese Academy of Sciences · The Hong Kong University of Science and Technology (Guangzhou) · HiThink Research · AMD · The University of Hong Kong · EPFL · Li Auto Inc.
Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.
Figures & tables
Figure 1: Overview of Language2Drive. Language2Drive evaluates autonomous-driving VLA instruction following across four scopes: (A) Atomic actions, (B) Persistent behaviors, (C) Scene-grounded instructions, and (D) Long-horizon programs. To support these task, we build purpose-designed CARLA maps includes cities, underground parking lot, campus and port. We also curate relevant scenarios from AlpSim, NAVSIM, nuScenes, and WOD-E2E. The Task-Coverage chart summarizes the diversity of driving behaviors, while the Word Cloud illustrates the breadth of natural-language expressions. We further introduce the Paired Counterfactual Unit (PCU), which varies the instruction while keeping the driving scene fixed, directly testing whether language changes the vehicle’s behavior. Our benchmark highlights a key distinction: driving competence does not necessarily imply language controllability or reliable instruction following.
Figure 2: Benchmark construction process. The process to build the benchmark is as follows: (1) Source curation identifies decision opportunities; (2) Geometric grounding binds language to physical entities; (3) Paired specification fixes shared conditions and opposed outcomes; (4) Compilation separates public inputs from private criteria; (5) Validation checks the resulting tasks. The benchmark contains 64 CARLA and 30 RealSim PCUs.
System
Training data
Strict success
Relaxed proxy
Graded
Driving
Summary
SISR
CFPS
SemISR
Sem- CFPS
SCS
PCS
PSM
Drive Score
L2D
Alpamayo 1.5
NVIDIA AV
8.07
1.04
42.19
30.73
26.18
12.50
70.10
62.69
12.20
SimLingo
Action Dream
7.81
1.04
48.70
35.94
23.51
11.94
62.30
65.04
12.15
LMDrive
CARLA-64K
12.24
0.00
29.17
18.23
21.45
9.26
88.84
50.12
10.39
OneVL
NAVSIM
0.52
0.00
25.00
3.65
16.69
7.05
77.72
50.69
4.87
Qwen-Drive-1.0-4B
WOD-E2E
1.82
0.00
15.62
3.12
17.05
7.09
76.68
43.67
4.31
Table 1: CARLA-Language2Drive main results. 384 core rollouts and 192 paired instances per system (64 PCUs, three expressions). PSM uses 24 additional routes. All values are on a 0–100 scale; higher is better. Strict individual and paired success are shown first; L2D follows the component scores.
System
Strict success
Relaxed proxy
Graded
Driving
Summary
SISR
CFPS
SemISR
Sem- CFPS
SCS
PCS
Safety
Driving Score
L2D
Alpamayo2-Super
13.33
0.00
18.33
0.00
21.43
0.00
93.33
97.10
9.22
Alpamayo 1.5
11.67
0.00
15.00
0.00
18.12
1.11
100.00
97.17
8.38
SimLingo
6.67
3.33
13.33
6.67
16.67
10.00
68.33
83.09
7.58
AutoVLA
1.67
0.00
8.33
6.67
9.73
6.67
41.67
76.94
3.36
Qwen-Drive-1.0-4B
0.00
0.00
1.67
0.00
4.44
0.00
43.33
69.16
0.92
Table 2: RealSim main results. Scoring v0.2 on 30 PCUs from 23 reconstructed scenes: 60 rollouts and 30 pairs per system. All values are on a 0–100 scale; higher is better. Strict individual and paired success are shown first; L2D follows the component scores.
Figure 3: Advancement and behavioral response do not establish complete instruction execution. (a,b) P11 requests lane keeping (R0, blue) or a left lane change (R1, orange dashed). Alpamayo 1.5 reaches 100 DrivingScore in both arms but succeeds only on R0. P22 requests 3–7 or 8–12 km/h. Shading marks the requested speed bands, not their active time windows. Curves cover the full post-instruction interval; verification applies from entry into the annotated segment to exit and requires complete traversal and compliance at all active samples. SimLingo passes both criteria; Alpamayo 1.5 passes neither. Figure 9 gives the segment geometry.
Figure 4: Open-loop instruction following track on three datasets. Predicted trajectories are scored for exclusive binary semantic success (SemSISR/SemCFPS) and graded completion (SCS/PCS). All values use a 0–100 scale.
Execution setting
Instruction evaluation
Benchmark / protocol
World
CL
Language input
Matched context
Alt. targets
Behavior check
Para. test
Paired success
nuPlan 2021
R
✓
–
–
–
–
–
–
Bench2Drive 2024
S
✓
–
–
–
–
–
–
NAVSIM v2 2025
R
–
–
–
–
–
–
–
LangAuto 2024
S
✓
✓
–
–
–
–
–
Action Dreaming 2025
S
–
✓
✓
✓
✓
–
–
Table 3: Comparison of driving evaluation protocols. S: synthetic worlds; R: real-log-derived contexts, not on-road tests. CL: sequential closed-loop execution. Matched: instruction variants share a scene or route context. Alt.: variants have distinct scored behavioral targets. Behavior check: instruction-specific acceptance beyond route-level metrics. Para.: explicit paraphrase evaluation. Paired success: both opposed, feasible requests must succeed.
Appendix figures & tables45 assets
Supplementary material from the paper’s appendix.
Appendix
Map identifier
PCUs
Core rollouts
American-city0428
11
66
Town01
2
12
Town02
1
6
Town03
10
60
Town04
4
24
Town05
2
12
Appendix
Table 4: Actual CARLA base map manifest. These ten evaluated layouts include five custom non-Town assets. The distribution-level six-environment description and this evaluation manifest have different scopes.
Cohort
PCUs
Arms
One seed
Three seeds
Source
128
256
768
2304
Small
32
64
192
576
Base
64
128
384
1152
Large (loadable)
119
238
714
2142
Appendix
Table 5: Compiled source suite and nested CARLA cohorts. Columns count PCUs and potential Core executions; only the base single-seed cohort is the complete comparison.
Stratum
Category
PCUs
Rollouts
A
junction
7
42
A
lane
4
24
A
speed_start
2
12
A
waiting_parking
3
18
B
lane_constraint
4
24
B
object_relation
4
24
Appendix
Table 6: CARLA base task inventory, with three paraphrases per arm and seed 1.
PCU
Arm 0
Arm 1
A-001
Turn left at this intersection.
Go straight through this intersection.
B-025
Safely pass the slow truck.
Do not overtake; remain behind the slow truck.
C-001
Take Middleton Street.
Take E Gage Ave.
C-025
Park in space B204.
Park in space B205.
D-001
First visit B208, then go to F206.
First visit F206, then go to B208.
D-009
If E101 is occupied, park in E102; otherwise use E101.
If E101 is occupied, park in C104; otherwise use E102.
Appendix
Table 7: Examples of the evaluated CARLA instruction pairs.
Trajectory generation and tracking; multi-camera temporal input
Qwen-Drive-1.0-4B
Front/left/right RGB, ego
Backbone with planner-rl; sampled plans and trajectory tracking
OneVL
Single front view, ego/history
NAVSIM-format latent-prefix trajectory generation
Impromptu-VLA
Front/right/left RGB, ego history
Released 3B AD-finetuned QA-format waypoint generation
Appendix
Table 8: Complete CARLA system paths. A bridge denotes an interface that differs from the original end-to-end architecture.
System
SemSISR
SemCFPS
SCS
PCS
WOD-E2E
DriveMA-2B
38.38
1.24
49.33
7.88
Fast-dDrive
34.85
0.00
46.42
5.27
Qwen-Drive-SFT
34.44
0.00
50.58
9.37
Qwen-Drive-RL
35.06
0.00
49.58
7.58
NAVSIM
Appendix
Table 9: Open-loop instruction following on three datasets. Predicted trajectories are scored for exclusive binary semantic success (SemSISR/SemCFPS) and graded completion (SCS/PCS). All values use a 0–100 scale.
Source
Available source / construction
Evaluation subset
Queries per model
nuScenes
150 test scenes; 6,008 anchors; 83 candidates
75 approved PCUs in 75 scenes / 15 logs; dev 15, heldout 60
900 across 75 PCUs; 720 heldout, including controls
Table 12: Released nuScenes task and geographic distributions. The paired lane alternatives are change/keep, not left/right within the same PCU.
Parameter
Value or exact rule
Lane-task initial speed
4.5≤v0≤9.0 m/s; released range 4.745680–8.878580
Adjacent-lane query
7 m radius; polygon gap ≤0.8 m
Divider / direction
Divider exists and every segment type contains DASHED ; target yaw error ≤20∘
Relative target position
Lateral distance 2.4–5.0 m on the requested side; absolute forward offset ≤5.0 m
Candidate ranking
∣dlat−3.5∣+0.1∣dfwd∣ ; search left before right
Lane witness blend
Start 0.4 s; duration 2.0 s; initial centerline offset decays over 1.0 s
Appendix
Table 13: nuScenes target-construction parameters. These mining and witness parameters are distinct from the final scoring tolerances.
Arm
Predicate
Required samples
Lane-change arm 0
Reference point lies in the target lane corridor
t=2.25,2.50,2.75,3.00 s; 4 points
Lane-keep arm 1
Reference point lies in the current lane corridor
t=0.25,0.50,…,3.00 s; 12 points
Low-speed arm 0
3.2≤vk≤4.8 m/s
Interval ends tk=1.25,…,3.00 s; 8 intervals
High-speed arm 1
7.2≤vk≤8.8 m/s
Same 8 intervals
Appendix
Table 14: Complete nuScenes binary semantic conditions, in addition to D≥8 m. Every required sample must satisfy its condition.
Diagnostic
Exact definition / threshold
Invalid handling
Speed-target MAE
Mean ∣3.6v−v∗∣ on 8 intervals, v∗=15 or 30 km/h
Valid-arm mean
Speed-band violation
3.6meanmax(ℓ−v,0,v−h) , km/h
Valid-arm mean
Speed coverage
In-band fraction over 8 intervals, without displacement gate
Zero for invalid arms
Paired speed change
Δv=3.6[mean(vhigh)−mean(vlow)]
Valid-pair mean
Lane distance / coverage
Mean distance to own corridor; inside fraction over 4 change or 12 keep points
Distance conditional; coverage zero if invalid
Lane preference
q=mean(dcurrent−dtarget) on the common window; Δq=qchange−qkeep
Valid arm / pair
Appendix
Table 15: Additional nuScenes diagnostics and their full numeric conventions. Conditional means always accompany validity coverage.
Family
Scope
PCUs
Share
Counterfactual alternatives
Lane choice
A
12
31.58%
Keep versus change one lane; eight left-target and four right-target PCUs
Junction branch
A
1
2.63%
Left turn versus straight at a common junction
Pass / follow
A
1
2.63%
Follow the same actor versus pass using an adjacent lane
Speed band
B
7
18.42%
Two disjoint speed bands over the declared interval
Object-relative speed
B
2
5.26%
Two speed requirements while following the same actor
Landmark stop
C
10
26.32%
Distinct stopping locations grounded in visible landmarks
Appendix
Table 16: NAVSIM 1.0.0 task composition and documented counterfactual targets. Counts and percentages use the 38-PCU release, not the earlier 12-candidate pool.
Component
Frozen scoring rule / parameter source
Common grid / quality
8 poses at 0.5–4.0 s in the rear-axle frame; maximum acceleration 3.5 m/s 2 and maximum yaw rate 0.8 rad/s
Lane / local goal
Vehicle-footprint overlap, terminal progress, and target polygons from the per-PCU private contract
Junction goal
Goal region, active interval, and deadline stored in the per-PCU contract
Landmark stop
Landmark-grounded stop region, stopping threshold, and hold requirement stored in the private contract
Speed band
Per-PCU lower/upper speed bands and active time range; primitive completion decays linearly over the native tolerance
Object-relative speed
Speed-band requirement combined with the bound lead-actor geometry and relative-position constraints
Appendix
Table 17: NAVSIM task-specific scoring components. Global rules are shown here; per-PCU geometry and thresholds are read from the frozen private sidecars.
Model
Released inputs and instruction placement
Common-grid conversion
Qwen-Drive
Four frames of front/front-left/front-right; 16-point interpolation of observed 1.5 s history; full instruction in scene prompt
Four frames of three forward cameras; current velocity/acceleration; driving-instruction field
First eight released 0.5 s action-token poses
DriveMA-2B
Current three forward cameras; observed 1.5 s history and state; high-level-intent field
Released two-turn decision/trajectory protocol; 1 Hz positions linearly resampled to 0.5 s
ReCogDrive
Current front image; four observed poses and state; text replaces discrete command phrase
Released eight-pose DDIM planner output
OneVL
Current front image; three historical poses and state; full Command field
Released latent-chain-of-thought generation of eight 0.5 s poses
Appendix
Table 18: Frozen NAVSIM adapters. Input differences are retained and disclosed rather than hidden by the common output grid.
Model
Repository commit
Primary checkpoint revision
Qwen-Drive
28091c1532e869bc7aee91fc0aef6b3e6fd0b2e0
28484089a7cc8c335cf5089fb0745cf7c49b6eaa
AutoVLA
ba34eed74ce6729e7986592d0e66cbaca397b4fa
a7d7ba3ed7529b248d2694c2defa31b35208340f
DriveMA-2B
de20e6c64878bc3413d26d69362c3be9b3a91533
d5f4905d2032da943a7ae8f22d5b2b8efbe4b0f7
ReCogDrive
6b8d8f5e01346c71094651c81dcaf66405dbc04e
7346ede73f03ed577811024f1a5d16fb063a47b9
OneVL
f4e6f05572b2f66611259bcd5563529833f30469
c1e893e9a03b8312f9328d2b51267836a9f7bf8a
Appendix
Table 19: NAVSIM model revision identifiers from the frozen experiment protocol. Line breaks inside identifiers are typographic only.
Task family
Pairs
% of 563
% of 441
Requests
Evaluated PCUs
SPEED_CHOICE
175
31.08
39.68
350
165
STOP_GO
112
19.89
25.40
224
76
LATERAL_CHOICE
112
19.89
25.40
224
0
TURN_WAIT
28
4.97
6.35
56
0
TURN_STRAIGHT
14
2.49
3.17
28
0
UNRESOLVED
122
21.67
–
0
0
Appendix
Table 20: WOD-E2E P0 task-family distribution. Candidate construction, language-pair construction, and the final semantic-evaluation set are reported with separate denominators.
Criterion
Binary success
Completion 1
Completion 0
Stop tail speed
vtail≤0.6
≤0.6
≥2
Stop endpoint distance
dstop≤2
≤2
≥5
Go tail speed
vtail≥2
≥2
≤0.6
Go forward endpoint
x20≥8
≥8
≤0
Slow late speed
vlate≤ℓi
≤ℓi
≥hi
Normal late speed
vlate≥hi
≥hi
≤ℓi
Appendix
Table 21: WOD-E2E semantic scoring parameters. Speed and distance are reported in m/s and m, respectively. The speed-choice boundaries ℓi and hi are frozen per-PCU evaluator parameters.
System
Valid outputs
Own-reference ADE ↓
Aligned pairs
DriveMA-2B
882/882
2.520
6/441
Fast-dDrive
882/882
2.817
5/441
Qwen-Drive-SFT
882/882
2.959
3/441
Qwen-Drive-RL
876/882
2.924
2/436
Appendix
Table 22: WOD-E2E geometric diagnostics on the 441 instruction-paired samples. ADE is measured in meters.
Table 23: Recorded release identities used to bind the reported experiments to their public, private, evaluator, and scoring artifacts.
System
A
B
C
D
LMDrive
13.54
22.92
6.25
6.25
Alpamayo 1.5
3.12
16.67
12.50
0.00
SimLingo
13.54
12.50
4.17
1.04
Qwen-Drive-1.0-4B
0.00
4.17
3.12
0.00
OneVL
0.00
1.04
1.04
0.00
Impromptu-VLA
1.04
0.00
0.00
0.00
Appendix
Table 24: Strict CARLA instruction success by scope. Each scope has 16 PCUs, 96 rollouts, and 48 paired instances per system. A: atomic; B: persistent; C: grounded; D: program.
Figure 5: Strict CARLA success by control scope, with 96 rollouts per system and scope. Gold markers average six systems. D has mean SISR 1.22%; its seven strict successes all maintain an invariant until an event. Connecting lines join categorical scopes rather than a continuous difficulty axis.
Strict outcomes
Semantic
System
Flip
All three
Any
Flip/any (%)
Flip
LMDrive
3
14
17
17.65
5
Alpamayo 1.5
12
4
16
75.00
8
SimLingo
9
6
15
60.00
7
Qwen-Drive-1.0-4B
2
1
3
66.67
9
OneVL
2
0
2
100.00
5
Appendix
Table 25: Instruction sensitivity across three CARLA expressions. Counts use 128 semantic arms per system. “All three requires the same arm to succeed under every expression; “any requires at least one. Flip/any conditions on arms with some strict success.
System
A
B
C
D
Is
Im
All-3
LMDrive
0
0
0
0
3
5
14
SimLingo
2
0
0
0
9
7
6
Alpamayo 1.5
0
2
0
0
12
8
4
Qwen-Drive-1.0-4B
0
0
0
0
2
9
1
OneVL
0
0
0
0
2
5
0
Impromptu-VLA
0
0
0
0
1
15
0
Appendix
Table 26: CARLA paired and paraphrase diagnostics. A–D: successful pairs out of 48. Is/Im : numbers of arm groups with inconsistent strict/semantic outcomes across paraphrases, out of 128. All-3: arms strictly successful for all three paraphrases, out of 128. All systems have zero worst-paraphrase paired success out of 64 PCUs.
System
p0 S/M
p1 S/M
p2 S/M
p0 pair
p1 pair
p2 pair
LMDrive
15/36
17/38
15/38
0
0
0
SimLingo
13/61
8/64
9/62
2
0
0
Alpamayo 1.5
11/52
11/55
9/55
1
1
0
Qwen-Drive-1.0-4B
2/20
3/19
2/21
0
0
0
OneVL
0/31
2/33
0/32
0
0
0
Impromptu-VLA
1/25
0/23
0/28
0
0
0
Appendix
Table 27: CARLA per-paraphrase outcomes. S/M entries give strict/semantic success counts out of 128; pair columns count successful pairs out of 64.
System
Lbal
Dt
Driving credit
Language credit
L2D
CARLA
Alpamayo 1.5
17.56
66.40
0.66
11.54
12.20
SimLingo
18.26
63.67
0.64
11.51
12.15
LMDrive
14.09
69.48
0.69
9.69
10.39
OneVL
6.66
64.21
0.64
4.23
4.87
Qwen-Drive-1.0-4B
6.23
60.18
0.60
3.71
4.31
Appendix
Table 28: L2D components. The language and driving contributions sum to the final score before rounding. Calculations use the displayed component metrics in Tables 1 and 2 .
System
A: atomic action
B: persistent behavior
Overall
Alpamayo 1.5
4/12 (33.33)
5/48 (10.42)
9/60 (15.00)
Alpamayo2-Super
5/12 (41.67)
6/48 (12.50)
11/60 (18.33)
SimLingo
3/12 (25.00)
5/48 (10.42)
8/60 (13.33)
OneVL
0/12 (0.00)
0/48 (0.00)
0/60 (0.00)
ReCogDrive
0/12 (0.00)
0/48 (0.00)
0/60 (0.00)
Appendix
Table 29: RealSim relaxed semantic outcomes by scope for five systems. Counts use 12 atomic arms and 48 persistent arms; overall rates agree with Table 2 .
System
Pairs
Separation (m)
Speed pairs
Speed diff. (km/h)
Alpamayo2-Super
30
0.23
21
0.46
Alpamayo 1.5
30
0.15
21
0.21
SimLingo
30
2.61
21
3.51
OneVL
30
0.36
21
0.25
ReCogDrive
30
0.10
21
0.07
Qwen-Drive-1.0-4B
29
0.31
20
0.48
Appendix
Table 30: RealSim response to opposed instructions in the trajectory-response analysis. Spatial values are medians over complete valid pairs; speed values use the B1/B3 subset. Each pair first averages differences over 101 matched post-instruction time points. These describe response magnitude, with correctness evaluated separately by CFPS.
System
SI
GPE
PSM
MDQ
RLS
nRLS
LMDrive
0.8357
0.2684
0.8884
0.9340
0.4437
23
SimLingo
0.5419
0.7692
0.6230
0.8310
0.6079
23
Alpamayo 1.5
0.4480
0.3009
0.7010
0.9646
0.3152
21
Qwen-Drive-1.0-4B
0.4883
0.0056
0.7668
0.9964
0.0873
24
OneVL
0.2390
0.0463
0.7772
0.9967
0.1332
24
Impromptu-VLA
0.2574
0.0209
0.7743
0.9897
0.0627
23
Appendix
Table 31: Supplementary FDQ measures. SI, GPE, PSM, and MDQ each have n=24 ; the last column reports RLS coverage.
System
Strict
Sem
Safe
Obey
Fall.
Ref.
Alt.
LMDrive
2
3
5
0
0
2
0
SimLingo
0
4
0
0
0
0
0
Alpamayo 1.5
0
4
1
2
0
0
0
Qwen-Drive-1.0-4B
1
2
1
0
0
1
0
OneVL
0
2
0
0
0
0
0
Impromptu-VLA
0
2
0
1
0
0
0
Appendix
Table 32: Safety-priority diagnostic counts. Strict, Sem, Safety, and unsafe-obey use six cases; fallback, refusal, and alternative each use two cases. Responses are behavior-based.
Figure 6: Six environment families. Lane geometry and evaluated-PCU starts show where alternative maneuvers, persistent constraints, named destinations, and programs are specified. Each panel has its own metric scale. The ten evaluated map identifiers are listed in Table 4 .
Figure 7: Sixteen CARLA task families. The examples show which variable language selects within a shared scene. The complete task catalog remains in Tables 33 and 34 .
Figure 8: P11 reference geometry and scene context. The lane alternatives share an initial pose. Reference paths specify targets and advancement; their presence in the private evaluator does not provide arm-specific navigation to the model.
Figure 9: P22 reference speed, active region, and scene context. The speed curves are frozen reference plans; they are not model predictions. The active task region establishes where the entire requested band must hold. The model curves in Figure 3 show post-instruction response, with full-scoring verdicts reported separately.
PCU
Category
Map
Split
End (s)
A-001
junction
American-city0428
dev
45
A-002
junction
American-city0428
dev
45
A-004
junction
American-city0428
dev
45
A-006
junction
American-city0428
dev
45
A-008
junction
American-city0428
test
45
A-010
junction
American-city0428
test
45
Appendix
Table 33: All 64 evaluated CARLA PCUs, grouped by stratum. End is the arm deadline in simulation seconds. Every PCU has two arms and three paraphrases per arm.
PCU
Family
Scene
End (s)
2d72b487
B1
31ab4b94
7.0
09dd1b67
A2
2f380e80
8.0
5cbf0c52
B1
afeb04ed
7.0
c10334f0
B1
cdd9448c
7.0
b92f015d
B1
ce161092
7.0
b9667ab7
B1
0d4893f5
7.0
Appendix
Table 34: All 30 RealSim PCUs. Identifiers are short report identifiers. Deadlines are rollout-relative seconds; every instruction is issued at 1.7 s. Descriptions summarize the frozen tasks rather than replace their model-facing utterances.
Figure 10: A-006/p0: successful exit selection. Both the left and straight runs satisfy their full requested criteria.
Figure 11: A-006/left: strict success varies across equivalent requests. Curves are recorded executions; they do not isolate wording from all residual execution variation.
Figure 12: B-027/p1: successful passing and following. The two speed profiles accompany successful full-criterion verdicts.
Figure 13: C-025/p0: both named parking targets remain uncompleted. Each arm travels more than 160 m but fails its target criterion.
Figure 14: D-001/p0: local movement does not complete either requested sequence. Ordered stage completion determines the verdict.
Figure 15: D-009/p0: neither conditional target is completed. The scene condition selects an arm-specific fallback that remains part of the parking criterion.
Figure 16: D-026/p0: the requested visit-and-return programs remain unfinished. Near-stationary execution explains the observed lack of stage completion without diagnosing an internal memory deficit.
Figure 17: P11: measured trajectories. Each panel contrasts the two lane requests. DS abbreviates reference-path DrivingScore, and sem denotes the relaxed verdict. The full task specification appears in Figure 8 .
Figure 18: P22: measured trajectories. SimLingo completes both requested speed-band criteria. Spatial advancement alone does not determine speed compliance; Figure 3 shows the Alpamayo 1.5 and SimLingo speed histories, and Figure 9 shows the active region.
Recent progress in vision-language-action (VLA) models has enabled language-conditioned driving agents to execute natural-language navigation commands in closed-loop simulation, yet standard evaluations largely assume instructions are precise and well-formed. In deployment, instructions vary in phrasing and specificity, may omit critical qualifiers, and can occasionally include misleading, authority-framed text, leaving instruction-level robustness under-measured. We introduce ICR-Drive, a diagnostic framework for instruction counterfactual robustness in end-to-end language-conditioned autonomous driving. ICR-Drive generates controlled instruction variants spanning four perturbation families: Paraphrase, Ambiguity, Noise, and Misleading, where Misleading variants conflict with the navigation goal and attempt to override intent. We replay identical CARLA routes under matched simulator configurations and seeds to isolate performance changes attributable to instruction language. Robustness is quantified using standard CARLA Leaderboard metrics and per-family performance degradation relative to the baseline instruction. Experiments on LMDrive and BEVDriver show that minor instruction changes can induce substantial performance drops and distinct failure modes, revealing a reliability gap for deploying embodied foundation models in safety-critical driving.
Kaiser Hamid, Can Cui, Nade Liang
Texas Tech University · Bosch Center for Artificial Intelligence (BCAI)
Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.
Vision-language-action models (VLA) promise human-steerable autonomous driving, but their training is bottlenecked by the scarcity of frames paired with natural-language instructions: while camera streams and expert trajectories are logged at scale, language annotations (e.g., turn left at the intersection) remain scarce and expensive to acquire. To address this challenge, we introduce Latent Action Driving Annotations (LADA), a three-stage pipeline that transforms abundant unlabelled observation-trajectory pairs into a substrate for language-conditioned control. First, we train a latent action model with a vector-quantised bottleneck, producing a compact codebook of high-level vehicle intents. Second, a small language-annotated subset is used to train a vision-language translator to map observations and language instructions into this codebook. Third, we train a driving VLA on observation-latent-action pairs over the full unlabelled corpus. Using fewer than 5% of language annotations and without leveraging any auxiliary chain-of-thought reasoning or visual question answering streams, LADA achieves a Driving Score of 87.98 and a Success Rate of 70.46% on the closed-loop Bench2Drive benchmark, matching or surpassing fully supervised baselines.
Alexey Zakharov, Kemal Oksuz, Puneet K. Dokania
Robert Bosch GmbH, Germany · Five AI Ltd., United Kingdom