Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?

Zihan You; Zehua Fan; Wenqi Lyu; Aoqi Wang; Ni Wang; Yifei Yang; Linge Zhao; Huan Li

cs.ROOct 12, 2026

Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?

Authors: Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, +14 more

Organizations: Institute for AI Industry Research (AIR), Tsinghua University · Southeast University · Shanghai Jiao Tong University · University of Adelaide · Beijing Information Science and Technology University · Amazon · Beijing Jiaotong University · Wuhan University · Huazhong University of Science and Technology · University of Science and Technology of China · NIO · University of Texas at Austin · Chinese Academy of Sciences · The Hong Kong University of Science and Technology (Guangzhou) · HiThink Research · AMD · The University of Hong Kong · EPFL · Li Auto Inc.

Abstract

Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.

Figures & tables

Figure 1: Overview of Language2Drive. Language2Drive evaluates autonomous-driving VLA instruction following across four scopes: (A) Atomic actions, (B) Persistent behaviors, (C) Scene-grounded instructions, and (D) Long-horizon programs. To support these task, we build purpose-designed CARLA maps includes cities, underground parking lot, campus and port. We also curate relevant scenarios from AlpSim, NAVSIM, nuScenes, and WOD-E2E. The Task-Coverage chart summarizes the diversity of driving behaviors, while the Word Cloud illustrates the breadth of natural-language expressions. We further introduce the Paired Counterfactual Unit (PCU), which varies the instruction while keeping the driving scene fixed, directly testing whether language changes the vehicle’s behavior. Our benchmark highlights a key distinction: driving competence does not necessarily imply language controllability or reliable instruction following.

Figure 2: Benchmark construction process. The process to build the benchmark is as follows: (1) Source curation identifies decision opportunities; (2) Geometric grounding binds language to physical entities; (3) Paired specification fixes shared conditions and opposed outcomes; (4) Compilation separates public inputs from private criteria; (5) Validation checks the resulting tasks. The benchmark contains 64 CARLA and 30 RealSim PCUs.

System	Training data	Strict success		Relaxed proxy		Graded		Driving		Summary
		SISR	CFPS	SemISR	Sem- CFPS	SCS	PCS	PSM	Drive Score	L2D
Alpamayo 1.5	NVIDIA AV	8.07	1.04	42.19	30.73	26.18	12.50	70.10	62.69	12.20
SimLingo	Action Dream	7.81	1.04	48.70	35.94	23.51	11.94	62.30	65.04	12.15
LMDrive	CARLA-64K	12.24	0.00	29.17	18.23	21.45	9.26	88.84	50.12	10.39
OneVL	NAVSIM	0.52	0.00	25.00	3.65	16.69	7.05	77.72	50.69	4.87
Qwen-Drive-1.0-4B	WOD-E2E	1.82	0.00	15.62	3.12	17.05	7.09	76.68	43.67	4.31

Table 1: CARLA-Language2Drive main results. 384 core rollouts and 192 paired instances per system (64 PCUs, three expressions). PSM uses 24 additional routes. All values are on a 0–100 scale; higher is better. Strict individual and paired success are shown first; L2D follows the component scores.

System	Strict success		Relaxed proxy		Graded		Driving		Summary
System	SISR	CFPS	SemISR	Sem- CFPS	SCS	PCS	Safety	Driving Score	L2D
Alpamayo2-Super	13.33	0.00	18.33	0.00	21.43	0.00	93.33	97.10	9.22
Alpamayo 1.5	11.67	0.00	15.00	0.00	18.12	1.11	100.00	97.17	8.38
SimLingo	6.67	3.33	13.33	6.67	16.67	10.00	68.33	83.09	7.58
AutoVLA	1.67	0.00	8.33	6.67	9.73	6.67	41.67	76.94	3.36
Qwen-Drive-1.0-4B	0.00	0.00	1.67	0.00	4.44	0.00	43.33	69.16	0.92

Table 2: RealSim main results. Scoring v0.2 on 30 PCUs from 23 reconstructed scenes: 60 rollouts and 30 pairs per system. All values are on a 0–100 scale; higher is better. Strict individual and paired success are shown first; L2D follows the component scores.

Figure 3: Advancement and behavioral response do not establish complete instruction execution. (a,b) P11 requests lane keeping (R0, blue) or a left lane change (R1, orange dashed). Alpamayo 1.5 reaches 100 DrivingScore in both arms but succeeds only on R0. P22 requests 3–7 or 8–12 km/h. Shading marks the requested speed bands, not their active time windows. Curves cover the full post-instruction interval; verification applies from entry into the annotated segment to exit and requires complete traversal and compliance at all active samples. SimLingo passes both criteria; Alpamayo 1.5 passes neither. Figure 9 gives the segment geometry.

Figure 4: Open-loop instruction following track on three datasets. Predicted trajectories are scored for exclusive binary semantic success (SemSISR/SemCFPS) and graded completion (SCS/PCS). All values use a 0–100 scale.

	Execution setting		Instruction evaluation
Benchmark / protocol	World	CL	Language input	Matched context	Alt. targets	Behavior check	Para. test	Paired success
nuPlan 2021	R	✓	–	–	–	–	–	–
Bench2Drive 2024	S	✓	–	–	–	–	–	–
NAVSIM v2 2025	R	–	–	–	–	–	–	–
LangAuto 2024	S	✓	✓	–	–	–	–	–
Action Dreaming 2025	S	–	✓	✓	✓	✓	–	–

Table 3: Comparison of driving evaluation protocols. S: synthetic worlds; R: real-log-derived contexts, not on-road tests. CL: sequential closed-loop execution. Matched: instruction variants share a scene or route context. Alt.: variants have distinct scored behavioral targets. Behavior check: instruction-specific acceptance beyond route-level metrics. Para.: explicit paraphrase evaluation. Paired success: both opposed, feasible requests must succeed.

Appendix figures & tables45 assets

Supplementary material from the paper’s appendix.

Appendix

Map identifier	PCUs	Core rollouts
American-city0428	11	66
Town01	2	12
Town02	1	6
Town03	10	60
Town04	4	24
Town05	2	12

Appendix

Table 4: Actual CARLA base map manifest. These ten evaluated layouts include five custom non-Town assets. The distribution-level six-environment description and this evaluation manifest have different scopes.

Cohort	PCUs	Arms	One seed	Three seeds
Source	128	256	768	2304
Small	32	64	192	576
Base	64	128	384	1152
Large (loadable)	119	238	714	2142

Appendix

Table 5: Compiled source suite and nested CARLA cohorts. Columns count PCUs and potential Core executions; only the base single-seed cohort is the complete comparison.

Stratum	Category	PCUs	Rollouts
A	junction	7	42
A	lane	4	24
A	speed_start	2	12
A	waiting_parking	3	18
B	lane_constraint	4	24
B	object_relation	4	24

Appendix

Table 6: CARLA base task inventory, with three paraphrases per arm and seed 1.

PCU	Arm 0	Arm 1
A-001	Turn left at this intersection.	Go straight through this intersection.
B-025	Safely pass the slow truck.	Do not overtake; remain behind the slow truck.
C-001	Take Middleton Street.	Take E Gage Ave.
C-025	Park in space B204.	Park in space B205.
D-001	First visit B208, then go to F206.	First visit F206, then go to B208.
D-009	If E101 is occupied, park in E102; otherwise use E101.	If E101 is occupied, park in C104; otherwise use E102.

Appendix

Table 7: Examples of the evaluated CARLA instruction pairs.

System	Effective input	Action path / important configuration
LMDrive	Four RGB views, LiDAR, ego state	Native visual encoder/LLM, waypoint head, PID; public-language local-goal adaptation
SimLingo	Front RGB, motion state	InternVL2-1B, epoch013; native waypoints/PID
Alpamayo 1.5	Custom multi-camera temporal history	Trajectory generation and tracking; multi-camera temporal input
Qwen-Drive-1.0-4B	Front/left/right RGB, ego	Backbone with planner-rl; sampled plans and trajectory tracking
OneVL	Single front view, ego/history	NAVSIM-format latent-prefix trajectory generation
Impromptu-VLA	Front/right/left RGB, ego history	Released 3B AD-finetuned QA-format waypoint generation

Appendix

Table 8: Complete CARLA system paths. A bridge denotes an interface that differs from the original end-to-end architecture.

System	SemSISR	SemCFPS	SCS	PCS
WOD-E2E
DriveMA-2B	38.38	1.24	49.33	7.88
Fast-dDrive	34.85	0.00	46.42	5.27
Qwen-Drive-SFT	34.44	0.00	50.58	9.37
Qwen-Drive-RL	35.06	0.00	49.58	7.58
NAVSIM

Appendix

Table 9: Open-loop instruction following on three datasets. Predicted trajectories are scored for exclusive binary semantic success (SemSISR/SemCFPS) and graded completion (SCS/PCS). All values use a 0–100 scale.

Source	Available source / construction	Evaluation subset	Queries per model
nuScenes	150 test scenes; 6,008 anchors; 83 candidates	75 approved PCUs in 75 scenes / 15 logs; dev 15, heldout 60	900 across 75 PCUs; 720 heldout, including controls
NAVSIM	OpenScene test; 40 reviewed candidates	38 released PCUs / 19 original recordings	304: two arms $\times$ four wordings $\times$ 38 PCUs
WOD-E2E	38/93 validation shards; 43,394 frames; 203 rated anchors	563 complete-window candidates; 441 with P0 drafts; 241 motion-proxy pairs	882 draft requests; 482 belong to the proxy subset

Appendix

Table 10: Open-loop coverage of the constructed Language2Drive subsets. Candidate, qualified-PCU, and request counts are reported separately.

Source	Horizon	Interval	Points	Output and fixed anchor reference
nuScenes	3.0 s	0.25 s	12	$(x,y)$ ; ego_pose_origin
NAVSIM	4.0 s	0.50 s	8	$(x,y,\psi)$ ; rear-axle origin
WOD-E2E	5.0 s	0.25 s	20	$(x,y)$ ; rear-axle midpoint

Appendix

Table 11: Prediction grids and dataset-specific anchor reference points. Future points exclude

t=0

Category	Released	Dev	Heldout
Lane change / keep	22	5	17
Left-change target	12	3	9
Right-change target	10	2	8
Scoped speed: approximately 15 / 30 km/h	53	10	43
Boston-seaport	41	7	34
Singapore-onenorth	12	4	8

Appendix

Table 12: Released nuScenes task and geographic distributions. The paired lane alternatives are change/keep, not left/right within the same PCU.

Parameter	Value or exact rule
Lane-task initial speed	$4.5\leq v_{0}\leq 9.0$ m/s; released range 4.745680–8.878580
Adjacent-lane query	7 m radius; polygon gap $\leq 0.8$ m
Divider / direction	Divider exists and every segment type contains DASHED ; target yaw error $\leq 20^{\circ}$
Relative target position	Lateral distance 2.4–5.0 m on the requested side; absolute forward offset $\leq 5.0$ m
Candidate ranking	$\|d_{\mathrm{lat}}-3.5\|+0.1\|d_{\mathrm{fwd}}\|$ ; search left before right
Lane witness blend	Start 0.4 s; duration 2.0 s; initial centerline offset decays over 1.0 s

Appendix

Table 13: nuScenes target-construction parameters. These mining and witness parameters are distinct from the final scoring tolerances.

Arm	Predicate	Required samples
Lane-change arm 0	Reference point lies in the target lane corridor	$t=2.25,2.50,2.75,3.00$ s; 4 points
Lane-keep arm 1	Reference point lies in the current lane corridor	$t=0.25,0.50,\ldots,3.00$ s; 12 points
Low-speed arm 0	$3.2\leq v_{k}\leq 4.8$ m/s	Interval ends $t_{k}=1.25,\ldots,3.00$ s; 8 intervals
High-speed arm 1	$7.2\leq v_{k}\leq 8.8$ m/s	Same 8 intervals

Appendix

Table 14: Complete nuScenes binary semantic conditions, in addition to

D\geq 8

m. Every required sample must satisfy its condition.

Diagnostic	Exact definition / threshold	Invalid handling
Speed-target MAE	Mean $\|3.6v-v_{}\|$ on 8 intervals, $v_{}=15$ or 30 km/h	Valid-arm mean
Speed-band violation	$3.6\,\mathrm{mean}\max(\ell-v,0,v-h)$ , km/h	Valid-arm mean
Speed coverage	In-band fraction over 8 intervals, without displacement gate	Zero for invalid arms
Paired speed change	$\Delta v=3.6[\mathrm{mean}(v_{\mathrm{high}})-\mathrm{mean}(v_{\mathrm{low}})]$	Valid-pair mean
Lane distance / coverage	Mean distance to own corridor; inside fraction over 4 change or 12 keep points	Distance conditional; coverage zero if invalid
Lane preference	$q=\mathrm{mean}(d_{\mathrm{current}}-d_{\mathrm{target}})$ on the common window; $\Delta q=q_{\mathrm{change}}-q_{\mathrm{keep}}$	Valid arm / pair

Appendix

Table 15: Additional nuScenes diagnostics and their full numeric conventions. Conditional means always accompany validity coverage.

Family	Scope	PCUs	Share	Counterfactual alternatives
Lane choice	A	12	31.58%	Keep versus change one lane; eight left-target and four right-target PCUs
Junction branch	A	1	2.63%	Left turn versus straight at a common junction
Pass / follow	A	1	2.63%	Follow the same actor versus pass using an adjacent lane
Speed band	B	7	18.42%	Two disjoint speed bands over the declared interval
Object-relative speed	B	2	5.26%	Two speed requirements while following the same actor
Landmark stop	C	10	26.32%	Distinct stopping locations grounded in visible landmarks

Appendix

Table 16: NAVSIM 1.0.0 task composition and documented counterfactual targets. Counts and percentages use the 38-PCU release, not the earlier 12-candidate pool.

Component	Frozen scoring rule / parameter source
Common grid / quality	8 poses at 0.5–4.0 s in the rear-axle frame; maximum acceleration 3.5 m/s 2 and maximum yaw rate 0.8 rad/s
Lane / local goal	Vehicle-footprint overlap, terminal progress, and target polygons from the per-PCU private contract
Junction goal	Goal region, active interval, and deadline stored in the per-PCU contract
Landmark stop	Landmark-grounded stop region, stopping threshold, and hold requirement stored in the private contract
Speed band	Per-PCU lower/upper speed bands and active time range; primitive completion decays linearly over the native tolerance
Object-relative speed	Speed-band requirement combined with the bound lead-actor geometry and relative-position constraints

Appendix

Table 17: NAVSIM task-specific scoring components. Global rules are shown here; per-PCU geometry and thresholds are read from the frozen private sidecars.

Model	Released inputs and instruction placement	Common-grid conversion
Qwen-Drive	Four frames of front/front-left/front-right; 16-point interpolation of observed 1.5 s history; full instruction in scene prompt	One reasoning-mode RL-planner sample; 10 Hz future points numbered 5,10, $\ldots$ ,40
AutoVLA	Four frames of three forward cameras; current velocity/acceleration; driving-instruction field	First eight released 0.5 s action-token poses
DriveMA-2B	Current three forward cameras; observed 1.5 s history and state; high-level-intent field	Released two-turn decision/trajectory protocol; 1 Hz positions linearly resampled to 0.5 s
ReCogDrive	Current front image; four observed poses and state; text replaces discrete command phrase	Released eight-pose DDIM planner output
OneVL	Current front image; three historical poses and state; full Command field	Released latent-chain-of-thought generation of eight 0.5 s poses

Appendix

Table 18: Frozen NAVSIM adapters. Input differences are retained and disclosed rather than hidden by the common output grid.

Model	Repository commit	Primary checkpoint revision
Qwen-Drive	28091c1532e869bc7aee91fc0aef6b3e6fd0b2e0	28484089a7cc8c335cf5089fb0745cf7c49b6eaa
AutoVLA	ba34eed74ce6729e7986592d0e66cbaca397b4fa	a7d7ba3ed7529b248d2694c2defa31b35208340f
DriveMA-2B	de20e6c64878bc3413d26d69362c3be9b3a91533	d5f4905d2032da943a7ae8f22d5b2b8efbe4b0f7
ReCogDrive	6b8d8f5e01346c71094651c81dcaf66405dbc04e	7346ede73f03ed577811024f1a5d16fb063a47b9
OneVL	f4e6f05572b2f66611259bcd5563529833f30469	c1e893e9a03b8312f9328d2b51267836a9f7bf8a

Appendix

Table 19: NAVSIM model revision identifiers from the frozen experiment protocol. Line breaks inside identifiers are typographic only.

Task family	Pairs	% of 563	% of 441	Requests	Evaluated PCUs
SPEED_CHOICE	175	31.08	39.68	350	165
STOP_GO	112	19.89	25.40	224	76
LATERAL_CHOICE	112	19.89	25.40	224	0
TURN_WAIT	28	4.97	6.35	56	0
TURN_STRAIGHT	14	2.49	3.17	28	0
UNRESOLVED	122	21.67	–	0	0

Appendix

Table 20: WOD-E2E P0 task-family distribution. Candidate construction, language-pair construction, and the final semantic-evaluation set are reported with separate denominators.

Criterion	Binary success	Completion $1$	Completion $0$
Stop tail speed	$v_{\mathrm{tail}}\leq 0.6$	$\leq 0.6$	$\geq 2$
Stop endpoint distance	$d_{\mathrm{stop}}\leq 2$	$\leq 2$	$\geq 5$
Go tail speed	$v_{\mathrm{tail}}\geq 2$	$\geq 2$	$\leq 0.6$
Go forward endpoint	$x_{20}\geq 8$	$\geq 8$	$\leq 0$
Slow late speed	$v_{\mathrm{late}}\leq\ell_{i}$	$\leq\ell_{i}$	$\geq h_{i}$
Normal late speed	$v_{\mathrm{late}}\geq h_{i}$	$\geq h_{i}$	$\leq\ell_{i}$

Appendix

Table 21: WOD-E2E semantic scoring parameters. Speed and distance are reported in m/s and m, respectively. The speed-choice boundaries

\ell_{i}

and

h_{i}

are frozen per-PCU evaluator parameters.

System	Valid outputs	Own-reference ADE $\downarrow$	Aligned pairs
DriveMA-2B	882/882	2.520	6/441
Fast-dDrive	882/882	2.817	5/441
Qwen-Drive-SFT	882/882	2.959	3/441
Qwen-Drive-RL	876/882	2.924	2/436

Appendix

Table 22: WOD-E2E geometric diagnostics on the 441 instruction-paired samples. ADE is measured in meters.

Artifact	Recorded SHA-256
nuScenes public inputs	b2d3a6e2deccb91e1e7d1694ff9589d5577eb89e834c674dc36768bd213962d5
nuScenes private sidecars	f2506b158aa34440999ef6215fe21dc08fccd5138d6db5a0ac4fdc5e8d9ea7ce
nuScenes review sheet	6744858cf01921f43ba6fca11ef7d63d6f3a451203f897c87e289e4a6301b30f
nuScenes binary scorer	436bf1fcb8393b2b2da27974c363eef813dc73849a7063b1aeb947c29fef1eeb
NAVSIM public archive	0e9f010ae39f9decc049ff4ff9f79109e92cd27df27f55b5cc80a18f35d0e799
NAVSIM private archive	e520a6f4455b82e1e6dbfe3c1de30b0786153f5a3dbedf859aae740812bfea6f

Appendix

Table 23: Recorded release identities used to bind the reported experiments to their public, private, evaluator, and scoring artifacts.

System	A	B	C	D
LMDrive	13.54	22.92	6.25	6.25
Alpamayo 1.5	3.12	16.67	12.50	0.00
SimLingo	13.54	12.50	4.17	1.04
Qwen-Drive-1.0-4B	0.00	4.17	3.12	0.00
OneVL	0.00	1.04	1.04	0.00
Impromptu-VLA	1.04	0.00	0.00	0.00

Appendix

Table 24: Strict CARLA instruction success by scope. Each scope has 16 PCUs, 96 rollouts, and 48 paired instances per system. A: atomic; B: persistent; C: grounded; D: program.

Figure 5: Strict CARLA success by control scope, with 96 rollouts per system and scope. Gold markers average six systems. D has mean SISR 1.22%; its seven strict successes all maintain an invariant until an event. Connecting lines join categorical scopes rather than a continuous difficulty axis.

	Strict outcomes				Semantic
System	Flip	All three	Any	Flip/any (%)	Flip
LMDrive	3	14	17	17.65	5
Alpamayo 1.5	12	4	16	75.00	8
SimLingo	9	6	15	60.00	7
Qwen-Drive-1.0-4B	2	1	3	66.67	9
OneVL	2	0	2	100.00	5

Appendix

Table 25: Instruction sensitivity across three CARLA expressions. Counts use 128 semantic arms per system. “All three requires the same arm to succeed under every expression; “any requires at least one. Flip/any conditions on arms with some strict success.

System	A	B	C	D	$I_{s}$	$I_{m}$	All-3
LMDrive	0	0	0	0	3	5	14
SimLingo	2	0	0	0	9	7	6
Alpamayo 1.5	0	2	0	0	12	8	4
Qwen-Drive-1.0-4B	0	0	0	0	2	9	1
OneVL	0	0	0	0	2	5	0
Impromptu-VLA	0	0	0	0	1	15	0

Appendix

Table 26: CARLA paired and paraphrase diagnostics. A–D: successful pairs out of 48.

I_{s}/I_{m}

: numbers of arm groups with inconsistent strict/semantic outcomes across paraphrases, out of 128. All-3: arms strictly successful for all three paraphrases, out of 128. All systems have zero worst-paraphrase paired success out of 64 PCUs.

System	p0 S/M	p1 S/M	p2 S/M	p0 pair	p1 pair	p2 pair
LMDrive	15/36	17/38	15/38	0	0	0
SimLingo	13/61	8/64	9/62	2	0	0
Alpamayo 1.5	11/52	11/55	9/55	1	1	0
Qwen-Drive-1.0-4B	2/20	3/19	2/21	0	0	0
OneVL	0/31	2/33	0/32	0	0	0
Impromptu-VLA	1/25	0/23	0/28	0	0	0

Appendix

Table 27: CARLA per-paraphrase outcomes. S/M entries give strict/semantic success counts out of 128; pair columns count successful pairs out of 64.

System	$L_{\rm bal}$	$D_{t}$	Driving credit	Language credit	L2D
CARLA
Alpamayo 1.5	17.56	66.40	0.66	11.54	12.20
SimLingo	18.26	63.67	0.64	11.51	12.15
LMDrive	14.09	69.48	0.69	9.69	10.39
OneVL	6.66	64.21	0.64	4.23	4.87
Qwen-Drive-1.0-4B	6.23	60.18	0.60	3.71	4.31

Appendix

Table 28: L2D components. The language and driving contributions sum to the final score before rounding. Calculations use the displayed component metrics in Tables 1 and 2 .

System	A: atomic action	B: persistent behavior	Overall
Alpamayo 1.5	4/12 (33.33)	5/48 (10.42)	9/60 (15.00)
Alpamayo2-Super	5/12 (41.67)	6/48 (12.50)	11/60 (18.33)
SimLingo	3/12 (25.00)	5/48 (10.42)	8/60 (13.33)
OneVL	0/12 (0.00)	0/48 (0.00)	0/60 (0.00)
ReCogDrive	0/12 (0.00)	0/48 (0.00)	0/60 (0.00)

Appendix

Table 29: RealSim relaxed semantic outcomes by scope for five systems. Counts use 12 atomic arms and 48 persistent arms; overall rates agree with Table 2 .

System	Pairs	Separation (m)	Speed pairs	Speed diff. (km/h)
Alpamayo2-Super	30	0.23	21	0.46
Alpamayo 1.5	30	0.15	21	0.21
SimLingo	30	2.61	21	3.51
OneVL	30	0.36	21	0.25
ReCogDrive	30	0.10	21	0.07
Qwen-Drive-1.0-4B	29	0.31	20	0.48

Appendix

Table 30: RealSim response to opposed instructions in the trajectory-response analysis. Spatial values are medians over complete valid pairs; speed values use the B1/B3 subset. Each pair first averages differences over 101 matched post-instruction time points. These describe response magnitude, with correctness evaluated separately by CFPS.

System	SI	GPE	PSM	MDQ	RLS	$n_{RLS}$
LMDrive	0.8357	0.2684	0.8884	0.9340	0.4437	23
SimLingo	0.5419	0.7692	0.6230	0.8310	0.6079	23
Alpamayo 1.5	0.4480	0.3009	0.7010	0.9646	0.3152	21
Qwen-Drive-1.0-4B	0.4883	0.0056	0.7668	0.9964	0.0873	24
OneVL	0.2390	0.0463	0.7772	0.9967	0.1332	24
Impromptu-VLA	0.2574	0.0209	0.7743	0.9897	0.0627	23

Appendix

Table 31: Supplementary FDQ measures. SI, GPE, PSM, and MDQ each have

n=24

; the last column reports RLS coverage.

System	Strict	Sem	Safe	Obey	Fall.	Ref.	Alt.
LMDrive	2	3	5	0	0	2	0
SimLingo	0	4	0	0	0	0	0
Alpamayo 1.5	0	4	1	2	0	0	0
Qwen-Drive-1.0-4B	1	2	1	0	0	1	0
OneVL	0	2	0	0	0	0	0
Impromptu-VLA	0	2	0	1	0	0	0

Appendix

Table 32: Safety-priority diagnostic counts. Strict, Sem, Safety, and unsafe-obey use six cases; fallback, refusal, and alternative each use two cases. Responses are behavior-based.

Figure 6: Six environment families. Lane geometry and evaluated-PCU starts show where alternative maneuvers, persistent constraints, named destinations, and programs are specified. Each panel has its own metric scale. The ten evaluated map identifiers are listed in Table 4 .

Figure 7: Sixteen CARLA task families. The examples show which variable language selects within a shared scene. The complete task catalog remains in Tables 33 and 34 .

Figure 8: P11 reference geometry and scene context. The lane alternatives share an initial pose. Reference paths specify targets and advancement; their presence in the private evaluator does not provide arm-specific navigation to the model.

Figure 9: P22 reference speed, active region, and scene context. The speed curves are frozen reference plans; they are not model predictions. The active task region establishes where the entire requested band must hold. The model curves in Figure 3 show post-instruction response, with full-scoring verdicts reported separately.

PCU	Category	Map	Split	End (s)
A-001	junction	American-city0428	dev	45
A-002	junction	American-city0428	dev	45
A-004	junction	American-city0428	dev	45
A-006	junction	American-city0428	dev	45
A-008	junction	American-city0428	test	45
A-010	junction	American-city0428	test	45

Appendix

Table 33: All 64 evaluated CARLA PCUs, grouped by stratum. End is the arm deadline in simulation seconds. Every PCU has two arms and three paraphrases per arm.

PCU	Family	Scene	End (s)
2d72b487	B1	31ab4b94	7.0
09dd1b67	A2	2f380e80	8.0
5cbf0c52	B1	afeb04ed	7.0
c10334f0	B1	cdd9448c	7.0
b92f015d	B1	ce161092	7.0
b9667ab7	B1	0d4893f5	7.0

Appendix

Table 34: All 30 RealSim PCUs. Identifiers are short report identifiers. Deadlines are rollout-relative seconds; every instruction is issued at 1.7 s. Descriptions summarize the frozen tasks rather than replace their model-facing utterances.

Figure 10: A-006/p0: successful exit selection. Both the left and straight runs satisfy their full requested criteria.

Figure 11: A-006/left: strict success varies across equivalent requests. Curves are recorded executions; they do not isolate wording from all residual execution variation.

Figure 12: B-027/p1: successful passing and following. The two speed profiles accompany successful full-criterion verdicts.

Figure 13: C-025/p0: both named parking targets remain uncompleted. Each arm travels more than 160 m but fails its target criterion.

Figure 14: D-001/p0: local movement does not complete either requested sequence. Ordered stage completion determines the verdict.

Figure 15: D-009/p0: neither conditional target is completed. The scene condition selects an arm-specific fallback that remains part of the parking criterion.

Figure 16: D-026/p0: the requested visit-and-return programs remain unfinished. Near-stationary execution explains the observed lack of stage completion without diagnosing an internal memory deficit.

Figure 17: P11: measured trajectories. Each panel contrasts the two lane requests. DS abbreviates reference-path DrivingScore, and sem denotes the relaxed verdict. The full task specification appears in Figure 8 .

Figure 18: P22: measured trajectories. SimLingo completes both requested speed-band criteria. Spatial advancement alone does not determine speed compliance; Figure 3 shows the Alpamayo 1.5 and SimLingo speed histories, and Figure 9 shows the active region.

Explore similar work

ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Driving
Apr 7, 2026Kaiser Hamid, Can Cui, Nade LiangRobustness of VLA Models Autonomous Driving Safety Evaluation
Texas Tech University · Bosch Center for Artificial Intelligence (BCAI)
SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
Oct 14, 2026Shaoyuan Xie, Zihan Zhang, Jingxuan Wang +6Autonomous Driving End-to-End Autonomous Driving
UC Irvine · UC San Diego · University of Cincinnati +1
VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving
Jun 10, 2026Thach Nguyen, Danhua Guo, Tom Lampo +2Autonomous Driving Benchmarks CoT Faithfulness
Uber AV Labs

Apr 7, 2026J/K move · Enter open · S save

ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Driving

Kaiser Hamid, Can Cui, Nade Liang

Texas Tech University · Bosch Center for Artificial Intelligence (BCAI)

Robustness of VLA Models Autonomous Driving Safety Evaluation Autonomous Driving Benchmarks Autonomous Driving Simulation

Open paper AskPDF arXiv

Recent progress in vision-language-action (VLA) models has enabled language-conditioned driving agents to execute natural-language navigation commands in closed-loop simulation, yet standard evaluations largely assume instructions are precise and well-formed. In deployment, instructions vary in phrasing and specificity, may omit critical qualifiers, and can occasionally include misleading, authority-framed text, leaving instruction-level robustness under-measured. We introduce ICR-Drive, a diagnostic framework for instruction counterfactual robustness in end-to-end language-conditioned autonomous driving. ICR-Drive generates controlled instruction variants spanning four perturbation families: Paraphrase, Ambiguity, Noise, and Misleading, where Misleading variants conflict with the navigation goal and attempt to override intent. We replay identical CARLA routes under matched simulator configurations and seeds to isolate performance changes attributable to instruction language. Robustness is quantified using standard CARLA Leaderboard metrics and per-family performance degradation relative to the baseline instruction. Experiments on LMDrive and BEVDriver show that minor instruction changes can induce substantial performance drops and distinct failure modes, revealing a reliability gap for deploying embodied foundation models in safety-critical driving.

Abstract

Explore similar work

ICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Driving

SafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving

VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving