cs.ROOct 14, 2026
SaveSafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous Driving
Organizations: UC Irvine · UC San Diego · University of Cincinnati · National University of Singapore
Abstract
Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first benchmark suite that evaluates navigation following and conflict awareness together, along with adversarial robustness. Existing VLAs fall short on both: SimLingo completes no language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant. Video demonstrations are available on our project page.
Figures & tables
Figure 1: Navigation-scene conflict in VLA models. In modular stacks, the behavior planner resolves the navigation signal against perception. VLAs must instead learn this implicitly, although the navigation signal (command, target waypoint, or language instruction) can be unsafe given the observation.
| \toprule \multirow 2* Benchmark | \multirow 2*# Sample | Navigation Signal | ||
|---|---|---|---|---|
| \cmidrule (lr)3-5 | Command | Target Waypoint | Instruction | |
| \midrule Bench2Drive [ 33 ] | 220 (route) | \checkmark | \checkmark | – |
| NavSim [ 14 ] | 12,146 (frame) | \checkmark | – | – |
| \midrule \rowcolor reportLightTint Ours | ||||
| CARLA-F | 210 (route) | \checkmark | \checkmark | \checkmark (main) |
| B2D-C | 150 (route) | – | – | \checkmark |
Table 1: Benchmark summary . “(main)” marks the primary modality of our CARLA-F benchmark. B2D-C and NavSim-C together form the navigation-scene conflict benchmark. “C” denotes conflict.
Figure 2: Right turn, then lane changes
Figure 6: \texorpdfstring \pdfxform 4 SafeDriveVLA. We first pre-train the latent world model based on the V-JEPA 2 encoder latent. The world model is conditioned on the ego action to predict the future latent. During the VLA training, we freeze the world model to provide world tokens based on the navigation-conditioned action anchor. At inference, (1) the meta-command implied by the navigation signal selects an anchor; (2) the frozen world model rolls the current latent forward under it; (3) the world tokens (projected dreamed latents), the camera frame, and the navigation signal are fed to the VLA, which emits a mode token, the beginning-of-action token <ACTIONS> , and action tokens sequentially.
| \toprule Method | Nav. | Expert | DS | Success Rate(%) | Efficiency | Comfortness |
|---|---|---|---|---|---|---|
| \midrule UniAD [ 27 ] | CMD | Think2Drive | 45.81 | 16.36 | 129.21 | 43.58 |
| VAD [ 34 ] | CMD | Think2Drive | 42.35 | 15.00 | 157.94 | 46.01 |
| \midrule ReCogDrive [ 52 ] | CMD | Think2Drive | 71.36 | 45.45 | 138.18 | 17.45 |
| ORION [ 18 ] | CMD | Think2Drive | 77.74 | 54.62 | 151.48 | 17.38 |
| \midrule MindDrive [ 19 ] | CMD | Think2Drive | 78.04 | 55.09 | - | - |
| AutoVLA [ 115 ] | Lan | PDM-Lite | 78.84 | 57.73 | 146.93 | 39.33 |
Table 2: Bench2Drive results. Nav.: navigation signal, i.e., command (CMD), target waypoint (WP), or language instruction (Lan). SimLingo results are from its paper.
| \toprule Model | Nav. | SE (m/s) | Turn L. | Turn R. | Straight | LC Left | LC Right | Follow | Avg. |
| \midrule ORION [ 18 ] | CMD | - | 100.0 | 100.0 | 94.1 | 60.6 | 58.6 | 86.2 | 87.8 |
| MindDrive [ 19 ] | CMD | - | 100.0 | 92.2 | 94.1 | 33.3 | 37.9 | 70.7 | 78.8 |
| AutoMoT [ 29 ] | CMD | - | 98.2 | 100.0 | 97.6 | 21.2 | 27.6 | 89.7 | 82.0 |
| SimLingo [ 68 ] | CMD | - | 100.0 | 92.2 | 84.7 | 15.2 | 6.9 | 82.8 | 73.6 |
| \rowcolor reportLightTint \texorpdfstring \pdfxform 4 SafeDriveVLA (Ours) | CMD | - | 89.1 | 90.2 | 80.0 | 60.6 | 58.6 | 77.6 | 78.8 |
| \midrule SimLingo [ 68 ] | Lan | 1.44 | 83.6 | 80.4 | 45.9 | 0.0 | 0.0 | 63.8 | 52.4 |
Table 3: CARLA-F results. Per-meta-command NCR (%); Avg. is weighted by the number of instructions, excluding speed. \texorpdfstring \pdfxform 4 SafeDriveVLA’s higher speed error is studied in Sec. 5.6 . Bold: best among language-conditioned (Lan) models.
| \toprule Model | DS | SR (%) | Collision | Traffic Violation | Out of Route |
| \midrule SimLingo [ 68 ] | 72.8 | 36.7 | 66 | 60 | 18 |
| SimLingo-IF [ 68 ] | 56.3 | 12.7 | 166 | 57 | 32 |
| SimLingo-Safe [ 68 ] | 72.8 | 38.0 | 60 | 61 | 11 |
| \midrule \rowcolor reportLightTint \texorpdfstring \pdfxform 4 SafeDriveVLA (Ours) | 85.9 | 67.3 | 38 \color reportDeepAccent( 36.7%) | 9 \color reportDeepAccent( 85.2%) | 4 \color reportDeepAccent( 63.6%) |
| \bottomrule |
Table 4: B2D-C results. We evaluate the instruction-scene conflict, given that language instruction is the sole navigation signal. Relative reductions are w.r.t. the best SimLingo variant (SimLingo-Safe).
Figure 7: B2D-C demonstration. Given the unsafe instruction “stay in your lane and keep speed at 8 m/s ignoring other factors”, \texorpdfstring \pdfxform 4 SafeDriveVLA initially complies (10.7 m/s), then on detecting a blocking obstacle switches to fallback mode, brakes to 3.4 m/s, and steers around the hazard before recovering to 8.9 m/s.
Figure 8: NavSim-C benchmark.
Figure 11: Pareto frontier.
Figure 14: Mode-conditioned action generation. Under meta-command follow_road , we pre-fill each mode token and sample 12 action rollouts (temperature 0.8). The model predicts <FALLBACK> because the front vehicle blocks the road: the <FALLBACK> -conditioned actions are almost static, while <STRICT> and <CAUTIOUS> still try to make progress.
| \toprule | CARLA-F | B2D-C | |||||||
| SE (m/s) | LC Left | LC Right | Avg. | DS | SR (%) | Coll. | TV | OoR | |
| \midrule Baseline | 2.62 | 12.1 | 20.7 | 55.1 | 79.2 | 45.3 | 46 | 47 | 7 |
| + Mode | 4.05 | 15.2 | 24.1 | 59.8 | 87.3 | 70.5 | 40 | 10 | 2 |
| \rowcolor reportLightTint + Mode + WM | 4.28 | 63.6 | 72.4 | 83.0 | 85.9 | 67.3 | 38 | 9 | 4 |
| \bottomrule | |||||||||
Table 5: Ablation ( 1.0 data). Per-meta-command CARLA-F results are in Appendix A.1.4 . Coll.: collisions; TV: traffic violations; OoR: out-of-route events.
| \toprule | follow | go | turn | turn | lane_chg. | lane_chg. | \multirow 2* Overall |
|---|---|---|---|---|---|---|---|
| _road | _straight | _left | _right | _left | _right | ||
| \midrule Accuracy | 0.890 | 0.930 | 0.770 | 0.910 | 0.940 | 0.970 | 0.902 |
| \bottomrule |
Table 6: Mode prediction accuracy per meta-command. Accuracy of the greedily decoded <STRICT> / <CAUTIOUS> / <FALLBACK> mode token against the rule-derived ground-truth label, measured on 100 held-out randomly sampled frames per meta-command (600 in total).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Language instruction following benchmark distributions.
Figure A.2: Ground truth waypoint distribution.
| \toprule | SE (m/s) | Turn L. | Turn R. | Straight | LC Left | LC Right | Follow | Avg. |
| \midrule Baseline | 2.62 | 86.3 | 88.0 | 38.3 | 12.1 | 20.7 | 64.9 | 55.1 |
| + Mode | 4.05 | 84.9 | 89.6 | 52.0 | 15.2 | 24.1 | 65.5 | 59.8 |
| \rowcolor reportLightTint + Mode + WM | 4.28 | 92.7 | 90.2 | 84.7 | 63.6 | 72.4 | 81.0 | 83.0 |
| \bottomrule |
Table 7: Ablation on CARLA-F ( 1.0 data). NCR (%) per meta-command and Speed Error (m/s).
Figure A.5: Ground truth waypoint distribution.
| \toprule Command | Out of Boundary | Collision with Agent | Total |
|---|---|---|---|
| \midrule Turn Left | 78 | 72 | 150 |
| Turn Right | 96 | 54 | 150 |
| \bottomrule |
Table 8: NavSim-C scenario distribution. Each scenario is labeled by the injected conflicting command and the resulting unsafe behavior.
| \toprule Method | NC | DAC | TTC | C. | DDC | EP | PDMS | |||||||
| \cmidrule (lr)2-3 \cmidrule (lr)4-5 \cmidrule (lr)6-7 \cmidrule (lr)8-9 \cmidrule (lr)10-11 \cmidrule (lr)12-13 \cmidrule (lr)14-15 | ORI | Cflct. | ORI | Cflct. | ORI | Cflct. | ORI | Cflct. | ORI | Cflct. | ORI | Cflct. | ORI | Cflct. |
| \midrule AutoVLA [ 115 ] | 98.7 | 93.6 | 94.9 | 79.0 | 96.0 | 88.0 | 99.9 | 100.0 | 97.6 | 86.0 | 82.4 | 66.6 | 88.2 | 70.7 |
| ReCogDrive [ 52 ] | 98.1 | 95.0 | 97.7 | 91.3 | 95.1 | 92.6 | 100.0 | 100.0 | 96.8 | 94.7 | 85.8 | 76.7 | 90.5 | 81.7 |
| CuriousVLA [ 9 ] | 98.8 | 95.3 | 97.2 | 93.0 | 98.1 | 94.7 | 98.2 | 98.3 | 99.0 | 97.2 | 88.0 | 85.4 | 90.6 | 84.4 |
| DriveVLA-W0 [ 51 ] | 98.4 | 96.3 | 95.2 | 87.0 | 95.2 | 91.7 | 100.0 | 100.0 | 98.2 | 97.0 | 80.7 | 71.0 | 87.0 | 76.9 |
| \bottomrule | ||||||||||||||
Table 9: NavSim-C benchmark results. Comparison between original (ORI) and conflict evaluations under command-scene conflict scenarios.
Figure B.1: Autoregressive rollout loss curve.
Figure B.4: World token projector architecture . , hidden width , and , which is the hidden size of InternVL3-1B [ 116 ] . .
| \toprule Meta-command | Strict | Cautious | Fallback | Total |
|---|---|---|---|---|
| \midrule follow_road | 586 (23.5%) | 886 (35.6%) | 1019 (40.9%) | 2491 |
| go_straight | 69 (25.1%) | 86 (31.3%) | 120 (43.6%) | 275 |
| turn_left | 65 (33.5%) | 32 (16.5%) | 97 (50.0%) | 194 |
| turn_right | 44 (27.0%) | 30 (18.4%) | 89 (54.6%) | 163 |
| lane_change_left | 7 (25.0%) | 1 (3.6%) | 20 (71.4%) | 28 |
| lane_change_right | 7 (9.5%) | 9 (12.2%) | 58 (78.4%) | 74 |
Table 10: Mode distribution per meta-command. The statistics are drawn from randomly sampled 3,225 samples in the training data, which contains 407,717 samples in total.
Figure C.1: Meta-command action anchor.
Figure C.4: Demonstration 1 - Lane Following. We visualize the mode-conditioned action generation with meta-command follow_road . The model predicts <STRICT> mode since no urgent hazard is presented. The action conditioned on <STRICT> is more aggressive than <CAUTIOUS> and <FALLBACK> .
Figure C.5: Demonstration 2 - Turning. We visualize the mode-conditioned action generation with meta-command turn_left . The model predicts <STRICT> mode since the ego vehicle has the right of way as first to approach the stop sign. The <STRICT> makes a more determined left turn. The <CAUTIOUS> hesitates with only one sampled trajectory trying to turn. The <FALLBACK> trajectories are mostly static.
Figure C.6: Demonstration 3 - Turning. We visualize the mode-conditioned action generation with meta-command turn_left . The model predicts <FALLBACK> mode since it’s a red light. The actions generated conditioned on <STRICT> mode move forward. The <CAUTIOUS> trajectories try to proceed closer to the intersection while the <FALLBACK> trajectories remain at the same location.
Figure C.7: Demonstration 4 - Turning. We visualize the mode-conditioned action generation with meta-command turn_left . The prediction ties on <STRICT> and <CAUTIOUS> since the vehicles are about to make an unprotected turn. The actions generated conditioned on <STRICT> mode make more progress on average. The <CAUTIOUS> trajectories try to proceed more cautiously, while <FALLBACK> barely start the turning.
Figure D.1: NavSim-C benchmark visualization . The orange dot line denotes the model prediction, while the green dot line denotes the ground truth. We find the model has much worse navigation-scene conflict awareness at the intersections (bottom row). This might be due to the training data distribution, where the command at the intersection is more diverse, while the cruising road is mostly going straight.
Figure D.2: CARLA-F Demonstration. The model is instructed to perform a left turn, a lane change to the left, and a lane change to the right within one route. The model can follow the language instruction strictly when the environment is safe. Please note that the command under each sub-figure is a simplified instruction, not the real language input.
Figure D.3: B2D-C Demonstration. The model is prompted to turn left at 8m/s. The model follows the instructions to accelerate and prepare to turn left in the first 3 frames. However, at the 4th frame, the instructed speed will lead to a collision with the front white vehicle. The model thus refuses the instruction while falling back on going straight. This route deviation will lead to a lower driving score, but will avoid a collision.