Vision-Language-Action models (VLAs) and World-Action Models (WAMs) have emerged in autonomous driving with the promise of strong generalization to long-tail scenarios. Although these models achieve state-of-the-art results on autonomous driving benchmarks, most existing benchmarks are dominated by nominal scenarios and therefore cannot test this promise. To test one such long-tail capability that is critical for safety, we introduce ReactBench-AD, a benchmark that measures how driving models react to objects that appear in the ego vehicle's path. Built on real NAVSIM and PhysicalAI-Autonomous-Vehicles scenes, ReactBench-AD composites a crossing pedestrian, a standing pedestrian, or a stationary car consistently across all camera views and history frames. Each edited scene thus forms an exact, pixel-level counterfactual pair with its original, which allows us to isolate the causal effect of the inserted object on the predicted trajectory. Evaluating four VLAs and two WAMs, we find that they react weakly, often barely at all. Finally, we test how easily one of the evaluated VLAs, ReCogDrive, can be taught to brake hard for imminent hazards: a short fine-tuning stage on edited scenes substantially improves its reactions, with the learned behavior generalizing to unseen objects and fully real scenes, at minimal cost to nominal driving. These results call into question the out-of-the-box long-tail capabilities of these models, but suggest that they can be a strong foundation for acquiring them with little additional training, highlighting the value of real or synthetic data that demonstrates the long-tail behaviors nominal datasets rarely contain. Code and data will be made publicly available.
Figures & tables
Figure 1: We insert an object, here a standing pedestrian, into the ego vehicle’s path in a real driving scene, so that the edited scene SE and the original scene SO differ only in the pixels of the inserted object. Bottom: distance travelled by each model’s front bumper over its 4 s plan in the original ( yO ) and edited ( yE ) scene, with the pedestrian’s extent shown in yellow. All six evaluated VLAs and WAMs shorten their plans only slightly, if at all, and drive through the pedestrian, whereas the ReCogDrive IL checkpoint stops before it after a short fine-tuning on edited scenes (IL + RT, Sec. 5 ).
Figure 2: Overview of ReactBench-AD . Left: we pick a target on the ego vehicle’s ground-truth trajectory (dark line), namely the ego’s front-bumper position at a look-ahead time τ∈{0.5,1.0,1.5,2.0}s after the current frame t (circles). At each history timestep, a standing object is placed at the target, whereas a moving one, such as the crossing pedestrian shown here, is placed along a path that reaches the target at t+τ (here τ=0.5s ). Middle: we composite the object at these positions into every camera view and history frame of the model’s input (top: the original scene at the current frame t ). Right: in the edited scene, we score the model’s plans for the original ( yO , dashed) and edited ( yE , solid) input ( Eq. 1 ), here for ReCogDrive.
Crossing pedestrian
Standing pedestrian
Car
Model
Metric
τ=0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
ReactNAV
Alpamayo 1.5
ΔP
+0.25
+3.10
+7.16
+8.71
+0.09
+3.66
+11.93
+17.63
+0.12
+3.84
+15.92
+28.81
ΔCS
−1.51
−2.73
−2.68
−2.29
−1.63
−4.52
−5.44
−4.83
−1.23
−4.29
−7.05
−7.25
Alpamayo 2
ΔP
+0.06
+2.11
+6.75
+8.36
+0.03
+3.25
+9.42
+12.33
+0.12
+5.36
+19.39
+22.43
ΔCS
−1.11
−1.77
−1.42
−0.73
−1.49
−3.10
−3.18
−2.50
−1.19
−2.96
−4.17
−4.22
Table 1: Results on ReactNAV and ReactPAI by model, inserted object, and look-ahead time τ (s). ΔP is in percentage points (higher is better); ΔCS is in km/h (lower is better). Gray: the paired 95% bootstrap CI includes zero.
Figure 3: Trajectory invariance rate, TIRϵ : the percentage of samples whose original and edited trajectories differ by an ADE below ϵ , i.e., for which the model essentially ignores the inserted object (higher means less reactive). Results are aggregated across all four look-ahead times τ .
Figure 4: Effect of inserted objects on the CoTs of Alpamayo 1.5 and Alpamayo 2 on ReactPAI , aggregated across all four look-ahead times τ . Bars show the distribution of the CoT’s leading action: slow/stop (stop, yield, or slow down), keep distance (keep a distance from or follow a lead vehicle), or other . CoT changed : percentage of edited-scene CoTs that differ from the original-scene CoT of the same sample. Names the actor : percentage of edited-scene CoTs that mention a pedestrian or, for inserted cars, a vehicle (in parentheses: original-scene CoTs).
Figure 5: Effect of CoT swapping on NC for Alpamayo 1.5 (A1.5) and Alpamayo 2 (A2) on ReactPAI at τ=2.0s . The first and second letters give the source of the visual input and of the CoT: original (O) or edited (E). Whiskers indicate 95% bootstrap CIs.
ΔP ( v<20km/h )
Model
Stage
PDMS
Cross. ped.
Stand. ped.
Car
ReCogDrive
VLM SFT
84.1
+2.38
+4.99
+10.39
IL
86.5
+0.85
+1.36
+4.88
RL
90.8
+0.51
+0.62
+3.29
SimWAM
IL
90.3
+0.45
+2.10
+6.13
RL
91.9
+0.40
+1.25
+4.14
Table 2: Safety improvement ΔP (in percentage points) on ReactNAV at τ=2.0s for each training-stage checkpoint of ReCogDrive and SimWAM, over samples with an initial ego-vehicle speed below 20 km/h ( n=1762 ). Gray: the paired 95% bootstrap CI of the difference to the previous training stage includes zero. PDMS on navtest as reported in [ 22 , 53 ] .
ReactNAV
navtest
ΔP↑
ΔCS↓
NC↑
DAC↑
TTC ↑
EP ↑
C ↑
PDMS ↑
IL
+0.5
−0.3
98.1
94.4
94.2
80.6
100.0
86.2
IL + RT
+41.4
−9.9
98.7
94.5
94.9
79.3
99.9
86.0
Table 3: Results of ReCogDrive’s IL checkpoint (IL) and of the same checkpoint fine-tuned on ReactTrain (IL + RT) on ReactNAV , aggregated across all look-ahead times τ , and on navtest (PDMS and its sub-metrics on LQR-executed trajectories, from our own evaluation). Units are as in Tab. 1 .
Figure 6: Plans of the IL and IL + RT checkpoints for the 291 unseen-objects samples, each at one random τ , in the original ( yO ) and edited ( yE ) scenes; each line traces the front bumper over 4 s. Although ReactTrain contains no animals, IL + RT shortens its plans in the edited scenes, while its original-scene plans stay close to those of the IL checkpoint.
Unseen objects
Real scenes
ΔP↑
ΔCS↓
P↑
CS↓
IL
+0.1
−0.2
6.8
21.4
IL + RT
+28.1
−8.7
54.2
14.3
Table 4: Generalization of the IL and IL + RT checkpoints to unseen objects and real scenes, aggregated across all four look-ahead times τ . Real scenes have no original trajectory, so we report absolute values: P in percent and the mean contact speed, CS , in km/h.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Randomly selected ReactNAV (top) and ReactPAI (bottom) samples for each inserted object and look-ahead time τ , shown in the edited front-camera image at the current frame t . Each row shows a single scene, whose four look-ahead variants share the same asset.
τ (s)
Object
0.5
1.0
1.5
2.0
All
ReactNAV
Crossing ped.
99.9
99.9
99.5
99.3
99.7
Standing ped.
100.0
100.0
99.9
99.8
99.9
Car
100.0
100.0
99.8
99.6
99.8
ReactPAI
Appendix
Table 5: Share of edited samples (%) in which D-FINE-X detects the inserted object at its location, by object and look-ahead time τ .
Model
τ=0.5
1.0
1.5
2.0
Mean
ReCogDrive
0.0
0.0
1.0
22.7
5.9
DriveWAM
0.0
0.0
0.6
28.9
7.4
SimWAM
0.0
0.0
0.0
23.4
5.8
Appendix
Table 6: P (in percent) on the 308 real scenes with a stationary vehicle ahead, by look-ahead time τ (s).
km/h
5–10
10–15
15–20
20–25
25–30
30–35
35–40
Total
ReactNAV
325
634
803
680
484
271
31
3,228
ReactPAI
180
188
199
137
127
110
104
1,045
Appendix
Table 7: Number of samples in each ego-vehicle speed interval at the current frame for ReactNAV and ReactPAI .
Model
Action head
Inputs
Outputs
Cameras
Frames per camera
Visual history (s)
Ego history (s)
Vel./acc.
Reasoning
Trajectory horizon (s)
Alpamayo 1.5
Transformer Decoder
4
4
0.3
1.5
No
Yes
6.4
Alpamayo 2
Transformer Decoder
6
4
0.3
1.5
No
Yes
6.4
AutoVLA
–
3
4
1.5
–
Yes
Optional
5.0
ReCogDrive
Diffusion Transformer
1
1
0
1.5
Yes
No
4.0
DriveWAM
–
1
1
0
1.5
Yes
No
4.0
Appendix
Table 8: Comparison of the action heads, input configurations, and outputs of the evaluated models. Visual and ego history denote the temporal spans of camera observations and past ego states, respectively; a visual history of 0 s indicates a single current frame.
Crossing pedestrian
Standing pedestrian
Car
Model
Metric
τ=0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
ReactNAV
Alpamayo 1.5
P
O
0.00
0.06
2.11
5.89
0.00
0.03
1.55
5.11
0.00
0.00
1.15
3.69
E
0.25
3.16
9.26
14.59
0.09
3.69
13.48
22.74
0.12
3.84
17.07
32.50
Δ
+0.25
+3.10
+7.16
+8.71
+0.09
+3.66
+11.93
+17.63
+0.12
+3.84
+15.92
+28.81
NCins
O
0.00
0.15
2.94
9.23
0.00
0.19
3.22
11.06
0.00
0.03
1.86
6.32
Appendix
Table 9: Detailed results on ReactNAV and ReactPAI by model, inserted object, and look-ahead time τ (s). O and E denote the trajectories predicted from the original and edited scene, respectively, both evaluated in the edited scene, and Δ=E−O . P and NCins are in percent and their deltas in percentage points (higher is better). CS is the mean contact speed in km/h over the samples for which both trajectories result in an at-fault collision (lower is better). Deltas are computed before rounding.
Figure 8: Safety improvement as a function of ego-vehicle speed at the current frame aggregated across all four look-ahead times τ . Shaded areas indicate 95% bootstrap CIs.
Figure 9: Example predictions at a look-ahead time of τ=2.0s , one for each model and inserted object: Alpamayo 1.5 and Alpamayo 2 on ReactPAI , and the other four models on ReactNAV . Each example shows the edited front-camera image at the current frame, cropped around the inserted object, next to a bird’s-eye view of the edited scene. The arrow marks the crossing pedestrian’s motion over the 4 s horizon. Dashed and solid lines show the model’s plans for the original ( yO ) and edited ( yE ) scenes, respectively, as front-bumper paths over the first 4 s. Examples are drawn at random from the samples whose original plan collides with the inserted object.
Cross. ped.
Stand. ped.
Car
Model
All
Avoid.
All
Avoid.
All
Avoid.
ReactNAV
Unavoidable (%)
23.0
28.3
15.8
Alpamayo 1.5
+0.25
+0.32
+0.09
+0.13
+0.12
+0.15
Alpamayo 2
+0.06
+0.08
+0.03
+0.04
+0.12
+0.15
AutoVLA
+0.00
+0.00
+0.00
+0.00
+0.03
+0.00
Appendix
Table 10: Safety improvement ΔP (in percentage points) at τ=0.5s on all samples (All, as in Tab. 1 ) and on the avoidable samples only (Avoid.). Unavoidable: share of samples excluded. Gray: the paired 95% bootstrap CI includes zero.
Crossing pedestrian
Standing pedestrian
Car
Side cameras
Metric
τ=0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
CAM_L1/R1
ΔP
+0.00
−0.03
+0.53
+0.90
+0.00
+0.16
+0.34
+0.96
+0.03
+0.87
+2.14
+3.26
ΔCS
+0.03
−0.02
−0.02
−0.03
−0.01
−0.08
−0.16
−0.16
−0.28
−0.59
−0.70
−0.67
TIRϵ
34.9
19.4
17.3
CAM_L0/R0
ΔP
+0.00
+0.03
+0.12
+0.34
+0.00
+0.03
+0.16
+0.56
+0.03
+0.65
+1.83
+2.57
ΔCS
+0.01
+0.00
−0.04
−0.01
−0.01
−0.06
−0.10
−0.10
−0.23
−0.45
−0.59
−0.51
Appendix
Table 11: AutoVLA on ReactNAV with CAM_L1/R1 (released code; used in all other results) or CAM_L0/R0 as its front-left and front-right views, the latter for both the original and the edited scene. ΔP is in percentage points, ΔCS in km/h, and TIRϵ in percent at ϵ=0.05m , aggregated across all four look-ahead times. Gray: the paired 95% bootstrap CI includes zero.
Figure 10: Change in the no-collision rate, NC , of Alpamayo 1.5 and Alpamayo 2 on edited ReactPAI scenes at τ=2.0s when their CoT is replaced by a fixed one, relative to their own generated CoT. Whiskers indicate 95% bootstrap CIs.
Condition
Fixed CoT
Accelerate
Accelerate since the lane is clear ahead
Resume speed
Resume speed since the lane is clear ahead
Slow down
Slow down due to the ⟨ reason ⟩
Yield
Yield due to the ⟨ reason ⟩
Immediate stop
Come to a full stop immediately due to the ⟨ reason ⟩
Inserted object
⟨ reason ⟩
Appendix
Table 12: Fixed CoTs used in Fig. 10 . Each fixed CoT replaces the model’s generated CoT verbatim, and ⟨ reason ⟩ is filled in according to the inserted object (bottom).
Alpamayo 1.5
Alpamayo 2
Difference
ADEKO
0.89 [0.81, 0.97]
0.55 [0.50, 0.61]
+ 0.33 [0.27, 0.40]
ADERS
0.68 [0.61, 0.76]
0.69 [0.63, 0.75]
− 0.01 [ − 0.09, 0.09]
FDEKO
2.18 [1.98, 2.38]
1.36 [1.22, 1.51]
+ 0.82 [0.64, 1.00]
FDERS
1.96 [1.74, 2.19]
1.98 [1.80, 2.18]
− 0.03 [ − 0.28, 0.24]
Appendix
Table 13: Attention knockout on 300 original ReactPAI samples, measured over the first 4 s. KO: CoT hidden from the action expert, with the same initial noise as the natural rollout. RS: initial noise resampled, CoT unchanged. All values are in meters; means with 95% bootstrap CIs.
Signal (%)
Safe signal (%)
Object
G=8
16
32
G=8
16
32
Crossing pedestrian
29.0
43.0
53.0
1.0
3.0
3.0
Standing pedestrian
9.0
16.0
26.0
0.0
0.0
1.0
Car
0.0
0.0
0.0
0.0
0.0
0.0
All
12.7
19.7
26.3
0.3
1.0
1.3
Appendix
Table 14: Percentage of GRPO groups with a non-zero gradient (Signal) and with at least one safe trajectory above the group mean reward (Safe signal) for ReCogDrive’s IL checkpoint on 300 hazardous scenes, as a function of group size G .
Figure 11: Randomly selected ReactTrain samples for each inserted object and look-ahead time τ , shown in the edited front-camera image at the current frame t . Each column shows a different source sample, with its hazardous variant in the top row and its benign variant in the bottom row; both use the same asset.
Figure 12: Artificial target trajectory for a ReactTrain sample with a standing pedestrian (orange box) inserted at τ=1.0s . The ego vehicle (black box) decelerates along a cubic spline fitted to the logged ground-truth waypoints and stops (blue box) a margin dm before the pose at which it would first contact the pedestrian (dashed box).
Figure 13: Randomly selected samples of the unseen-objects test set, one per animal, for each look-ahead time τ , shown in the edited front-camera image at the current frame t .
Figure 14: Randomly selected samples of the real-scenes test set, shown in the unedited front-camera image at the current frame t . The target is the nearest vehicle ahead in the ego lane.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.
Reactive capability is a key property of data-driven behavior world model simulators for autonomous driving simulation systems. With this capability, simulated world agents can respond feasibly to autonomous vehicle (AV) behaviors that differ from the log. However, existing behavior simulation benchmarks do not directly measure reactive capability. They often let the simulator jointly control the AV and surrounding agents and evaluate realism through log similarity or open-loop prediction metrics. In this work, we introduce ReactSim-Bench for evaluating the reactive capability of behavior world model simulation in autonomous driving. We decouple the control of agents and the AV, using AV behaviors that differ from the log and require agents to respond as independent AV inputs. To obtain these AV behaviors, we construct a pipeline that uses an AV planner model to generate candidate behaviors and filters the data using rules and manual verification. Collision metrics, map-based metrics, and kinematic feasibility metrics are used to evaluate the safety and rule compliance of reactive responses. We construct 2,636 test scenarios with three categories and conduct a systematic evaluation of state-of-the-art models across multiple architectures, including Transformer-based, diffusion-based, and next-token-prediction-based models. We further analyze how replan frequency affects performance and provide insights for future studies.
Zhiyuan Zhang, Yanlun Peng, Jianing Zhang +7
Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University · Great Wall Motor · Institute of Trustworthy Embodied AI (TEAI), Fudan University +2
Vision-Language-Action (VLA) models are gaining attention in autonomous driving, inheriting commonsense priors from large language models (LLMs) and widening the navigation interface from a fixed command set to free-form natural-language instructions. However, this open-ended interface exposes a new vulnerability surface: an inattentive driver may issue instructions that are unsafe given the current scene, and a malicious user may issue dangerous ones on purpose. We present the first benchmark suite that evaluates navigation following and conflict awareness together, along with adversarial robustness. Existing VLAs fall short on both: SimLingo completes no language-instructed lane change in closed loop, and all models degrade sharply under conflicting signals. We propose SafeDriveVLA, the first driving VLA that (1) explicitly predicts a three-level navigation-compliance mode to decouple conflict reasoning from action generation, and (2) uses navigation-conditioned world model dreaming to inform this prediction. Unlike prior world models that roll out expert or planned actions, ours dreams of strictly following the navigation signal, showing when following is safe. SafeDriveVLA reaches 88.16 Driving Score (DS) and 68.64% Success Rate (SR) on Bench2Drive, lifts closed-loop language following by 49.3% and reduces collisions under unsafe instructions from 60 to 38 compared to the best SimLingo variant. Video demonstrations are available on our project page.
Shaoyuan Xie, Zihan Zhang, Jingxuan Wang +6
UC Irvine · UC San Diego · University of Cincinnati +1