Vision-Language-Action models (VLAs) and World-Action Models (WAMs) have emerged in autonomous driving with the promise of strong generalization to long-tail scenarios. Although these models achieve state-of-the-art results on autonomous driving benchmarks, most existing benchmarks are dominated by nominal scenarios and therefore cannot test this promise. To test one such long-tail capability that is critical for safety, we introduce ReactBench-AD, a benchmark that measures how driving models react to objects that appear in the ego vehicle's path. Built on real NAVSIM and PhysicalAI-Autonomous-Vehicles scenes, ReactBench-AD composites a crossing pedestrian, a standing pedestrian, or a stationary car consistently across all camera views and history frames. Each edited scene thus forms an exact, pixel-level counterfactual pair with its original, which allows us to isolate the causal effect of the inserted object on the predicted trajectory. Evaluating four VLAs and two WAMs, we find that they react weakly, often barely at all. Finally, we test how easily one of the evaluated VLAs, ReCogDrive, can be taught to brake hard for imminent hazards: a short fine-tuning stage on edited scenes substantially improves its reactions, with the learned behavior generalizing to unseen objects and fully real scenes, at minimal cost to nominal driving. These results call into question the out-of-the-box long-tail capabilities of these models, but suggest that they can be a strong foundation for acquiring them with little additional training, highlighting the value of real or synthetic data that demonstrates the long-tail behaviors nominal datasets rarely contain. Code and data will be made publicly available.
Figures & tables
Figure 1: We insert an object, here a standing pedestrian, into the ego vehicle’s path in a real driving scene, so that the edited scene SE and the original scene SO differ only in the pixels of the inserted object. Bottom: distance travelled by each model’s front bumper over its 4 s plan in the original ( yO ) and edited ( yE ) scene, with the pedestrian’s extent shown in yellow. All six evaluated VLAs and WAMs shorten their plans only slightly, if at all, and drive through the pedestrian, whereas the ReCogDrive IL checkpoint stops before it after a short fine-tuning on edited scenes (IL + RT, Sec. 5 ).
Figure 2: Overview of ReactBench-AD . Left: we pick a target on the ego vehicle’s ground-truth trajectory (dark line), namely the ego’s front-bumper position at a look-ahead time τ∈{0.5,1.0,1.5,2.0}s after the current frame t (circles). At each history timestep, a standing object is placed at the target, whereas a moving one, such as the crossing pedestrian shown here, is placed along a path that reaches the target at t+τ (here τ=0.5s ). Middle: we composite the object at these positions into every camera view and history frame of the model’s input (top: the original scene at the current frame t ). Right: in the edited scene, we score the model’s plans for the original ( yO , dashed) and edited ( yE , solid) input ( Eq. 1 ), here for ReCogDrive.
Crossing pedestrian
Standing pedestrian
Car
Model
Metric
τ=0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
ReactNAV
Alpamayo 1.5
ΔP
+0.25
+3.10
+7.16
+8.71
+0.09
+3.66
+11.93
+17.63
+0.12
+3.84
+15.92
+28.81
ΔCS
−1.51
−2.73
−2.68
−2.29
−1.63
−4.52
−5.44
−4.83
−1.23
−4.29
−7.05
−7.25
Alpamayo 2
ΔP
+0.06
+2.11
+6.75
+8.36
+0.03
+3.25
+9.42
+12.33
+0.12
+5.36
+19.39
+22.43
ΔCS
−1.11
−1.77
−1.42
−0.73
−1.49
−3.10
−3.18
−2.50
−1.19
−2.96
−4.17
−4.22
Table 1: Results on ReactNAV and ReactPAI by model, inserted object, and look-ahead time τ (s). ΔP is in percentage points (higher is better); ΔCS is in km/h (lower is better). Gray: the paired 95% bootstrap CI includes zero.
Figure 3: Trajectory invariance rate, TIRϵ : the percentage of samples whose original and edited trajectories differ by an ADE below ϵ , i.e., for which the model essentially ignores the inserted object (higher means less reactive). Results are aggregated across all four look-ahead times τ .
Figure 4: Effect of inserted objects on the CoTs of Alpamayo 1.5 and Alpamayo 2 on ReactPAI , aggregated across all four look-ahead times τ . Bars show the distribution of the CoT’s leading action: slow/stop (stop, yield, or slow down), keep distance (keep a distance from or follow a lead vehicle), or other . CoT changed : percentage of edited-scene CoTs that differ from the original-scene CoT of the same sample. Names the actor : percentage of edited-scene CoTs that mention a pedestrian or, for inserted cars, a vehicle (in parentheses: original-scene CoTs).
Figure 5: Effect of CoT swapping on NC for Alpamayo 1.5 (A1.5) and Alpamayo 2 (A2) on ReactPAI at τ=2.0s . The first and second letters give the source of the visual input and of the CoT: original (O) or edited (E). Whiskers indicate 95% bootstrap CIs.
ΔP ( v<20km/h )
Model
Stage
PDMS
Cross. ped.
Stand. ped.
Car
ReCogDrive
VLM SFT
84.1
+2.38
+4.99
+10.39
IL
86.5
+0.85
+1.36
+4.88
RL
90.8
+0.51
+0.62
+3.29
SimWAM
IL
90.3
+0.45
+2.10
+6.13
RL
91.9
+0.40
+1.25
+4.14
Table 2: Safety improvement ΔP (in percentage points) on ReactNAV at τ=2.0s for each training-stage checkpoint of ReCogDrive and SimWAM, over samples with an initial ego-vehicle speed below 20 km/h ( n=1762 ). Gray: the paired 95% bootstrap CI of the difference to the previous training stage includes zero. PDMS on navtest as reported in [ 22 , 53 ] .
ReactNAV
navtest
ΔP↑
ΔCS↓
NC↑
DAC↑
TTC ↑
EP ↑
C ↑
PDMS ↑
IL
+0.5
−0.3
98.1
94.4
94.2
80.6
100.0
86.2
IL + RT
+41.4
−9.9
98.7
94.5
94.9
79.3
99.9
86.0
Table 3: Results of ReCogDrive’s IL checkpoint (IL) and of the same checkpoint fine-tuned on ReactTrain (IL + RT) on ReactNAV , aggregated across all look-ahead times τ , and on navtest (PDMS and its sub-metrics on LQR-executed trajectories, from our own evaluation). Units are as in Tab. 1 .
Figure 6: Plans of the IL and IL + RT checkpoints for the 291 unseen-objects samples, each at one random τ , in the original ( yO ) and edited ( yE ) scenes; each line traces the front bumper over 4 s. Although ReactTrain contains no animals, IL + RT shortens its plans in the edited scenes, while its original-scene plans stay close to those of the IL checkpoint.
Unseen objects
Real scenes
ΔP↑
ΔCS↓
P↑
CS↓
IL
+0.1
−0.2
6.8
21.4
IL + RT
+28.1
−8.7
54.2
14.3
Table 4: Generalization of the IL and IL + RT checkpoints to unseen objects and real scenes, aggregated across all four look-ahead times τ . Real scenes have no original trajectory, so we report absolute values: P in percent and the mean contact speed, CS , in km/h.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Randomly selected ReactNAV (top) and ReactPAI (bottom) samples for each inserted object and look-ahead time τ , shown in the edited front-camera image at the current frame t . Each row shows a single scene, whose four look-ahead variants share the same asset.
τ (s)
Object
0.5
1.0
1.5
2.0
All
ReactNAV
Crossing ped.
99.9
99.9
99.5
99.3
99.7
Standing ped.
100.0
100.0
99.9
99.8
99.9
Car
100.0
100.0
99.8
99.6
99.8
ReactPAI
Appendix
Table 5: Share of edited samples (%) in which D-FINE-X detects the inserted object at its location, by object and look-ahead time τ .
Model
τ=0.5
1.0
1.5
2.0
Mean
ReCogDrive
0.0
0.0
1.0
22.7
5.9
DriveWAM
0.0
0.0
0.6
28.9
7.4
SimWAM
0.0
0.0
0.0
23.4
5.8
Appendix
Table 6: P (in percent) on the 308 real scenes with a stationary vehicle ahead, by look-ahead time τ (s).
km/h
5–10
10–15
15–20
20–25
25–30
30–35
35–40
Total
ReactNAV
325
634
803
680
484
271
31
3,228
ReactPAI
180
188
199
137
127
110
104
1,045
Appendix
Table 7: Number of samples in each ego-vehicle speed interval at the current frame for ReactNAV and ReactPAI .
Model
Action head
Inputs
Outputs
Cameras
Frames per camera
Visual history (s)
Ego history (s)
Vel./acc.
Reasoning
Trajectory horizon (s)
Alpamayo 1.5
Transformer Decoder
4
4
0.3
1.5
No
Yes
6.4
Alpamayo 2
Transformer Decoder
6
4
0.3
1.5
No
Yes
6.4
AutoVLA
–
3
4
1.5
–
Yes
Optional
5.0
ReCogDrive
Diffusion Transformer
1
1
0
1.5
Yes
No
4.0
DriveWAM
–
1
1
0
1.5
Yes
No
4.0
Appendix
Table 8: Comparison of the action heads, input configurations, and outputs of the evaluated models. Visual and ego history denote the temporal spans of camera observations and past ego states, respectively; a visual history of 0 s indicates a single current frame.
Crossing pedestrian
Standing pedestrian
Car
Model
Metric
τ=0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
ReactNAV
Alpamayo 1.5
P
O
0.00
0.06
2.11
5.89
0.00
0.03
1.55
5.11
0.00
0.00
1.15
3.69
E
0.25
3.16
9.26
14.59
0.09
3.69
13.48
22.74
0.12
3.84
17.07
32.50
Δ
+0.25
+3.10
+7.16
+8.71
+0.09
+3.66
+11.93
+17.63
+0.12
+3.84
+15.92
+28.81
NCins
O
0.00
0.15
2.94
9.23
0.00
0.19
3.22
11.06
0.00
0.03
1.86
6.32
Appendix
Table 9: Detailed results on ReactNAV and ReactPAI by model, inserted object, and look-ahead time τ (s). O and E denote the trajectories predicted from the original and edited scene, respectively, both evaluated in the edited scene, and Δ=E−O . P and NCins are in percent and their deltas in percentage points (higher is better). CS is the mean contact speed in km/h over the samples for which both trajectories result in an at-fault collision (lower is better). Deltas are computed before rounding.
Figure 8: Safety improvement as a function of ego-vehicle speed at the current frame aggregated across all four look-ahead times τ . Shaded areas indicate 95% bootstrap CIs.
Figure 9: Example predictions at a look-ahead time of τ=2.0s , one for each model and inserted object: Alpamayo 1.5 and Alpamayo 2 on ReactPAI , and the other four models on ReactNAV . Each example shows the edited front-camera image at the current frame, cropped around the inserted object, next to a bird’s-eye view of the edited scene. The arrow marks the crossing pedestrian’s motion over the 4 s horizon. Dashed and solid lines show the model’s plans for the original ( yO ) and edited ( yE ) scenes, respectively, as front-bumper paths over the first 4 s. Examples are drawn at random from the samples whose original plan collides with the inserted object.
Cross. ped.
Stand. ped.
Car
Model
All
Avoid.
All
Avoid.
All
Avoid.
ReactNAV
Unavoidable (%)
23.0
28.3
15.8
Alpamayo 1.5
+0.25
+0.32
+0.09
+0.13
+0.12
+0.15
Alpamayo 2
+0.06
+0.08
+0.03
+0.04
+0.12
+0.15
AutoVLA
+0.00
+0.00
+0.00
+0.00
+0.03
+0.00
Appendix
Table 10: Safety improvement ΔP (in percentage points) at τ=0.5s on all samples (All, as in Tab. 1 ) and on the avoidable samples only (Avoid.). Unavoidable: share of samples excluded. Gray: the paired 95% bootstrap CI includes zero.
Crossing pedestrian
Standing pedestrian
Car
Side cameras
Metric
τ=0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
CAM_L1/R1
ΔP
+0.00
−0.03
+0.53
+0.90
+0.00
+0.16
+0.34
+0.96
+0.03
+0.87
+2.14
+3.26
ΔCS
+0.03
−0.02
−0.02
−0.03
−0.01
−0.08
−0.16
−0.16
−0.28
−0.59
−0.70
−0.67
TIRϵ
34.9
19.4
17.3
CAM_L0/R0
ΔP
+0.00
+0.03
+0.12
+0.34
+0.00
+0.03
+0.16
+0.56
+0.03
+0.65
+1.83
+2.57
ΔCS
+0.01
+0.00
−0.04
−0.01
−0.01
−0.06
−0.10
−0.10
−0.23
−0.45
−0.59
−0.51
Appendix
Table 11: AutoVLA on ReactNAV with CAM_L1/R1 (released code; used in all other results) or CAM_L0/R0 as its front-left and front-right views, the latter for both the original and the edited scene. ΔP is in percentage points, ΔCS in km/h, and TIRϵ in percent at ϵ=0.05m , aggregated across all four look-ahead times. Gray: the paired 95% bootstrap CI includes zero.
Figure 10: Change in the no-collision rate, NC , of Alpamayo 1.5 and Alpamayo 2 on edited ReactPAI scenes at τ=2.0s when their CoT is replaced by a fixed one, relative to their own generated CoT. Whiskers indicate 95% bootstrap CIs.
Condition
Fixed CoT
Accelerate
Accelerate since the lane is clear ahead
Resume speed
Resume speed since the lane is clear ahead
Slow down
Slow down due to the ⟨ reason ⟩
Yield
Yield due to the ⟨ reason ⟩
Immediate stop
Come to a full stop immediately due to the ⟨ reason ⟩
Inserted object
⟨ reason ⟩
Appendix
Table 12: Fixed CoTs used in Fig. 10 . Each fixed CoT replaces the model’s generated CoT verbatim, and ⟨ reason ⟩ is filled in according to the inserted object (bottom).
Alpamayo 1.5
Alpamayo 2
Difference
ADEKO
0.89 [0.81, 0.97]
0.55 [0.50, 0.61]
+ 0.33 [0.27, 0.40]
ADERS
0.68 [0.61, 0.76]
0.69 [0.63, 0.75]
− 0.01 [ − 0.09, 0.09]
FDEKO
2.18 [1.98, 2.38]
1.36 [1.22, 1.51]
+ 0.82 [0.64, 1.00]
FDERS
1.96 [1.74, 2.19]
1.98 [1.80, 2.18]
− 0.03 [ − 0.28, 0.24]
Appendix
Table 13: Attention knockout on 300 original ReactPAI samples, measured over the first 4 s. KO: CoT hidden from the action expert, with the same initial noise as the natural rollout. RS: initial noise resampled, CoT unchanged. All values are in meters; means with 95% bootstrap CIs.
Signal (%)
Safe signal (%)
Object
G=8
16
32
G=8
16
32
Crossing pedestrian
29.0
43.0
53.0
1.0
3.0
3.0
Standing pedestrian
9.0
16.0
26.0
0.0
0.0
1.0
Car
0.0
0.0
0.0
0.0
0.0
0.0
All
12.7
19.7
26.3
0.3
1.0
1.3
Appendix
Table 14: Percentage of GRPO groups with a non-zero gradient (Signal) and with at least one safe trajectory above the group mean reward (Safe signal) for ReCogDrive’s IL checkpoint on 300 hazardous scenes, as a function of group size G .
Figure 11: Randomly selected ReactTrain samples for each inserted object and look-ahead time τ , shown in the edited front-camera image at the current frame t . Each column shows a different source sample, with its hazardous variant in the top row and its benign variant in the bottom row; both use the same asset.
Figure 12: Artificial target trajectory for a ReactTrain sample with a standing pedestrian (orange box) inserted at τ=1.0s . The ego vehicle (black box) decelerates along a cubic spline fitted to the logged ground-truth waypoints and stops (blue box) a margin dm before the pose at which it would first contact the pedestrian (dashed box).
Figure 13: Randomly selected samples of the unseen-objects test set, one per animal, for each look-ahead time τ , shown in the edited front-camera image at the current frame t .
Figure 14: Randomly selected samples of the real-scenes test set, shown in the unedited front-camera image at the current frame t . The target is the nearest vehicle ahead in the ego lane.
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University · Great Wall Motor · Institute of Trustworthy Embodied AI (TEAI), Fudan University +2