SAFER: Navigating the Exploration-Exploitation Dilemma in World-Model-Based Planning for Autonomous Driving
Authors: Yansong Qu, Zilin Huang, Zihao Sheng, Sikai Chen, Samuel Labi
Organizations: Lyles School of Civil and Construction Engineering, Purdue University, West Lafayette 47907, USA · Department of Civil and Environmental Engineering, University of Wisconsin–Madison, Madison 53706, USA
An autonomous vehicle must foresee the consequences of its actions, especially in the rare, time-critical situations where crashes happen. A world model offers this foresight, but only within the coverage of its training data, which poses an exploration-exploitation dilemma. Within this coverage a planner can hardly improve on the data, and beyond it the model's errors mislead it. Logs of competent driving rarely contain the failures that would widen the coverage, and agents that learn from interaction gain them only by failing themselves, without being shown what to do instead. We propose SAFER, a framework that learns offline from failures that others make, judged by an expert, and keeps learning online from its own driving. Offline, partially trained agents and scripted, recoverable perturbations Stage failures in simulation, and an expert Annotates every visited state with its action and value, so that much of what the planner explores is covered before it drives. In planning, a chunked latent world model Foresees the consequences fast enough for real-time use, and a risk-aware sampling-based planner Evaluates candidate maneuvers by their return minus a bounded collision penalty. Online, a loop continuously Refines the model wherever this driving leaves the coverage of the offline data. In closed-loop evaluation against the compared methods, SAFER reaches the highest route completion and reward per step on the CARLA Leaderboard routes, 0.95 and 0.77 from a privileged bird's-eye view, and the highest episode reward in MetaDrive, 53.4 in regular and 49.0 in safety-critical traffic. Project page: https://ys-qu.github.io/safer-website.
Figures & tables
Figure 1: Learning paradigms for autonomous driving, compared by where the experience beyond competent driving comes from. The glyph under each paradigm shows the coverage of its data (gray) and how the paradigm treats it. Exploiting (blue) spreads the candidates over the coverage and takes the best one up to its edge, but no further. Exploring (orange) grows the coverage at the price of failures (red crosses), the paradigm’s own or, for SAFER, failures that others make, labeled by the expert (teal). A step out of the coverage that does not grow it is misjudged (?). (a) Imitation learning copies expert demonstrations and never observes a failure; reinforcement learning discovers consequences by failing in the environment. (b) A world model learned online ( Hafner et al., 2020 ; Li et al., 2024 ) still explores by failing. A world model learned from logged expert driving ( Wang et al., 2024b ; Zheng et al., 2025 ) covers competent driving only, and a planner that searches beyond this coverage favors the actions that the model misjudges. (c) SAFER broadens the coverage offline with staged failures that an expert annotates, plans with a fast chunked world model that prices collision risk, and extends the coverage online from the planner’s own driving. The BEV frames, candidates and risk curves in (c) are from a recorded collision of the corpus (Fig. 5 a).
Figure 2: An everyday analogy for SAFER (top) and the corresponding stages of the method (bottom, detailed in Fig. 3 ); the numbered badges link each stage to its scene. (1) Parents praise what works and point to negative examples, so that the child need not fail first. (a) Likewise, weaker drivers and scripted perturbations stage failures in simulation, and the expert labels every visited state with its action and value. (2) The child learns what each choice leads to. Hard work (action) brings good grades (reward), a far goal is worth pursuing (value), and staying out all night is risky (risk). (b) Likewise, the world model foresees these quantities for each candidate action chunk, and (c) the planner scores the candidates by their return minus a price on risk. (3) Away from home, school and work keep reshaping the beliefs while the core ones stay. (d) Likewise, the planner’s own driving refines the dynamics and the reward and risk heads, while the prior and the value, whose targets only the expert provides, stay fixed.
Figure 3: Technical overview of SAFER, illustrated with its CARLA instance. The numbered badges link each part to its scene of the analogy in Fig. 2 . (a) Stage and Annotate: the expert, weaker Roach checkpoints and scripted hold and late-brake events drive until the first violation, and the expert labels every visited state with its action and value (Section 3.2 ). (b) Foresee: from the context Mt of the past BEVs, the dynamics imagines the 16 latents of a candidate chunk in one pass, and heads read them as the behavior-cloned prior, the reward and its five factors, survival, value and collision risk (Section 3.3 ). (c) Evaluate: every 0.1 s, the planner scores 135 candidate chunks on their imagined futures (Eq. ( 28 )), refits its sampling distribution to the 16 best in each of two iterations and executes the first action (Section 3.5 ). (d) Refine: the planner’s own driving fine-tunes the dynamics, the reward and the risk head online, while the encoder, the prior and the value stay frozen and the planner scores with slowly moving EMA copies (Section 3.6 ). The crash frame in (a), the context frames in (b) and the candidates and risk curves in (c) are from a recorded collision of the corpus (Fig. 5 a); the other two BEV frames are illustrative.
Figure 4: Experimental environments. Top: CARLA. (a) The six towns of the evaluation, aerial view with the lane network inset; the corpus is collected in Town01, Town03, Town04 and Town06, while Town02 and Town05 contribute no training data. (b) Scenes of evaluation episodes in a front camera added for illustration; the Town04 scene lies on a held-out test route. (c) The agent’s input in the Town01 scene of (b): the bird’s-eye view, 38 m wide (route, lanes, vehicles and pedestrians over four past frames, fading, and traffic lights at their stop lines; the ego is the white box), and the ego state, i.e. the last applied control and the planar velocity. Bottom: MetaDrive with CAT. (d) Road layouts of six Waymo test scenarios. (e) Test scenario 401 at 3.2 s in its logged traffic (left) and against CAT’s opponent (right): top-down views, heading up, with the ego’s logged route (blue), the opponent’s logged path (white, dashed) and the path CAT generates for it (magenta), which crosses the ego’s route (star); below, the front camera at the same moment. (f) The agent’s 101 -dimensional input at the right-hand moment of (e): 72 distances, drawn as rays, to vehicles along 30 directions (lidar, cyan), to road edges along 30 (yellow) and to lane lines along 12 (pink), and 29 state entries: the ego’s state, navigation features, the relative states of the three nearest vehicles and the lateral offset from the logged trajectory.
Set
Driver
Steps
Episodes
Collisions
Red lights
Perturbed
Dexp
expert
10.00 M
1,901
1,669
133
0%
Dint
13 intermediate checkpoints + noise
10.05 M
52,023
28,629
10,006
21.6%
Dtgt
expert + noise + events (two runs)
2.00 M
338
268
46
24.8%
total
22.06 M
54,262
30,566
10,185
added by one turn of the data flywheel
Dint+
13 intermediate checkpoints + noise
4.00 M
20,931
11,439
4,000
21.8%
Table 1: Offline corpus in CARLA. Steps are transitions at 10 Hz. Episodes end on the first safety violation; the intermediate policies therefore produce many short episodes. “Perturbed” counts steps whose executed action differs from the behavior policy’s proposal. The lower block lists the sets that one turn of the data flywheel adds (Section 4.2 ); in Dtgt+ , scripted pedestrians step into the ego’s path.
Figure 5: Two planning decisions of the offline planner on recorded states from Dint . From left to right: the BEV at the decision time (route light gray, road dark gray, lane markings magenta, vehicles blue, pedestrians cyan, lighter shades are earlier frames; ego white) with the 270 scored candidates drawn as paths and colored by their score J , the executed plan (green), the behavior-cloned prior (dotted) and, in (a), the recorded driver’s actions up to the impact (dashed red, star), and a red cross where a candidate’s predicted collision probability first exceeds 0.5 ; paths are sketched from the action chunks with a kinematic bicycle model and the predicted speed, not decoded positions; predicted return against risk penalty ρˉ (log scale) for all candidates, with the plan (star), the prior (triangle) and, in (a), the recorded actions (cross); the acceleration command of the 60 best candidates, the plan, the prior and the recorded actions over the imagined horizon (negative values brake); and the predicted probability that a collision has occurred by each imagined step.
Algorithm
Input
Driving
Route
Infraction
score ↑
compl. ↑
penalty ↑
CILRS ( Codevilla et al., 2019 )
RGB
0.05
0.14
0.55
LBC ( Chen et al., 2020 )
RGB
0.09
0.18
0.73
TransFuser ( Prakash et al., 2021 )
RGB + LiDAR
0.17
0.52
0.42
NEAT ( Chitta et al., 2021 )
RGB
0.22
0.42
0.65
MMFN ( Zhang et al., 2022 )
RGB + LiDAR + HD map
0.23
0.47
0.63
Table 3: Comparison with the baselines on the CARLA Leaderboard. The end-to-end methods (top) are given with their published results. TD3+BC, IQL, TD-MPC2 and DreamerV3 (middle) are trained by us on SAFER’s corpus, with the frozen encoder of the expert and without its labels, and evaluated as SAFER.
Per 100 km ↓
Reward
Speed
Completed
Blocked
Collisions
Red
Stop
Algorithm
per step ↑
(m/s) ↑
(%) ↑
(%) ↓
Pedestrians
Vehicles
lights
signs
TD-MPC2 ( Hansen et al., 2024 )
−0.51±0.18
2.61±0.22
43.0±8.0
52.6±6.0
5.6±1.4
27.2±0.2
3.8±2.0
0.7±1.2
TD3+BC ( Fujimoto and Gu, 2021 )
0.53±0.02
3.08±0.09
68.0±3.0
30.3±2.6
8.4±2.6
27.9±3.3
4.7±0.6
2.6±0.8
IQL ( Kostrikov et al., 2022 )
0.47±0.09
3.20±0.05
75.4±2.7
21.5±2.7
4.2±2.5
19.3±4.3
3.9±2.1
3.9±0.9
DreamerV3 ( Hafner et al., 2025a )
0.71±0.01
3.26±0.05
88.2±2.6
11.8±2.6
1.1±0.9
17.1±6.0
2.1±1.6
0.0±0.0
Table 4: Further metrics of the methods of Table 3 whose input is exactly that of SAFER, under the same protocol: mean ± standard deviation over three seeds. Reward of Eq. ( 1 ) per step, average speed (route distance completed over the duration of the episode), share of the routes completed, share of the episodes in which the ego is blocked, and collisions and traffic-rule infractions per 100 km driven.
Figure 6: How early the risk head foresees recorded collisions ( 150 events per collision partner, 150 collision-free control segments; world model fed the recorded history and the driver’s executed actions). (a) Median and interquartile range of P^1.6 , the predicted probability of a collision within the 1.6 s horizon, against the time to impact, for the 450 collision segments and the controls; the dotted line marks the moment the impact enters the horizon. (b) Share of collisions with P^imp>0.5 , i.e. predicted by the step at which they actually happen, per collision partner against the time to impact, and the false-alarm rate of the controls ( P^1.6>0.5 at any of their 4,500 decisions). (c) AUC of P^1.6 between collision and control segments at each time to impact.
Figure 7: The imagined reachable set and its risk area in two closed-loop decisions of the full system. Left: a front camera added for visualization, which the agent does not observe. Middle: (a) the BEV that the agent observes; (b) a top-down view drawn from the simulator state, with CAT’s opponent (magenta) and the trajectory that CAT generates for it (dashed), which the planner does not know. In both views the 270 candidate chunks that the planner scored at that step are drawn as in B , as the paths that they trace under a kinematic bicycle model at the speeds that the world model predicts, colored by the predicted reward of each imagined step; together they sample the reachable set. Red: the risk area, i.e. the risk head’s probability of a priced safety event (CARLA: collision; MetaDrive: vehicle contact plus leaving the road, object contact staying at zero), averaged at each place over the imagined states of all candidates that pass there and drawn with an opacity that grows with the probability; it is the same field in both views. Orange: executed plan; dotted: behavior-cloned prior. Right: the same probability along the horizon for every candidate (gray), the plan (orange) and the prior (dotted), with their risk terms Wρˉ of Eq. ( 26 ).
All windows
Changing windows
Factor
Imagined
Copied
Imagined
Copied
Traffic light
93.5
90.8
76.7
2.7
Pedestrian near the path
98.0
98.3
56.7
51.3
Vehicle ahead
94.8
95.0
49.3
30.0
Table 6: Reading the imagination of the stage-1 world model with its scene-factor head: share of recorded windows (%) in which the state read 1.6 s ahead in the imagined latents matches the rule label of the recorded frame, against copying the state the head reads at the decision time. All windows: 400 drawn at random; changing windows: 150 per factor in which the state 1.6 s ahead differs from the present one (the light changes color, a pedestrian comes near the path, a vehicle ahead appears or leaves).
Figure 8: Explaining the imagination with the scene-factor head, on recorded windows of the corpus; the model receives the recorded history and the actions the driver executed. Each row: the sentence composed from the factors read in the imagined latents and the recorded outcome; the BEV at the decision time t and the recorded frames 0.5 , 1.0 and 1.6 s later (ego white); and the imagined and recorded state over the 1.6 s horizon in steps of 0.1 s: (a), (b) the traffic light, (c) a pedestrian near the path (magenta).
Controller
Rollouts
Time per decision (ms)
Frequency (Hz)
BC prior
0
5.8±0.4
173
MPPI
270
23.1±3.2
43
+ risk term
270
22.3±2.3
45
+ online, 106 steps
270
22.0±2.3
45
Table 7: Inference time per driving decision (observation encoding and controller) on one Quadro RTX 6000 with a batch of one: mean ± standard deviation over 2,000 decisions replayed from recorded closed-loop episodes, and the decision frequency it allows. All controllers use the same model of 15.5 M parameters. Rollouts: imagined 16 -step action chunks scored per decision. The agent acts at 10 Hz.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 1: Planning steps of the full system in normal driving ( B ). Left: a front camera added for visualization (not observed by the agent); right: the BEV that the planner receives, cropped around the candidates. Paths: the 270 candidate chunks scored at that step, sketched with a kinematic bicycle model at the predicted speeds and colored by the predicted reward of each imagined step; orange: executed plan; dotted: behavior-cloned prior; crosses: the first imagined step at which the predicted collision probability reaches 0.5 . (a) Straight road with oncoming traffic; (b) right turn at a T-junction; (c) junction where the route turns right while a few chunks, the prior among them, continue straight towards a red light.
Figure 2: Planning steps of the full system in safety-critical scenes, drawn as in Fig. 1 . (d) A car stopped in the lane ahead; (e) a police car ahead; (f) a slower car ahead in a queue. In each scene the chunks that the risk head predicts to collide (in (d) and (f), the behavior-cloned prior among them) carry the crosses, and the executed plan stays clear of them.
Figure 3: Planning steps of the full system on CAT’s MetaDrive benchmark in regular traffic ( B ). Left: a front camera placed as in Fig. 1 and rendered for visualization (the agent observes a 101 -dimensional vector); right: a top-down view drawn from the simulator’s state, cropped around the candidates. Paths: the 270 candidate chunks, colored by the predicted reward of each imagined step (CAT’s reward: progress in meters per step); orange: executed plan; dotted: behavior-cloned prior; red crosses: the first imagined step at which the predicted probability of a vehicle contact reaches 0.5 ; blue crosses: the same for leaving the road. (a) Junction with an oncoming car; (b) right turn at a junction at 8.3 m/s; (c) three-lane road on which 8% of the candidates are predicted to leave the road on the left.
Figure 4: Planning steps of the full system on CAT’s MetaDrive benchmark in safety-critical scenes, drawn as in Fig. 3 ; magenta: CAT’s opponent in the top-down view, and its generated trajectory. (d) Right turn across crossing traffic; (e) a slow car in the lane to the right, in regular traffic; (f) the opponent turning across a left turn. In each scene the behavior-cloned prior is among the candidates predicted to make contact, and the executed plan is not.
hold 30 – 80 steps, late brake 5 – 10 steps, recovery 30 – 60 steps
World model
latent width / decoder
256 / 4 layers, 4 heads, MLP ratio 4
context C , horizon H
16 , 16 steps ( 1.6 s at 10 Hz)
reward, value bins
B=101 bins, uniform in symlog space on [−3,3] and [−10,10]
Appendix
Table 1: Hyper-parameters of the evaluated CARLA system.
Data collection
scripted events
hazard brake instead of hold and late brake: 5 – 10 steps at a brake level drawn from U[0.3,1] , triggered when a road user ahead on the route is within 20 m and at most 3 s from collision, the ego drives at 2 – 10 m/s and the expert’s acceleration command is above −0.3 ; recovery 30 – 60 steps
value labels
policy evaluation of the noise-free expert: its two Q-networks, initialized from its critic, fitted on all transitions of the corpus to one-step targets with the expert’s action at the next state; 105 steps, batch 1,024 , lr 3×10−4 , target-network rate 0.005 , discount 0.99
World model
encoder
two-layer MLP, 101→256→256 , initialized from the first two layers of the expert’s actor
reward factors
progress along the route (linear, scale 2 m) and the indicators of arrival, leaving the road, vehicle contact and object contact (sigmoid)
Offline training
Appendix
Table 2: Settings of the evaluated MetaDrive system that differ from those of the CARLA system; all other settings are those of Table 1 .
Autonomous driving requires safe planning, but most learning-based planners lack explicit self-correction ability: once an unsafe action is proposed, there is no mechanism to correct it. Thus, we propose CorrectionPlanner, an autoregressive planner with self-correction that models planning as motion-token generation within a propose, evaluate, and correct loop. At each planning step, the policy proposes an action, namely a motion token, and a learned collision critic predicts whether it will induce a collision within a short horizon. If the critic predicts a collision, we retain the sequence of historical unsafe motion tokens as a self-correction trace, generate the next motion token conditioned on it, and repeat this process until a safe motion token is proposed or the safety criterion is met. This self-correction trace, consisting of all unsafe motion tokens, represents the planner's correction process in motion-token space, analogous to a reasoning trace in language models. We train the planner with imitation learning followed by model-based reinforcement learning using rollouts from a pretrained world model that realistically models agents' reactive behaviors. Closed-loop evaluations show that CorrectionPlanner reduces collision rate by over 20% on Waymax and achieves state-of-the-art planning scores on nuPlan.
Yihong Guo, Dongqiangzi Ye, Sijia Chen +2
Department of Computer Science, Johns Hopkins University, Baltimore, USA. · XPENG Motors.
End-to-End (E2E) autonomous driving models have shown growing capability in recent years, with performance improving on increasingly challenging benchmarks. However, modern generative E2E planners still suffer from a substantial number of catastrophic failures in safety-critical scenarios. We find that many such failures arise from violations of physical constraints and safety requirements, leading to unsafe behavior. Motivated by this finding, in this paper, we focus on improving safety outcomes in generative end-to-end driving with a targeted reduction of catastrophic planning failures, instead of enhancing average planning quality. Towards this end, we propose DriveSafer, a failure-aware safety framework for end-to-end planners. DriveSafer explicitly steers generative planners towards safe behaviors leveraging both training-time safety constraints and inference-time safety guidance. Compared to the state-of-the-art DiffusionDrive model, on the NAVSIM benchmark, DriveSafer reduces the number of catastrophic failures (PDMS=0) by 48%, with over 65% reduction in drivable-area compliance failures.
Autonomous driving requires choosing a safe and efficient plan as surrounding traffic evolves. Generate-and-select planners propose multiple trajectories and score them for execution, and they have outperformed representative direct-prediction baselines on NAVSIM. Their scorer must compare plans that were never executed. Driving logs record the future of only the executed trajectory, so matching the logged future can leave predictions for the alternatives unconstrained; a simulator, in contrast, can label the outcome of every candidate. We introduce World4Scorer, which builds the scorer as a trajectory-conditioned JEPA-style predictor: it predicts a state for each candidate and reads the candidate's scores from that state. Simulator outcome labels supervise the states of all candidates, and the observed future of the executed trajectory anchors the predictor to real scene evolution. Because one predictor produces every candidate's state, the anchor can constrain shared parameters used to score unexecuted plans, while the future itself is needed only during training. Generated candidates mostly score well, so a scene-matched bank adds low-scoring plans to the outcome supervision; framewise choices can conflict, so inertial re-ranking keeps consecutive selections consistent. World4Scorer achieves state-of-the-art NAVSIM-v2 performance and a strong adapted-system result on closed-loop Bench2Drive. With the LeWM world model and planning budget fixed, outcome-based scoring also improves manipulation planning on the OGBench-Cube benchmark.
Jieyuan Pei, Meiyi Lu, Sining Ang +12
Institute for AI Industry Research (AIR), Tsinghua University · HiThink Research · The Hong Kong University of Science and Technology (Guangzhou) +8