Proposal-conditioned scene predictions can lack a common object-level reference across candidate trajectories. Aggregate planning scores also leave the objects and times associated with violations implicit. We present WTR-Drive (World-Task-Risk Driving), a Drive-JEPA-based planner that separates shared scene estimation from proposal-conditioned reasoning. A proposal-independent World Memory estimates object states, map structure, and future motion by re-reading the original visual features at each refinement round. Proposal feedback updates a residual Task branch without writing back to World Memory. This branch guides refinement through candidate-specific retrieval and gated visual feedback. A final Task readout supplies candidate-conditioned features for scoring. We derive structured risk targets from existing candidate-scoring rollouts without additional simulator calls. For at-fault collision and time-to-collision violations, the targets encode event existence, association with shared object slots, and the first violation time bin. On the NAVSIM navtest split, WTR-Drive achieves 94.66 PDMS on v1 and 88.24 EPDMS on v2 (expert-frame sequential evaluation). These scores are 0.96 and 0.44 points above the published Drive-JEPA values, respectively. Candidate-level analysis shows that higher best-candidate scores can coexist with lower selected-trajectory scores, revealing a gap between candidate quality and selection effectiveness.
Figures & tables
Fig. 1: World–Task–Risk formulation. (a) World: all candidates share one scene estimate; (b) Task: candidates p1 (teal, solid) and p2 (orange, dashed) read the shared Task Memory. (c) Risk: only p2 collides with A at t∗ ; the orange footprint marks the ego vehicle at first contact.
Fig. 2: Overview of WTR-Drive. (a) Multi-camera visual encoding and proposal initialization. (b) In four shared-parameter rounds, World Memory re-reads the original visual features, while proposal feedback updates Task Memory. Candidate-specific reads and gated visual feedback carry task information into refinement without writing back to World Memory, so it stays proposal-independent. (c) World readouts estimate scene structure and object motion. Separate scoring and risk heads predict candidate scores and event existence, object association, and the first violation time bin.
Symbol
Meaning (shape for tensors)
P,T,I
Candidates, forecast steps, rounds
i,p,t,o,r
Round, candidate, step, object, risk indices
N,Nv,d
Scene slots, visual tokens, channels
F0,Fctx(i)
Original / Task-conditioned vision, Nv×d
Q(i),Q
Proposal / readout features, P×T×d
Y(i)
Trajectory coordinates, P×T×3
TABLE I: Notation used in the Method.
Method
Backbone
Input
NC ↑
DAC ↑
EP ↑
TTC ↑
C ↑
PDMS ↑
DiffusionDrive [ 7 ]
ResNet-34
C+L
98.20
96.20
82.20
94.70
100.00
88.10
GoalFlow [ 8 ]
V2-99
C+L
98.40
98.30
85.00
94.60
100.00
90.30
DiffusionDriveV2 [ 28 ]
ResNet-34
C+L
98.30
97.90
87.50
94.80
99.90
91.20
iPad [ 4 ]
ResNet-34
C
98.60
98.30
88.00
94.90
100.00
91.70
DriveSuprim [ 9 ]
ViT-L
C
98.60
98.60
91.30
95.50
100.00
93.50
Drive-JEPA [ 3 ]
ViT-L
C
99.10
98.20
91.40
96.20
99.90
93.70
TABLE II: Planning performance on NAVSIM navtest
Configuration
Readout
PDMS ↑
EPDMS ↑
EPDMS H↑
Training configurations
Scene-query baseline
–
94.20
88.32
91.59
+ map refinement
–
94.43
88.45
91.84
w/o subscore loss
On
94.35
88.39
91.88
+ ranking loss
On
94.22
88.35
91.75
WTR-Drive
On
94.66
88.24
91.55
TABLE III: Ablation: training variants and Task readout interventions on NAVSIM navtest. PDMS: v1; EPDMS and EPDMS H : v2.
Fig. 3: Qualitative comparisons on NAVSIM v1, manually selected by category from 14 paired scenes: seven favorable cases and one failure (f). Panels report PDMS and one submetric: (a,b) TTC, (c,d) NC (collision avoidance), (e,f) DAC (road compliance), (g,h) EP (progress).
Fig. 4: Decoder refinement of 32 candidates for two scenes: straight driving (a)–(d) and a right turn (e)–(h). Red marks the GT-matched trajectory at each iteration; the annotation reports its average displacement error (ADE, m) over 4 s. The matched trajectory may change across iterations and need not be the score-selected trajectory.
Fig. 5: World–Task representations and collision diagnostics. (a) World object estimates. (b)–(d) Task attention for the selected candidate, a detected collision, and a missed collision; colors show read weights, and percentages give the weight on Q14, the same object slot in (a)–(d). (e, f) Predicted/ground-truth maps: road (green), walkway (blue), and centerline (orange). (g)–(i) Collision probabilities p for the candidates in (b)–(d), with threshold 0.50; “GT safe” and “GT risk” denote rollout collision labels. Solid black boxes mark event objects at the annotated rollout times; orange boxes mark predicted associations. BEV axes: lateral/forward distance (m).
Autonomous driving requires choosing a safe and efficient plan as surrounding traffic evolves. Generate-and-select planners propose multiple trajectories and score them for execution, and they have outperformed representative direct-prediction baselines on NAVSIM. Their scorer must compare plans that were never executed. Driving logs record the future of only the executed trajectory, so matching the logged future can leave predictions for the alternatives unconstrained; a simulator, in contrast, can label the outcome of every candidate. We introduce World4Scorer, which builds the scorer as a trajectory-conditioned JEPA-style predictor: it predicts a state for each candidate and reads the candidate's scores from that state. Simulator outcome labels supervise the states of all candidates, and the observed future of the executed trajectory anchors the predictor to real scene evolution. Because one predictor produces every candidate's state, the anchor can constrain shared parameters used to score unexecuted plans, while the future itself is needed only during training. Generated candidates mostly score well, so a scene-matched bank adds low-scoring plans to the outcome supervision; framewise choices can conflict, so inertial re-ranking keeps consecutive selections consistent. World4Scorer achieves state-of-the-art NAVSIM-v2 performance and a strong adapted-system result on closed-loop Bench2Drive. With the LeWM world model and planning budget fixed, outcome-based scoring also improves manipulation planning on the OGBench-Cube benchmark.
Jieyuan Pei, Meiyi Lu, Sining Ang +12
Institute for AI Industry Research (AIR), Tsinghua University · HiThink Research · The Hong Kong University of Science and Technology (Guangzhou) +8
Joint-Embedding Predictive Architectures (JEPAs) enable future prediction in latent space without pixel reconstruction. However, latent predictive consistency alone does not establish the relative driving quality of candidate plans. We present ActWorld-JEPA, a joint-embedding formulation for candidate-based driving. Where Plan-Conditioned JEPA (PC-JEPA) uses global and step-wise trajectory intentions to guide recurrent latent prediction, then combines the predicted future with the current scene through gated cross-attention; Relational Residual Reranking (R3) models agreement and discrepancy between plan and future representations under planning supervision, yielding bounded, gated residual corrections to the base planning scores. This formulation connects latent prediction to candidate selection using only observed context and proposed trajectories at inference. Benchmark-specific implementations report 94.1 PDMS on NAVSIM v1 navtest, with the final selector calibrated using aggregate NAVTEST feedback, and 91.6 and 42.7 EPDMS on NAVSIM v2 navtest and navhard, respectively. The nuScenes evaluation reports a mean L2 error of 1.20 m and a predicted collision rate of 0.76% over 1--6 s. This work provides an effective path from JEPA's predictive representations to driving decisions informed by anticipated consequences.
Ziying Song, Shengkai Zhang, Caiyan Jia +10
Nanyang Technological University · Beijing Jiaotong University · Dalian University of Technology +6
World models support autonomous driving by evaluating candidate trajectories through predicted future scene evolution. However, accurate future prediction does not ensure road-compliant execution; when all candidates violate road boundaries, re-ranking cannot select a feasible trajectory. We propose Physics-Guided Search Distillation (PGSD), a single-view planning framework that combines future representations and execution feedback for verified beyond-pool repair through three components: (1) Execution-Aware Selection checks ego-vehicle road compliance under simulated execution and selects the highest-scoring feasible trajectory; (2) a Clearance-Aware Search Teacher generates beyond-pool corrections and selects demonstrations with larger minimum road-boundary clearance during execution to buffer imitation errors; and (3) a Future-Conditioned Repair Student predicts one correction from rewarder features and execution feedback for single-rollout verification. The pretrained planner and rewarder remain frozen; no future scenes are generated online. On NAVSIM v1, single-view PGSD achieves 90.97 PDMS, exceeding all compared prior single-view planners and retaining 73.2% of the search teacher's PDMS gain over execution-aware selection. On NAVSIM v2 navhard, it reaches 38.96 EPDMS, improving on WorldDrive by 11.24 points (40.5%). The student replaces up to ten search rollouts with a single verification rollout per repair trigger. HUGSIM Hard examples provide qualitative evidence of zero-shot transfer.
Hongyu Zhang, Xiaoshuai Hao, Heye Huang +7
Yanshan University · Xiaomi Corporation · KAIST +2