ActWorld-JEPA: Action-Conditioned JEPA World Models Meet Latent Planning for End-to-End Autonomous Driving
Authors: Ziying Song, Shengkai Zhang, Caiyan Jia, Lei Yang, Lin Liu, Yuchen Liu, Ziyang Liu, Zirui Li, Gang Wang, Peiliang Wu, Li Wang, Yadan Luo, Chen Lv
Organizations: Nanyang Technological University · Beijing Jiaotong University · Dalian University of Technology · North University of China · Tsinghua University · NingboTech University · Yan Shan University · Beijing Institute of Technology · The University of Queensland
Joint-Embedding Predictive Architectures (JEPAs) enable future prediction in latent space without pixel reconstruction. However, latent predictive consistency alone does not establish the relative driving quality of candidate plans. We present ActWorld-JEPA, a joint-embedding formulation for candidate-based driving. Where Plan-Conditioned JEPA (PC-JEPA) uses global and step-wise trajectory intentions to guide recurrent latent prediction, then combines the predicted future with the current scene through gated cross-attention; Relational Residual Reranking (R3) models agreement and discrepancy between plan and future representations under planning supervision, yielding bounded, gated residual corrections to the base planning scores. This formulation connects latent prediction to candidate selection using only observed context and proposed trajectories at inference. Benchmark-specific implementations report 94.1 PDMS on NAVSIM v1 navtest, with the final selector calibrated using aggregate NAVTEST feedback, and 91.6 and 42.7 EPDMS on NAVSIM v2 navtest and navhard, respectively. The nuScenes evaluation reports a mean L2 error of 1.20 m and a predicted collision rate of 0.76% over 1--6 s. This work provides an effective path from JEPA's predictive representations to driving decisions informed by anticipated consequences.
Figures & tables
Figure 1: From predictive representations to candidate-conditioned planning. (a) JEPA-style representation learning predicts target features from observed context, as in V-JEPA [ 3 ] . (b) Drive-JEPA [ 51 ] combines video-JEPA representations with multimodal trajectory distillation and momentum-aware selection. (c) ActWorld-JEPA conditions latent prediction on candidate plans and relates the predicted futures to trajectory ranking through PC-JEPA and R3 . Both observed scene context and candidate intention contribute to the decision. (d) Planning performance. PDMS on NAVSIM v1 navtest [ 10 ] , comparing ActWorld-JEPA with Epona [ 65 ] , DiffusionDrive [ 30 ] , DIVER [ 45 ] , GraphWorld [ 44 ] , GoalFlow [ 57 ] , DriveFuture [ 16 ] , and Drive-JEPA [ 51 ] .
Figure 2: The overall architecture of ActWorld-JEPA. The driving encoder extracts the current representation, and the planner produces candidate trajectories. PC-JEPA encodes global and step-wise plan conditions, initializes plan-modulated latents, and rolls out candidate-specific futures. Cross-attention and gated residual interaction fuse predicted future evidence with the current scene. R3 combines plan and fused-future features to rank the original candidates. The rightmost trajectory panel illustrates future-guided selection.
Figure 3: Details of latent prediction and trajectory reranking. (a) Plan-future compatibility modeling combines projected features and their agreement and discrepancy to predict a score correction Δstk and confidence logit ρtk . (b) Future-consistent candidate ranking gates the bounded correction, adds the baseline score btk , and selects an original trajectory. (c) JEPA rollout predicts latent futures conditioned on step-wise trajectory intentions over the planning horizon.
Method
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
E2E-based Methods
VADv2 [ 7 ]
97.2
89.1
91.6
100
76.0
80.9
UniAD [ 19 ]
97.8
91.9
92.9
100
78.8
83.4
TransFuser [ 9 ]
97.7
92.8
92.8
100
79.2
84.0
PARA-Drive [ 55 ]
97.9
92.4
93.0
99.8
79.3
84.0
TransFusermmt∗ [ 9 ]
96.2
95.4
90.7
100
80.7
85.1
Table 1: Planning performance on NAVSIM v1 navtest. Protocols: cited studies and Sec. 4.1 . Dark rules separate blocks, each sorted by increasing PDMS with column maxima in bold. ‘mmt’: multimodal TransFuser; TF: TransFuser; ∗ : re-implementation.
Method
NC ↑
DAC ↑
DDC ↑
TL ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
E2E-based Methods
TransFuser [ 9 ]
96.9
89.9
97.8
99.7
87.1
95.4
92.7
98.3
87.2
76.7
Hydra-MDP++ [ 25 ]
97.2
97.5
99.4
99.6
83.1
96.5
94.4
98.2
70.9
81.4
DIVER [ 45 ]
97.5
95.0
98.3
99.8
87.8
95.9
93.0
98.5
87.7
82.2
DriveSuprim [ 62 ]
97.5
96.5
99.4
99.6
88.4
96.6
95.5
98.3
77.0
83.1
DiffusionDrive [ 30 ]
98.2
95.9
99.4
99.8
87.5
97.3
96.8
98.3
87.7
84.5
Table 2: Planning performance on NAVSIM v2 navtest. Evaluation: Sec. 4.1 . Dark rules separate blocks, each sorted by increasing EPDMS with column maxima in bold.
Method
Stage
NC ↑
DAC ↑
DDC ↑
TL ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
E2E-based Methods
TransFuser [ 9 ]
\scriptsize1⃝
96.2
79.5
99.1
99.5
84.1
95.1
94.2
97.5
79.1
23.1
\scriptsize2⃝
77.7
70.2
84.2
98.0
85.1
75.6
45.4
95.7
75.9
DiffusionDrive [ 30 ]
\scriptsize1⃝
96.0
79.7
97.4
99.5
81.3
93.1
90.8
96.8
73.8
24.2
\scriptsize2⃝
82.1
72.2
88.5
98.7
85.1
78.8
49.2
89.3
71.2
DIVER [ 45 ]
\scriptsize1⃝
96.4
80.2
96.0
98.8
82.1
94.2
90.6
96.5
65.2
26.8
Table 3: Planning performance on NAVSIM v2 navhard. [ 5 ] EPDMS is the combined two-stage score; ∗ denotes official-weight reproduction. Bold marks the best result within each block separated by dark horizontal rules, separately for each stage and the combined score.
Method
Venue
L2 (m)↓
Col. Rate (%)↓
1s
2s
3s
4s
5s
6s
Avg.
1s
2s
3s
4s
5s
6s
Avg.
UniAD [ 19 ]
CVPR’23
0.47
0.91
1.35
1.91
2.47
3.07
1.70
0.25
0.36
0.61
0.99
1.64
2.51
1.06
SparseDrive [ 49 ]
CVPR’23
0.43
0.87
1.23
1.75
2.32
2.95
1.59
0.19
0.31
0.56
0.87
1.54
2.33
0.97
MomAD [ 42 ]
CVPR’25
0.41
0.85
1.13
1.67
1.98
2.45
1.42
0.17
0.30
0.54
0.83
1.43
2.13
0.90
LAW [ 26 ]
ICLR’25
0.40
0.87
1.16
1.71
2.03
2.61
1.46
0.19
0.33
0.57
0.86
1.51
2.31
0.96
Epona [ 65 ]
ICCV’25
0.39
0.91
1.17
1.73
2.02
2.75
1.50
0.14
0.18
0.45
0.74
1.48
2.23
0.87
Table 4: Planning results for 6-second long-horizon planning on the nuScenes validation set.
Components
NAVSIM v1
NAVSIM v2
PC-JEPA
R3
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
96.5
95.1
88.2
94.1
87.1
96.9
98.3
97.9
97.6
87.1
97.1
96.1
95.2
84.2
85.3
✓
97.6
96.6
90.8
94.6
90.5
98.4
98.6
98.8
99.6
88.1
98.0
97.2
97.6
84.8
89.6
✓
✓
99.0
98.4
92.2
96.1
94.1
98.7
98.7
99.1
99.9
88.7
98.2
97.7
97.9
85.0
91.6
Table 5: Comparison of the two core module configurations on the NAVSIM v1 and v2 navtest splits. PC-JEPA denotes Plan-Conditioned JEPA, and R3 denotes Relational Residual Reranking.
NAVSIM v1
NAVSIM v2
Variant
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
EPDMS ↑
Baseline
96.5
95.1
88.2
94.1
87.1
85.3
Shared Future
98.7
97.2
91.1
95.9
92.0
89.1
Single-Step Prediction
98.1
98.1
90.5
95.6
91.5
90.0
w/o C–F Interaction
97.9
97.1
90.9
95.2
91.8
90.3
Full PC-JEPA
99.0
98.4
92.2
96.1
94.1
91.6
Table 6: Ablation of core PC-JEPA designs on the NAVSIM v1 and v2 navtest splits. The PC-JEPA variants are compared with Full PC-JEPA while keeping R3 fixed. Baseline and Full match the reference and full configurations in Table 5 , respectively. Bold indicates the best PC-JEPA variant. C–F denotes current–future interaction.
NAVSIM v1
NAVSIM v2
Variant
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
EPDMS ↑
Baseline
96.5
95.1
88.2
94.1
87.1
85.3
Plan–Future Relations
w/o R3
97.6
96.6
90.8
94.6
90.5
89.6
Plan-Only Features
98.9
98.1
92.1
95.7
92.1
90.5
Simple Concat.
98.8
98.3
92.0
95.8
92.3
90.7
Table 7: Ablation of R3 designs on the NAVSIM v1 and v2 navtest splits. PC-JEPA is fixed across the R3 variants. Baseline and Full match Table 6 ; w/o R3 matches PC-JEPA-only in Table 5 . Bold indicates the best result at the displayed precision.
Method
Dim
Params (M)
PDMS ↑
FPS ↑
TransFuser
256
56.0
84.0
75.56
DiffusionDrive
256
60.7
88.1
43.53
Drive-JEPA
256
311.2
93.7
17.23
ActWorld-JEPA
256
324.2
94.1
21.33
Table 8: Model parameter size, inference speed, and performance. FPS covers perception, planning, and selection on one RTX 4090 (batch size 1; 64 shared navtrain inputs), excluding preprocessing, transfers, and scoring. Dim is the planning latent width; bold marks the best PDMS and FPS within each block. Baseline PDMS is published; ours uses aggregate navtest calibration, not an untouched-test estimate.
Figure 4: Qualitative comparison of trajectory selection by Drive-JEPA (top) and ActWorld-JEPA (bottom) across five challenging driving scenarios. Selected trajectories are shown in red.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
TPC (m) ↓
Method
Venue
4s
5s
6s
UniAD [ 19 ]
CVPR’23
1.49
1.81
2.41
VAD [ 22 ]
ICCV’23
1.55
1.73
2.17
SparseDrive [ 49 ]
ICRA’25
1.33
1.66
1.99
MomAD [ 42 ]
CVPR’25
1.19
1.45
1.61
ActWorld-JEPA (Ours)
–
1.04
1.31
1.60
Appendix
Table A1 : Trajectory prediction consistency on the nuScenes validation set. TPC is reported in meters at 4–6 s, with lower values indicating better temporal consistency.
Components
L2 (m) ↓
Col. Rate (%) ↓
PC-JEPA
R3
4 s
5 s
6 s
4 s
5 s
6 s
1.67
1.98
2.45
0.83
1.43
2.13
✓
1.32
1.83
2.41
0.75
1.38
2.09
✓
✓
1.31
1.82
2.39
0.72
1.33
2.03
Appendix
Table A2 : Ablation study of the two core modules on the nuScenes validation set. Results are reported at the challenging 4–6 s horizons. Lower is better for all metrics.
Variant
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
Baseline
96.9
98.3
97.9
97.6
87.1
97.1
96.1
95.2
84.2
85.3
Shared Future
97.9
98.5
98.7
98.5
88.3
97.2
97.0
97.0
83.2
89.1
Single-Step Prediction
98.5
98.6
99.0
99.8
88.6
98.1
97.6
97.1
83.6
90.0
w/o C–F Interaction
98.4
98.6
99.1
99.8
88.6
97.9
97.6
97.4
83.4
90.3
Full PC-JEPA
98.7
98.7
99.1
99.9
88.7
98.2
97.7
97.9
85.0
91.6
Appendix
Table A3 : Detailed NAVSIM v2 navtest results for the core PC-JEPA variants in Table 6 , with R3 fixed. Baseline and Full follow Table 5 . Other variant subscores are from the detailed ablation records; EPDMS follows Table 6 . C–F denotes current–future interaction. Bold indicates the best result at the displayed precision.
Variant
NC ↑
DAC ↑
DDC ↑
TLC ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ↑
Baseline
96.9
98.3
97.9
97.6
87.1
97.1
96.1
95.2
84.2
85.3
Plan–Future Relations
w/o R3
98.4
98.6
98.8
99.6
88.1
98.0
97.2
97.6
84.8
89.6
Plan-Only Features
98.5
98.6
99.1
99.8
88.6
97.9
97.7
97.9
84.9
90.5
Simple Concat.
98.6
98.7
99.1
98.7
88.7
98.1
97.7
97.9
84.7
90.7
w/o Compatibility
98.6
98.6
99.1
99.2
88.5
98.1
97.7
97.9
85.0
90.0
Appendix
Table A4 : Detailed NAVSIM v2 navtest results for the R3 variants in Table 7 , with PC-JEPA fixed. Baseline, w/o R3 , and Full follow Table 5 . Other variant subscores are from the detailed ablation records; EPDMS follows Table 7 . Bold indicates the best result at the displayed precision.
Figure A1 : Additional qualitative planning results on NAVSIM. Front-view semantic overlays and corresponding bird’s-eye-view trajectories across diverse driving scenarios illustrate the correspondence between semantic scene awareness and planning decisions. ActWorld-JEPA generates plausible trajectories adapted to road geometry and surrounding traffic, illustrating scene-dependent trajectory selection.
Figure A2 : Qualitative 6-second planning results on nuScenes. ActWorld-JEPA generates coherent long-horizon trajectories across diverse driving scenarios, adapting to road geometry and surrounding traffic while producing plausible planned motions.