ActWorld-JEPA: Action-Conditioned JEPA World Models Meet Latent Planning for End-to-End Autonomous Driving
Organizations: Nanyang Technological University · Beijing Jiaotong University · Dalian University of Technology · North University of China ·
Abstract
Joint-Embedding Predictive Architectures (JEPAs) enable future prediction in latent space without pixel reconstruction. However, latent predictive consistency alone does not establish the relative driving quality of candidate plans. We present ActWorld-JEPA, a joint-embedding formulation for candidate-based driving. Where Plan-Conditioned JEPA (PC-JEPA) uses global and step-wise trajectory intentions to guide recurrent latent prediction, then combines the predicted future with the current scene through gated cross-attention; Relational Residual Reranking () models agreement and discrepancy between plan and future representations under planning supervision, yielding bounded, gated residual corrections to the base planning scores. This formulation connects latent prediction to candidate selection using only observed context and proposed trajectories at inference. Benchmark-specific implementations report 94.1 PDMS on NAVSIM v1 navtest, with the final selector calibrated using aggregate NAVTEST feedback, and 91.6 and 42.7 EPDMS on NAVSIM v2 navtest and navhard, respectively. The nuScenes evaluation reports a mean error of 1.20 m and a predicted collision rate of 0.76% over 1--6 s. This work provides an effective path from JEPA's predictive representations to driving decisions informed by anticipated consequences.
Figures & tables
| Method | NC | DAC | TTC | Comf. | EP | PDMS |
| E2E-based Methods | ||||||
| VADv2 [ 7 ] | 97.2 | 89.1 | 91.6 | 100 | 76.0 | 80.9 |
| UniAD [ 19 ] | 97.8 | 91.9 | 92.9 | 100 | 78.8 | 83.4 |
| TransFuser [ 9 ] | 97.7 | 92.8 | 92.8 | 100 | 79.2 | 84.0 |
| PARA-Drive [ 55 ] | 97.9 | 92.4 | 93.0 | 99.8 | 79.3 | 84.0 |
| [ 9 ] | 96.2 | 95.4 | 90.7 | 100 | 80.7 | 85.1 |
| Method | NC | DAC | DDC | TL | EP | TTC | LK | HC | EC | EPDMS |
|---|---|---|---|---|---|---|---|---|---|---|
| E2E-based Methods | ||||||||||
| TransFuser [ 9 ] | 96.9 | 89.9 | 97.8 | 99.7 | 87.1 | 95.4 | 92.7 | 98.3 | 87.2 | 76.7 |
| Hydra-MDP++ [ 25 ] | 97.2 | 97.5 | 99.4 | 99.6 | 83.1 | 96.5 | 94.4 | 98.2 | 70.9 | 81.4 |
| DIVER [ 45 ] | 97.5 | 95.0 | 98.3 | 99.8 | 87.8 | 95.9 | 93.0 | 98.5 | 87.7 | 82.2 |
| DriveSuprim [ 62 ] | 97.5 | 96.5 | 99.4 | 99.6 | 88.4 | 96.6 | 95.5 | 98.3 | 77.0 | 83.1 |
| DiffusionDrive [ 30 ] | 98.2 | 95.9 | 99.4 | 99.8 | 87.5 | 97.3 | 96.8 | 98.3 | 87.7 | 84.5 |
| Method | Stage | NC | DAC | DDC | TL | EP | TTC | LK | HC | EC | EPDMS |
|---|---|---|---|---|---|---|---|---|---|---|---|
| E2E-based Methods | |||||||||||
| TransFuser [ 9 ] | \scriptsize1⃝ | 96.2 | 79.5 | 99.1 | 99.5 | 84.1 | 95.1 | 94.2 | 97.5 | 79.1 | 23.1 |
| \scriptsize2⃝ | 77.7 | 70.2 | 84.2 | 98.0 | 85.1 | 75.6 | 45.4 | 95.7 | 75.9 | ||
| DiffusionDrive [ 30 ] | \scriptsize1⃝ | 96.0 | 79.7 | 97.4 | 99.5 | 81.3 | 93.1 | 90.8 | 96.8 | 73.8 | 24.2 |
| \scriptsize2⃝ | 82.1 | 72.2 | 88.5 | 98.7 | 85.1 | 78.8 | 49.2 | 89.3 | 71.2 | ||
| DIVER [ 45 ] | \scriptsize1⃝ | 96.4 | 80.2 | 96.0 | 98.8 | 82.1 | 94.2 | 90.6 | 96.5 | 65.2 | 26.8 |
| 1s | 2s | 3s | 4s | 5s | 6s | 1s | 2s | 3s | 4s | 5s | 6s | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| [ 19 ] | CVPR’23 | 0.47 | 0.91 | 1.35 | 1.91 | 2.47 | 3.07 | 1.70 | 0.25 | 0.36 | 0.61 | 0.99 | 1.64 | 2.51 | 1.06 |
| [ 49 ] | CVPR’23 | 0.43 | 0.87 | 1.23 | 1.75 | 2.32 | 2.95 | 1.59 | 0.19 | 0.31 | 0.56 | 0.87 | 1.54 | 2.33 | 0.97 |
| [ 42 ] | CVPR’25 | 0.41 | 0.85 | 1.13 | 1.67 | 1.98 | 2.45 | 1.42 | 0.17 | 0.30 | 0.54 | 0.83 | 1.43 | 2.13 | 0.90 |
| [ 26 ] | ICLR’25 | 0.40 | 0.87 | 1.16 | 1.71 | 2.03 | 2.61 | 1.46 | 0.19 | 0.33 | 0.57 | 0.86 | 1.51 | 2.31 | 0.96 |
| [ 65 ] | ICCV’25 | 0.39 | 0.91 | 1.17 | 1.73 | 2.02 | 2.75 | 1.50 | 0.14 | 0.18 | 0.45 | 0.74 | 1.48 | 2.23 | 0.87 |
| Components | NAVSIM v1 | NAVSIM v2 | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PC-JEPA | NC | DAC | EP | TTC | PDMS | NC | DAC | DDC | TLC | EP | TTC | LK | HC | EC | EPDMS | |
| 96.5 | 95.1 | 88.2 | 94.1 | 87.1 | 96.9 | 98.3 | 97.9 | 97.6 | 87.1 | 97.1 | 96.1 | 95.2 | 84.2 | 85.3 | ||
| ✓ | 97.6 | 96.6 | 90.8 | 94.6 | 90.5 | 98.4 | 98.6 | 98.8 | 99.6 | 88.1 | 98.0 | 97.2 | 97.6 | 84.8 | 89.6 | |
| ✓ | ✓ | 99.0 | 98.4 | 92.2 | 96.1 | 94.1 | 98.7 | 98.7 | 99.1 | 99.9 | 88.7 | 98.2 | 97.7 | 97.9 | 85.0 | 91.6 |
| NAVSIM v1 | NAVSIM v2 | |||||
| Variant | NC | DAC | EP | TTC | PDMS | EPDMS |
| Baseline | 96.5 | 95.1 | 88.2 | 94.1 | 87.1 | 85.3 |
| Shared Future | 98.7 | 97.2 | 91.1 | 95.9 | 92.0 | 89.1 |
| Single-Step Prediction | 98.1 | 98.1 | 90.5 | 95.6 | 91.5 | 90.0 |
| w/o C–F Interaction | 97.9 | 97.1 | 90.9 | 95.2 | 91.8 | 90.3 |
| Full PC-JEPA | 99.0 | 98.4 | 92.2 | 96.1 | 94.1 | 91.6 |
| NAVSIM v1 | NAVSIM v2 | |||||
| Variant | NC | DAC | EP | TTC | PDMS | EPDMS |
| Baseline | 96.5 | 95.1 | 88.2 | 94.1 | 87.1 | 85.3 |
| Plan–Future Relations | ||||||
| w/o | 97.6 | 96.6 | 90.8 | 94.6 | 90.5 | 89.6 |
| Plan-Only Features | 98.9 | 98.1 | 92.1 | 95.7 | 92.1 | 90.5 |
| Simple Concat. | 98.8 | 98.3 | 92.0 | 95.8 | 92.3 | 90.7 |
| Method | Dim | Params (M) | PDMS | FPS |
|---|---|---|---|---|
| TransFuser | 256 | 56.0 | 84.0 | 75.56 |
| DiffusionDrive | 256 | 60.7 | 88.1 | 43.53 |
| Drive-JEPA | 256 | 311.2 | 93.7 | 17.23 |
| ActWorld-JEPA | 256 | 324.2 | 94.1 | 21.33 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| TPC (m) | ||||
|---|---|---|---|---|
| Method | Venue | 4s | 5s | 6s |
| UniAD [ 19 ] | CVPR’23 | 1.49 | 1.81 | 2.41 |
| VAD [ 22 ] | ICCV’23 | 1.55 | 1.73 | 2.17 |
| SparseDrive [ 49 ] | ICRA’25 | 1.33 | 1.66 | 1.99 |
| MomAD [ 42 ] | CVPR’25 | 1.19 | 1.45 | 1.61 |
| ActWorld-JEPA (Ours) | – | 1.04 | 1.31 | 1.60 |
| Components | L2 (m) | Col. Rate (%) | |||||
|---|---|---|---|---|---|---|---|
| PC-JEPA | 4 s | 5 s | 6 s | 4 s | 5 s | 6 s | |
| 1.67 | 1.98 | 2.45 | 0.83 | 1.43 | 2.13 | ||
| ✓ | 1.32 | 1.83 | 2.41 | 0.75 | 1.38 | 2.09 | |
| ✓ | ✓ | 1.31 | 1.82 | 2.39 | 0.72 | 1.33 | 2.03 |
| Variant | NC | DAC | DDC | TLC | EP | TTC | LK | HC | EC | EPDMS |
|---|---|---|---|---|---|---|---|---|---|---|
| Baseline | 96.9 | 98.3 | 97.9 | 97.6 | 87.1 | 97.1 | 96.1 | 95.2 | 84.2 | 85.3 |
| Shared Future | 97.9 | 98.5 | 98.7 | 98.5 | 88.3 | 97.2 | 97.0 | 97.0 | 83.2 | 89.1 |
| Single-Step Prediction | 98.5 | 98.6 | 99.0 | 99.8 | 88.6 | 98.1 | 97.6 | 97.1 | 83.6 | 90.0 |
| w/o C–F Interaction | 98.4 | 98.6 | 99.1 | 99.8 | 88.6 | 97.9 | 97.6 | 97.4 | 83.4 | 90.3 |
| Full PC-JEPA | 98.7 | 98.7 | 99.1 | 99.9 | 88.7 | 98.2 | 97.7 | 97.9 | 85.0 | 91.6 |
| Variant | NC | DAC | DDC | TLC | EP | TTC | LK | HC | EC | EPDMS |
|---|---|---|---|---|---|---|---|---|---|---|
| Baseline | 96.9 | 98.3 | 97.9 | 97.6 | 87.1 | 97.1 | 96.1 | 95.2 | 84.2 | 85.3 |
| Plan–Future Relations | ||||||||||
| w/o | 98.4 | 98.6 | 98.8 | 99.6 | 88.1 | 98.0 | 97.2 | 97.6 | 84.8 | 89.6 |
| Plan-Only Features | 98.5 | 98.6 | 99.1 | 99.8 | 88.6 | 97.9 | 97.7 | 97.9 | 84.9 | 90.5 |
| Simple Concat. | 98.6 | 98.7 | 99.1 | 98.7 | 88.7 | 98.1 | 97.7 | 97.9 | 84.7 | 90.7 |
| w/o Compatibility | 98.6 | 98.6 | 99.1 | 99.2 | 88.5 | 98.1 | 97.7 | 97.9 | 85.0 | 90.0 |
World Models for Autonomous Driving
World models for autonomous driving predict how scenes and traffic evolve under candidate actions to support planning, control, and simulation.
See all 170 papers