AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving

Haoran Zhu; Wancong Zhang; Yann LeCun; Anna Choromanska

cs.ROSep 28, 2026

AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving

Authors: Haoran Zhu, Wancong Zhang, Yann LeCun, Anna Choromanska

Organizations: New York University · AMI Labs

Abstract

Autonomous driving requires \textit{world models} that can understand the physical world, reason and plan, and operate safely. In this paper, we first systematically evaluate existing action-conditioned joint-embedding predictive architecture (JEPA) world models, including LeWM, DINO-WM, and JEPA-WM for end-to-end autonomous driving (E2EAD). To isolate world-model quality from policy learning, we employ a goal-conditioned zero-shot planning setting that evaluates these models using ground-truth future observations as goals, without training any driving policy. We find that existing JEPA-based world models are either accurate for driving but computationally expensive, or computationally efficient but insufficient for planning. To address this trade-off, we propose \textbf{AD-E2E-JEPA}, which introduces a SIGReg-regularized learnable projector applied to projected patch embeddings. The projector reduces the number of planning patches by $16\times$ and the embedding dimension by $4\times$ , achieving a $100\times$ inference speedup while retaining planning performance, with a 0.8-second runtime for an 8-frame rollout over 256 candidate trajectories. \textit{Without} training any driving policy, the world model itself reaches the goals located 20 meters away on average within the displacement of respectively 4.0/2.8 meters, using world-model rollouts over trajectory vocabularies of respectively 256/8,192 candidates. On the NAVSIMv2 benchmark, it achieves 67.3/72.9 EPDMS with multiplicative safety metrics and 84.1/86.5 EPDMS $^{\dagger}$ without them in goal-conditioned zero-shot planning. Experiments further show that the self-supervised pretrained projector improves downstream imitation learning performance from 80.2 to 85.4 EPDMS. The source code is available at https://github.com/HaoranZhuExplorer/AD-E2E-JEPA

Figures & tables

Figure 1: AD-E2E-JEPA is an action-conditioned, JEPA-based world model for end-to-end autonomous driving. Without training any driving policy, it enables driving via goal-conditioned zero-shot planning, where future sensor observations are specified as the goal and a trajectory is selected from the trajectory vocabulary solely via world-model rollouts. AD-E2E-JEPA demonstrates strong zero-shot driving performance, efficient planning, low displacement errors, high trajectory hit rates, and representations that transfer effectively to downstream imitation learning. Zero-shot planning metrics are evaluated on 100 subsampled test scenes for a fair comparison with the time-consuming DINO-WM/JEPA-WM. See Table 2 for AD-E2E-JEPA’s results on the full set of 12,146 test scenes.

Figure 2: AD-E2E-JEPA architecture: (a) A self-supervised, action-conditioned JEPA world model for E2EAD. The DINOv3 encoder is frozen, while (a.1) a learnable projector is shared by the context-history and target-future branches, compressing the representation by $16\times$ and its dimensionality by $4\times$ , with SIGReg maximizing information while avoiding collapse and improving planning performance. This enables a $100\times$ speedup in zero-shot goal-conditioned planning in (b) , where trajectories are selected via world-model rollouts over a driving vocabulary. (c) The projector pretrained through self-supervision also improves downstream imitation learning.

Split	Variant	GPUs	Batch size	Learning rate	$\lambda$	Training time
navtrain	LeWM	$4\times$ A100	8	$1\times 10^{-4}$	0.09	1 d
	DINO-WM	$4\times$ A100	64	$1\times 10^{-4}$	–	11 h
	JEPA-WM	$4\times$ A100	64	$1\times 10^{-4}$	–	13 h
	AD-E2E-JEPA	$1\times$ A100	128	$1\times 10^{-4}$	0.09	20 h
	+ rollout	$1\times$ A100	128	$1\times 10^{-4}$	0.09	1 d 22 h
trainval	AD-E2E-JEPA	$4\times$ A100	512	$2\times 10^{-4}$	0.025	2 d 5 h

Table 1: Training configurations.

Figure 3: Zero-shot goal-conditioned planning qualitative comparison. LeWM is efficient but insufficient for accurate planning, resulting in a shifted selected trajectory. DINO-WM and JEPA-WM better match the ground-truth trajectory. AD-E2E-JEPA achieves comparable planning quality to DINO-WM and JEPA-WM while being 100 $\times$ faster.

100 subsampled test scenes
		Driving performance		Efficiency	Geodesic accuracy				Reliability
Method	Split	EPDMS $\uparrow$	EPDMS ${}^{\dagger}\uparrow$	Time $\downarrow$	FDE $\downarrow$	$\Delta x$ $\downarrow$	$\Delta y$ $\downarrow$	$\Delta\theta$ $\downarrow$	Hit rate $\uparrow$
LeWM	navtrain	48.3	73.9	0.7	12.4	11.3	2.7	13.6	6/18
DINO-WM		68.3	91.4	91.8	3.9	3.4	1.1	5.9	40/73
JEPA-WM		74.2	90.9	101.0	4.0	3.4	1.2	5.2	45/75
AD-E2E-JEPA		76.6	92.4	0.8	4.2	3.9	1.0	4.7	27/59

Table 2: Zero-shot goal-conditioned planning on the NAVSIM dataset, evaluated on 100 subsampled test scenes and the full set of 12,146 test scenes, with the ground-truth future frame used as the target. Unless specified, the world model selects from 256 candidate trajectories. The NAVSIM-v2 navtest-stage-1 metrics EPDMS

\uparrow

and EPDMS

{}^{\dagger}\uparrow

measure driving performance, where EPDMS includes safety metrics and EPDMS † excludes them. Per-scene planning time (s) measures efficiency on an A100 GPU. FDE (m),

\Delta x

(m),

\Delta y

(m), and

\Delta\theta

(degrees) measure geodesic accuracy. Top-1/top-5 hit rate (%) measures world-model reliability.

\uparrow

: higher is better;

\downarrow

: lower is better. For the 100 subsampled-test scene setting, the extended comfort (EC) metrics for EPDMS and EPDMS † are excluded because the subsampled set does not contain temporally adjacent scenes. See Appendix A.4 for details of each submetric of EPDMS.

				NAVSIMv2 stage 1 driving metrics
Method	V	Type	Fr.	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS ${}^{*}\uparrow$	EPDMS $\uparrow$
Transfuser	MV	PF	1	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7	–
Latent-WAM	MV	PF	4	98.1	97.3	99.6	99.8	87.7	97.3	97.6	98.1	72.4	–	89.3
WA-JEPA	MV	PF	4	99.4	98.2	99.7	99.9	87.8	98.9	98.3	98.3	88.1	88.0	91.7
Drive-JEPA	SV	PB	2	98.4	98.6	99.1	99.8	88.4	97.8	97.6	97.9	84.8	87.8	–
DINOv3

Table 3: NAVSIMv2 navtest-stage-1 benchmark. V: single-view (SV) or multi-view (MV); Type: perception-free (PF) or perception-based (PB); Fr.: number of input frames. PB methods use perception annotations or trajectory-score labels. ∗ denotes results obtained before the human-filter bug fix in commit 359c7f7 in NAVSIM’s codebase.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Split	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$	EPDMS † $\uparrow$
LeWM	navtrain	78.5	77.0	84.0	97.0	68.6	76.0	86.0	70.0	–	48.3	73.9
DINO-WM		96.0	79.0	96.5	99.0	87.8	93.0	92.0	96.0	–	68.3	91.4
JEPA-WM		94.5	87.0	96.5	99.0	87.6	92.0	94.0	93.0	–	74.2	90.9
AD-E2E-JEPA		96.0	87.0	95.5	99.0	88.0	95.0	94.0	95.0	–	76.6	92.4
+ rollout		96.0	77.0	95.5	100.0	87.9	95.0	93.0	95.0	–	70.4	92.2
AD-E2E-JEPA	trainval	94.0	82.0	96.5	100.0	87.2	92.0	91.0	90.0	–	72.1	89.8

Appendix

Table 4: EPDMS details for 100 subsampled test scenes.

Method	Split	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$	EPDMS † $\uparrow$
LeWM	navtrain	82.0	67.5	81.4	98.5	66.1	79.5	79.0	66.7	13.7	39.8	66.7
AD-E2E-JEPA		93.1	82.4	95.6	99.5	80.6	90.9	89.4	89.6	21.7	63.5	80.1
+ rollout		94.7	80.9	93.8	99.7	84.0	92.5	87.9	91.8	34.6	64.9	83.1
AD-E2E-JEPA	trainval	92.8	82.8	95.3	99.4	82.0	90.6	89.8	88.5	19.5	63.2	80.1
+ rollout		95.2	82.0	94.4	99.6	85.2	93.3	89.3	92.3	35.5	67.3	84.1
+ rollout, 512 traj.		95.9	83.6	95.5	99.6	85.6	94.4	90.4	93.5	36.9	69.2	85.0

Appendix

Table 5: EPDMS details for the full set of 12,146 test scenes.

Figure 4: Downstream imitation learning architecture.

Explore similar work

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving
Jul 31, 2026Jiwei Yang, Zhengxian Chen, Chaosheng Huang +1Autonomous Driving Video Joint Embedding Predictive Architecture
School of Vehicle and Mobility, Tsinghua University · 2State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University
Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Jan 29, 2026Linhan Wang, Zichong Yang, Chen Bai +6Causality-Aware End-To-End Autonomous Driving Autonomous Driving
Virginia Tech · Purdue University · XPENG Motors +1
DriveVA: Video Action Models are Zero-Shot Drivers
Apr 5, 2026Mengmeng Liu, Diankun Zhang, Jiuming Liu +7Autonomous Driving Drives
University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1

Jul 31, 2026, cs.ROJ/K move · Enter open · S save

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving

Jiwei Yang, Zhengxian Chen, Chaosheng Huang, Jun Li

School of Vehicle and Mobility, Tsinghua University · 2State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University

Autonomous Driving Video Joint Embedding Predictive Architecture Ego-Trajectory Prediction Latent World Models Action Prediction S-Jepa+1

Open paper AskPDF arXiv

Existing autonomous-driving world models typically perform dense prediction of future videos, occupancy states, BEV representations, or agent motion. We argue that planning need not reconstruct the complete future world, but only focus on scene features that affect future ego action. Based on this perspective, we propose Auto-JEPA, an action-oriented latent world model that learns continuous future driving intent through joint-embedding prediction. Given visual observations, egomotion history, and navigation commands, Auto-JEPA predicts an intent embedding aligned with the latent representation of the future ego trajectory. The predicted intent retrieves executable trajectories from a fixed trajectory memory, which are then ranked by a scene-conditioned candidate selection module. Auto-JEPA keeps the visual encoder frozen, requires no explicit perception annotations, and uses no learned trajectory generator. By optimizing only task-specific modules for trajectory representation, intent prediction, and candidate selection, Auto-JEPA achieves 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2. Semantic occlusion experiments show that masking dynamic-agent regions induces an average intent change 2.97x that of equal-area random masking. Moreover, occluding vehicles that affect future driving substantially changes the predicted intent and selected trajectory, whereas both remain essentially unchanged when non-influential vehicles are occluded. These results show that future-intent prediction encourages the model to focus on planning-relevant visual features and supports high-quality planning without dense future-world modeling.

Abstract

Explore similar work

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

DriveVA: Video Action Models are Zero-Shot Drivers