AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving

Haoran Zhu; Wancong Zhang; Yann LeCun; Anna Choromanska

cs.ROSep 28, 2026

AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving

Authors: Haoran Zhu, Wancong Zhang, Yann LeCun, Anna Choromanska

Organizations: New York University · AMI Labs

Abstract

Autonomous driving requires \textit{world models} that can understand the physical world, reason and plan, and operate safely. In this paper, we first systematically evaluate existing action-conditioned joint-embedding predictive architecture (JEPA) world models, including LeWM, DINO-WM, and JEPA-WM for end-to-end autonomous driving (E2EAD). To isolate world-model quality from policy learning, we employ a goal-conditioned zero-shot planning setting that evaluates these models using ground-truth future observations as goals, without training any driving policy. We find that existing JEPA-based world models are either accurate for driving but computationally expensive, or computationally efficient but insufficient for planning. To address this trade-off, we propose \textbf{AD-E2E-JEPA}, which introduces a SIGReg-regularized learnable projector applied to projected patch embeddings. The projector reduces the number of planning patches by $16\times$ and the embedding dimension by $4\times$ , achieving a $100\times$ inference speedup while retaining planning performance, with a 0.8-second runtime for an 8-frame rollout over 256 candidate trajectories. \textit{Without} training any driving policy, the world model itself reaches the goals located 20 meters away on average within the displacement of respectively 4.0/2.8 meters, using world-model rollouts over trajectory vocabularies of respectively 256/8,192 candidates. On the NAVSIMv2 benchmark, it achieves 67.3/72.9 EPDMS with multiplicative safety metrics and 84.1/86.5 EPDMS $^{\dagger}$ without them in goal-conditioned zero-shot planning. Experiments further show that the self-supervised pretrained projector improves downstream imitation learning performance from 80.2 to 85.4 EPDMS. The source code is available at https://github.com/HaoranZhuExplorer/AD-E2E-JEPA

Figures & tables

Figure 1: AD-E2E-JEPA is an action-conditioned, JEPA-based world model for end-to-end autonomous driving. Without training any driving policy, it enables driving via goal-conditioned zero-shot planning, where future sensor observations are specified as the goal and a trajectory is selected from the trajectory vocabulary solely via world-model rollouts. AD-E2E-JEPA demonstrates strong zero-shot driving performance, efficient planning, low displacement errors, high trajectory hit rates, and representations that transfer effectively to downstream imitation learning. Zero-shot planning metrics are evaluated on 100 subsampled test scenes for a fair comparison with the time-consuming DINO-WM/JEPA-WM. See Table 2 for AD-E2E-JEPA’s results on the full set of 12,146 test scenes.

Figure 2: AD-E2E-JEPA architecture: (a) A self-supervised, action-conditioned JEPA world model for E2EAD. The DINOv3 encoder is frozen, while (a.1) a learnable projector is shared by the context-history and target-future branches, compressing the representation by $16\times$ and its dimensionality by $4\times$ , with SIGReg maximizing information while avoiding collapse and improving planning performance. This enables a $100\times$ speedup in zero-shot goal-conditioned planning in (b) , where trajectories are selected via world-model rollouts over a driving vocabulary. (c) The projector pretrained through self-supervision also improves downstream imitation learning.

Split	Variant	GPUs	Batch size	Learning rate	$\lambda$	Training time
navtrain	LeWM	$4\times$ A100	8	$1\times 10^{-4}$	0.09	1 d
	DINO-WM	$4\times$ A100	64	$1\times 10^{-4}$	–	11 h
	JEPA-WM	$4\times$ A100	64	$1\times 10^{-4}$	–	13 h
	AD-E2E-JEPA	$1\times$ A100	128	$1\times 10^{-4}$	0.09	20 h
	+ rollout	$1\times$ A100	128	$1\times 10^{-4}$	0.09	1 d 22 h
trainval	AD-E2E-JEPA	$4\times$ A100	512	$2\times 10^{-4}$	0.025	2 d 5 h

Table 1: Training configurations.

Figure 3: Zero-shot goal-conditioned planning qualitative comparison. LeWM is efficient but insufficient for accurate planning, resulting in a shifted selected trajectory. DINO-WM and JEPA-WM better match the ground-truth trajectory. AD-E2E-JEPA achieves comparable planning quality to DINO-WM and JEPA-WM while being 100 $\times$ faster.

100 subsampled test scenes
		Driving performance		Efficiency	Geodesic accuracy				Reliability
Method	Split	EPDMS $\uparrow$	EPDMS ${}^{\dagger}\uparrow$	Time $\downarrow$	FDE $\downarrow$	$\Delta x$ $\downarrow$	$\Delta y$ $\downarrow$	$\Delta\theta$ $\downarrow$	Hit rate $\uparrow$
LeWM	navtrain	48.3	73.9	0.7	12.4	11.3	2.7	13.6	6/18
DINO-WM		68.3	91.4	91.8	3.9	3.4	1.1	5.9	40/73
JEPA-WM		74.2	90.9	101.0	4.0	3.4	1.2	5.2	45/75
AD-E2E-JEPA		76.6	92.4	0.8	4.2	3.9	1.0	4.7	27/59

Table 2: Zero-shot goal-conditioned planning on the NAVSIM dataset, evaluated on 100 subsampled test scenes and the full set of 12,146 test scenes, with the ground-truth future frame used as the target. Unless specified, the world model selects from 256 candidate trajectories. The NAVSIM-v2 navtest-stage-1 metrics EPDMS

\uparrow

and EPDMS

{}^{\dagger}\uparrow

measure driving performance, where EPDMS includes safety metrics and EPDMS † excludes them. Per-scene planning time (s) measures efficiency on an A100 GPU. FDE (m),

\Delta x

(m),

\Delta y

(m), and

\Delta\theta

(degrees) measure geodesic accuracy. Top-1/top-5 hit rate (%) measures world-model reliability.

\uparrow

: higher is better;

\downarrow

: lower is better. For the 100 subsampled-test scene setting, the extended comfort (EC) metrics for EPDMS and EPDMS † are excluded because the subsampled set does not contain temporally adjacent scenes. See Appendix A.4 for details of each submetric of EPDMS.

				NAVSIMv2 stage 1 driving metrics
Method	V	Type	Fr.	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS ${}^{*}\uparrow$	EPDMS $\uparrow$
Transfuser	MV	PF	1	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7	–
Latent-WAM	MV	PF	4	98.1	97.3	99.6	99.8	87.7	97.3	97.6	98.1	72.4	–	89.3
WA-JEPA	MV	PF	4	99.4	98.2	99.7	99.9	87.8	98.9	98.3	98.3	88.1	88.0	91.7
Drive-JEPA	SV	PB	2	98.4	98.6	99.1	99.8	88.4	97.8	97.6	97.9	84.8	87.8	–
DINOv3

Table 3: NAVSIMv2 navtest-stage-1 benchmark. V: single-view (SV) or multi-view (MV); Type: perception-free (PF) or perception-based (PB); Fr.: number of input frames. PB methods use perception annotations or trajectory-score labels. ∗ denotes results obtained before the human-filter bug fix in commit 359c7f7 in NAVSIM’s codebase.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Split	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$	EPDMS † $\uparrow$
LeWM	navtrain	78.5	77.0	84.0	97.0	68.6	76.0	86.0	70.0	–	48.3	73.9
DINO-WM		96.0	79.0	96.5	99.0	87.8	93.0	92.0	96.0	–	68.3	91.4
JEPA-WM		94.5	87.0	96.5	99.0	87.6	92.0	94.0	93.0	–	74.2	90.9
AD-E2E-JEPA		96.0	87.0	95.5	99.0	88.0	95.0	94.0	95.0	–	76.6	92.4
+ rollout		96.0	77.0	95.5	100.0	87.9	95.0	93.0	95.0	–	70.4	92.2
AD-E2E-JEPA	trainval	94.0	82.0	96.5	100.0	87.2	92.0	91.0	90.0	–	72.1	89.8

Appendix

Table 4: EPDMS details for 100 subsampled test scenes.

Method	Split	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS $\uparrow$	EPDMS † $\uparrow$
LeWM	navtrain	82.0	67.5	81.4	98.5	66.1	79.5	79.0	66.7	13.7	39.8	66.7
AD-E2E-JEPA		93.1	82.4	95.6	99.5	80.6	90.9	89.4	89.6	21.7	63.5	80.1
+ rollout		94.7	80.9	93.8	99.7	84.0	92.5	87.9	91.8	34.6	64.9	83.1
AD-E2E-JEPA	trainval	92.8	82.8	95.3	99.4	82.0	90.6	89.8	88.5	19.5	63.2	80.1
+ rollout		95.2	82.0	94.4	99.6	85.2	93.3	89.3	92.3	35.5	67.3	84.1
+ rollout, 512 traj.		95.9	83.6	95.5	99.6	85.6	94.4	90.4	93.5	36.9	69.2	85.0

Appendix

Table 5: EPDMS details for the full set of 12,146 test scenes.

Figure 4: Downstream imitation learning architecture.

Explore similar work

Jul 31, 2026cs.RO

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving

Existing autonomous-driving world models typically perform dense prediction of future videos, occupancy states, BEV representations, or agent motion. We argue that planning need not reconstruct the complete future world, but only focus on scene features that affect future ego action. Based on this perspective, we propose Auto-JEPA, an action-oriented latent world model that learns continuous future driving intent through joint-embedding prediction. Given visual observations, egomotion history, and navigation commands, Auto-JEPA predicts an intent embedding aligned with the latent representation of the future ego trajectory. The predicted intent retrieves executable trajectories from a fixed trajectory memory, which are then ranked by a scene-conditioned candidate selection module. Auto-JEPA keeps the visual encoder frozen, requires no explicit perception annotations, and uses no learned trajectory generator. By optimizing only task-specific modules for trajectory representation, intent prediction, and candidate selection, Auto-JEPA achieves 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2. Semantic occlusion experiments show that masking dynamic-agent regions induces an average intent change 2.97x that of equal-area random masking. Moreover, occluding vehicles that affect future driving substantially changes the predicted intent and selected trajectory, whereas both remain essentially unchanged when non-influential vehicles are occluded. These results show that future-intent prediction encourages the model to focus on planning-relevant visual features and supports high-quality planning without dense future-world modeling.

Jiwei Yang, Zhengxian Chen, Chaosheng Huang +1

School of Vehicle and Mobility, Tsinghua University · 2State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University

Autonomous Driving Video Joint Embedding Predictive Architecture Ego-Trajectory Prediction Latent World Models Action Prediction S-Jepa+1

Ask PDF

Jan 29, 2026cs.CV

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.

Linhan Wang, Zichong Yang, Chen Bai +6

Virginia Tech · Purdue University · XPENG Motors +1

Causality-Aware End-To-End Autonomous Driving Autonomous Driving Video Joint Embedding Predictive Architecture Ego-Trajectory Prediction Joint-Embedding Predictive Architectures S-Jepa+1

Ask PDF

Apr 5, 2026cs.CV

DriveVA: Video Action Models are Zero-Shot Drivers

Generalization is a central challenge in autonomous driving, as real-world deployment requires robust performance under unseen scenarios, sensor domains, and environmental conditions. Recent world-model-based planning methods have shown strong capabilities in scene understanding and multi-modal future prediction, yet their generalization across datasets and sensor configurations remains limited. In addition, their loosely coupled planning paradigm often leads to poor video-trajectory consistency during visual imagination. To overcome these limitations, we propose DriveVA, a novel autonomous driving world model that jointly decodes future visual forecasts and action sequences in a shared latent generative process. DriveVA inherits rich priors on motion dynamics and physical plausibility from well-pretrained large-scale video generation models to capture continuous spatiotemporal evolution and causal interaction patterns. To this end, DriveVA employs a DiT-based decoder to jointly predict future action sequences (trajectories) and videos, enabling tighter alignment between planning and scene evolution. We also introduce a video continuation strategy to strengthen long-duration rollout consistency. DriveVA achieves an impressive PDM-based planning performance of 90.9 PDM score on the NAVSIM benchmark. Extensive experiments also demonstrate the zero-shot capability and cross-domain generalization of DriveVA, which reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes and 52.5% and 52.4% on the Bench2Drive built on CARLA v2 compared with the state-of-the-art world-model-based planner.

Mengmeng Liu, Diankun Zhang, Jiuming Liu +7

University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1

Autonomous Driving Drives World Model Planning Generative Video Models

Ask PDF