WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving

Xinlin Wang; Yujiao Xiang; Yuheng Zhou; Jingqi Wang; Minqing Huang; Jiajie Huang; Dongxu Wei; Tingguang Zhou

cs.CVAug 21, 2026

WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving

Authors: Xinlin Wang, Yujiao Xiang, Yuheng Zhou, Jingqi Wang, Minqing Huang, Jiajie Huang, Dongxu Wei, Tingguang Zhou,

Organizations: Afari Intelligent Drive · University of Electronic Science and Technology of China · Southeast University · Beijing University of Posts and Telecommunications ·

Abstract

Video Joint Embedding Predictive Architecture (V-JEPA) learns powerful spatiotemporal representations from video through self-supervised latent feature prediction. However, V-JEPA is built around random-mask completion and deterministic regression, making it fundamentally ill-suited for autonomous driving planning that demands future-directed prediction tightly coupled with action. To address this, we rethink the V-JEPA paradigm and present WA-JEPA, a V-JEPA-native world-action model designed for autonomous driving planning. Instead of random spatiotemporal masking, WA-JEPA employs hybrid future-masked pre-training, where the model infers future latents from observed context. Departing from deterministic regression, we recast future prediction as conditional flow matching over latent futures, which substantially improves the model's ability to generate plausible future latents for downstream planning. Finally, a joint future-action predictor is proposed to denoise future scene tokens and ego trajectories together in a unified spatiotemporal latent space, allowing action supervision to directly shape planning-relevant world representations. Pre-trained on nuPlan videos and fine-tuned on NAVSIM, WA-JEPA reaches 91.7 EPDMS on NAVSIM-v2, surpassing the strongest end-to-end and world-action baselines by 1.6 and 1.3 EPDMS, and, without HUGSIM-specific fine-tuning, attains the best HD-Score of 0.4462 on the closed-loop HUGSIM benchmark under the same evaluation protocol. These results validate V-JEPA-native world-action modeling as a powerful and scalable paradigm for autonomous driving planning. Code is available at https://github.com/AFARI-Research/WA-JEPA.

Figures & tables

Figure 1: Comparison of V-JEPA, video-based world-action models, and our WA-JEPA. Our method bridges semantic representation learning and predictive world modeling through future-frame masking.

Figure 2: Overview of WA-JEPA. Stage 1 adapts V-JEPA 2 to multi-view driving videos by predicting future representations under full-future and patch-level masks. Stage 2 initializes from this checkpoint and jointly predicts future scene representations and ego actions with the Joint Future-Action Flow Predictor.

End-to-End Methods
Method	Backbone	NC $\uparrow$	DAC $\uparrow$	DDC $\uparrow$	TLC $\uparrow$	EP $\uparrow$	TTC $\uparrow$	LK $\uparrow$	HC $\uparrow$	EC $\uparrow$	EPDMS ${}^{*}\uparrow$	EPDMS $\uparrow$
TransFuser ( Chitta et al. 2022 )	RegNetY-3.2GF	96.9	89.9	97.8	99.7	87.1	95.4	92.7	98.3	87.2	76.7	–
ARTEMIS ( Feng et al. 2025 )	ResNet-34	98.3	95.1	98.6	99.8	81.5	97.4	96.5	98.3	–	83.1	–
Hydra-MDP++ ( Li et al. 2025b )	V2-99	98.8	97.8	99.1	100	84.0	95.3	70.1	–	96.8	84.1	–
DiffusionDrive ( Liao et al. 2025 )	ResNet-34	98.2	95.9	99.4	99.8	87.5	97.3	96.8	98.3	87.7	–	84.5
Drive-JEPA ( Wang et al. 2026a )	ResNet-34	98.8	97.4	99.0	99.8	83.5	98.0	96.2	98.1	85.6	85.4	–

Table 1: Comparison with state-of-the-art methods on NAVSIM-v2 navtest . Methods are grouped by modeling paradigm. Backbone lists the primary visual or vision-language backbone used by each method. † marks results using auxiliary simulator-derived supervision. EPDMS ∗ refers to scores obtained before correction of the human-reference penalty-filter aggregation, whereas EPDMS reports the corrected scores.

All 436 scenarios
	WA-JEPA	LTF	DrivoR	UniAD	VAD
NC	0.6856	0.4428	0.5217	0.6555	0.4117
DAC	0.9635	0.9275	0.9559	0.9320	0.9028
TTC	0.6120	0.3751	0.4620	0.5156	0.2798
Comf.	0.6620	0.9478	0.9390	0.6633	0.9534
PDMS	0.5717	0.3653	0.4475	0.4940	0.2831

Table 2: Zero-shot closed-loop results on HUGSIM. Values are on the

[0,1]

scale; higher is better, and the best result in each row is bold.

Figure 3: Target-referenced PCA visualization of future latent predictions from flow matching (FM) and direct regression (Reg.). For each method, a separate PCA basis is fitted to its EMA target representations and applied to both target and predicted latents. Each map represents two consecutive future frames.

Method	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
End-to-End Methods
TransFuser	97.7	92.8	92.8	100	79.2	84.0
DiffusionDrive	98.2	96.2	94.7	100	82.2	88.1
Drive-JEPA	98.7	96.2	95.5	100	82.9	89.0
VLA Methods
ReCogDrive	97.9	97.3	94.9	100	87.3	90.8

Table 3: Comparison on NAVSIM-v1 navtest .

Figure 4: Temporal representation preservation with flow matching (FM) and direct regression (Reg.). A lower directional-similarity collapse gap and a change-magnitude collapse closer to 1 indicate better preservation of the target temporal dynamics.

Encoder	EPDMS $\uparrow$
MAE	83.8
SigLIP2	83.1
DINOv3	83.8
V-JEPA 2	89.5

Table 4: Ablations on NAVSIM-v2 navtest . (a) Vision encoder selection. (b) Masking strategy in Stage 1. The first row represents the baseline that skips Stage 1 pre-training and directly uses the original pre-trained V-JEPA 2 for Stage 2 training. (c) Joint future–action predictor and future-prediction methods in Stage 2. FM and Reg. denote flow matching and direct regression, respectively. The first row denotes an action-only Stage 2 baseline initialized from the Stage 1 checkpoint and fine-tuned with aciton supervision, without joint scene-action modeling or future prediction.

Figure 5: Zero-shot closed-loop rollouts of WA-JEPA on HUGSIM. Rows correspond to the four source datasets and columns to three scenario types. Each pair shows the front camera with the projected 4 s plan (left) and the BEV view with the planned trajectory and detected objects (right). The green region marks the drivable corridor, the yellow–red curve the planned trajectory, the green box the ego vehicle, and orange boxes other agents.

Figure 6: Trajectory predictions on representative NAVSIM scenarios: (a) left and right turns, (b) fork and gateway navigation, and (c) stopping and straight driving.

Statistic	EPDMS
Number of seeds	10
Mean	91.7014
Standard deviation	0.0531
Standard error	0.0168
95% $t$ -confidence interval	$[91.6634,\,91.7393]$
Median	91.6960

Table 7: Seed-level EPDMS variability for the main WA-JEPA experiment over a fixed set of ten seeds.

World Models for Autonomous Driving

World models for autonomous driving predict how scenes and traffic evolve under candidate actions to support planning, control, and simulation.

See all 142 papers

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving
Jul 31, 2026Jiwei Yang, Zhengxian Chen, Chaosheng Huang +1End-to-End Autonomous Driving Joint-Embedding Predictive Architecture
School of Vehicle and Mobility, Tsinghua University · 2State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University
DriveVA: Video Action Models are Zero-Shot Drivers
Apr 5, 2026Mengmeng Liu, Diankun Zhang, Jiuming Liu +7Action-Conditioned Video Generation World Models
University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1
Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Jan 29, 2026Linhan Wang, Zichong Yang, Chen Bai +6End-to-End Autonomous Driving NavSim
Virginia Tech · Purdue University · XPENG Motors +1

Jul 31, 2026J/K move · Enter open · S save

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving

Jiwei Yang, Zhengxian Chen, Chaosheng Huang, Jun Li

School of Vehicle and Mobility, Tsinghua University · 2State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University

End-to-End Autonomous Driving Joint-Embedding Predictive Architecture Autonomous Driving World Models for Autonomous Driving Autonomous Driving Planning

Open paper AskPDF arXiv

Existing autonomous-driving world models typically perform dense prediction of future videos, occupancy states, BEV representations, or agent motion. We argue that planning need not reconstruct the complete future world, but only focus on scene features that affect future ego action. Based on this perspective, we propose Auto-JEPA, an action-oriented latent world model that learns continuous future driving intent through joint-embedding prediction. Given visual observations, egomotion history, and navigation commands, Auto-JEPA predicts an intent embedding aligned with the latent representation of the future ego trajectory. The predicted intent retrieves executable trajectories from a fixed trajectory memory, which are then ranked by a scene-conditioned candidate selection module. Auto-JEPA keeps the visual encoder frozen, requires no explicit perception annotations, and uses no learned trajectory generator. By optimizing only task-specific modules for trajectory representation, intent prediction, and candidate selection, Auto-JEPA achieves 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2. Semantic occlusion experiments show that masking dynamic-agent regions induces an average intent change 2.97x that of equal-area random masking. Moreover, occluding vehicles that affect future driving substantially changes the predicted intent and selected trajectory, whereas both remain essentially unchanged when non-influential vehicles are occluded. These results show that future-intent prediction encourages the model to focus on planning-relevant visual features and supports high-quality planning without dense future-world modeling.

Abstract

World Models for Autonomous Driving

Explore similar work

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving

DriveVA: Video Action Models are Zero-Shot Drivers

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving