Video self-supervised learning through masked spatiotemporal prediction has emerged as a promising paradigm for learning feature representations from unlabeled data. However, existing methods typically rely on random masking, which indiscriminately removes regions irrespective of their semantic or temporal relevance. In ego-centric driving videos, this can weaken the pretext signal since safety-critical cues such as pedestrians, vehicles, lane boundaries, and dynamic interactions often occupy only a small portion of the frame, yet are central to downstream perception. We introduce V-JEPA4A, a domain-specialized variant of V-JEPA for autonomous driving that is pre-trained on publicly available driving videos with a novel saliency-driven masking policy. It accounts for semantically and temporally relevant context. The proposed policy preserves and predicts context according to semantic importance and temporal relevance, yielding more informative representation learning while retaining the efficiency of masked prediction. We evaluate the resulting encoders on four driving benchmarks spanning tracking, semantic segmentation, and depth estimation. The results demonstrate that V-JEPA4A reduces identity switches on BDD100k MOT by 25% over V-JEPA with random masking, achieves 73.2 mIoU on Cityscapes, and 3.75 RMSE on KITTI-2015 depth, while incurring only ~14% additional pre-training iteration overhead.
Figures & tables
Figure 1: Radar chart comparing V-JEPA4A with image- and video-based self-supervised baselines across five driving-relevant downstream evaluations. All axes report normalized scores relative to the best baseline, where the dashed reference ring denotes baseline parity. For lower-is-better metrics such as RMSE, IDS, and trajectory L2 error, scores are inverted before normalization, so larger values consistently indicate better performance.
Figure 2: Illustration of the V-JEPA4A training pipeline. Our saliency-guided mask generators first process a frame sequence to produce spatiotemporal masking tubes. The student encoder ES observes only the unmasked context tokens, while the target encoder ET processes the full (unmasked) clip to provide regression targets. ET can be operated in two modes: ( i ) EMA-updated , where its weights are a slow exponential moving average of ES (following the original V-JEPA [ 4 ] ), or ( ii ) frozen , where ET is initialized from a pre-trained checkpoint and kept fixed throughout training (following SALT [ 16 ] ). The lightweight predictor P maps student context embeddings to predictions of the masked target embeddings, and the combined LMSE+λLvar objective drives learning.
Figure 3: Mask overlays for different masking strategies: (a) Random, (b) Sobel, (c) Optical Flow, (d) Motion-Compensated Differencing ( MCD ). Red regions are held out for the student encoder during pre-training.
Method
Model
Patch size
Input resolution
Sem. seg. mIoU [%] ↑
Depth RMSE [m] ↓
MOT [IDS] ↓
DINOv2 [ 17 ]
ViT-B
1x14x14
224x224
66.5
5.31
10550
DINOv2 [ 17 ]
ViT-L
1x14x14
224x224
69.3
3.57
10283
DINOv3 [ 22 ]
ViT-L
1x16x16
512x512
72.0
4.10
9877
OpenCLIP [ 20 ]
ViT-B
1x14x14
224x224
54.7
5.15
14098
V-JEPA [ 4 ]
ViT-L
2x16x16
224x224
68.2
4.58
12002
VideoMAE [ 9 ]
ViT-B
1x14x14
518x518
42.1
8.01
14208
Table 1: Benchmark results for self-supervised pre-trained vision transformers. For depth estimation and semantic segmentation, the pretrained models were integrated as frozen vision encoders in a DPT model. For multi-object tracking, the ground truth detection boxes were used to pool feature representations for similarity-based matching across frames. Therefore, we report IDS to isolates association quality. Semantic segmentation was evaluated on Cityscapes [ 8 ] , depth prediction on KITTI-2015 [ 10 ] , and multi-object tracking on BDD100k [ 26 ] . All results were obtained on the respective validation sets.
Figure 4: Representative downstream-task visualizations over six-frame sequences. Each composite overlays semantic segmentation, monocular depth estimation, and multi-object tracking predictions on BDD100K scene b1c9c847-3bda4659 (top) and a nuScenes scene 0321 (bottom).
Method
NuScenes MQA
OmniDrive
ImpromptuVLA
Y/N Acc.
Cat. Count
Rel. dist. L1
Safe
Red light
Coll.
Driv. area
1s
2s
3s
4s
OPT-6.7B [ 15 ]
0.82
0.32
4.21
–
–
–
–
–
–
–
–
Omni-L (7B) [ 23 ]
–
–
–
72.1
59.2
34.3
49.1
–
–
–
–
Impromptu-VLA (7B) [ 7 ]
–
–
–
–
–
–
–
0.10
0.31
0.65
1.11
Qwen3-VJEPA (random)
0.71
0.29
4.07
64.7
31.7
25.2
58.6
0.11
0.38
0.73
1.30
Qwen3-VJEPA4A (MCD)
0.78
0.40
2.80
80.3
62.4
29.8
54.0
0.11
0.29
0.58
1.02
Table 2: Comparison of model performances across three driving-related visual question answer benchmarks. Bold = best, underline = second best within each column. The top section reports benchmark-specific reference results taken from the cited works. The controlled comparison is between the Qwen3-based models in the lower section, which only differ in the V-JEPA4A pre-training for the vision encoder.
BDD100k
Cityscapes
KITTI
Run
Configuration
IDS ↓
mIoU ↑
RMSE ↓
Masking strategy (ratio = 0.6)
1
MCD (baseline)
9134
64.6
4.52
2
Random
10047
48.4
6.90
3
Sobel
10358
48.2
6.14
4
Flow
9712
53.7
4.24
5
Sobel+Flow
9902
52.4
5.10
Table 3: Single-factor ablations. The baseline (Run 1, shaded) uses ViT-B, 518×518 , 8 frames, 1×14×14 patches, MCD masking at ratio 0.6, temporal PE, and an EMA-updated target encoder. Each group varies one factor.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Split
# Seqs
FPS
Resolution
Usage
train/val/test
KITTI [ 10 ]
28 / 0 / 11
170
10
1242×375
PT / FT
BDD100k [ 26 ]
70 / 10 / 20
1 100
30
1280×720
PT / FT
nuScenes [ 5 ]
700 / 150 / 150
1 000
12
1600×900
PT / FT
ImpromptuVLA [ 7 ] †
56 / 14 / 0
1.1
2
Variable
PT / FT
Cityscapes [ 8 ]
23 / 1 / 2
75
2.5
2048×1024
FT
Appendix
Table 4: Overview of the datasets used in pre-training ( PT ) and downstream fine-tuning ( FT ). FPS values correspond to the raw video frame rate; we subsample where necessary to obtain a uniform temporal resolution. The splits refer to labeled sequences, while the number of sequences counts all available video sequences, including those without labels.
Dataset
Clips
Frames/clip
FPS
Hours
BDD100K
58,610
16
2.0
130.24
nuScenes
25,042
16
2.0
55.65
KITTI
284
16
2.0
2.40
ImPromptU VLA †
15,715
16
2.0
34.92
† Excludes nuScenes and KITTI clips already listed above.
Appendix
Table 5: Pretraining data Hours are clip-hours computed as clips × frames per clip / FPS.
Group
Hyperparameter
Value
Shared ViT encoder architecture
Image size
518×518
Patch size
14
Architecture
ViT-B
Class token
No
Positional embedding
Multi-resolution RoPE (MRoPE)
Appendix
Table 6: V-JEPA4A encoder architecture and pre-training hyperparameters.
Mask comp. [ms]
Random
31.4
Sobel
59.7
Optical Flow
78.8
Mot.-Comp. Diff.
88.4
Sobel + Flow
93.5
Appendix
Table 7: Latency overhead of saliency-guided masking strategies relative to random masking (ViT-B/14, 518×518 , T=8 frames).
Task
Hyperparameter
Value
(A) Cityscapes semantic segmentation (DPT)
Num. classes
19 ( ignore_index =255 )
Loss
Cross-entropy
Num. frames / tubelet
1 / 1
Batch size
32
Learning rate η
10−6→2×10−4→10−5
Appendix
Table 8: Downstream task hyperparameters. All tasks use the shared frozen V-JEPA4A encoder from Tab. 6 ; only task-specific heads and adapters are trained.
Figure 5: Qualitative comparison of the four saliency-guided mask generators. Each row shows one generator: (a–b) , (c–d) Optical flow, (e–f) Motion-Compensated Differencing ( MCD ), and (g–h) Combined Sobel + Optical Flow. The left column displays the resulting mask overlay on the input frame, while the right column shows the corresponding saliency heatmap.
Figure 6: Additional downstream-task visualizations over six-frame sequences. The appendix figure collects the remaining BDD100K scenes b1d10d08-ec660956 and b21c86ac-2eb7ba16, together with nuScenes scenes 0253, 0397, and 0518. Each composite overlays semantic segmentation, monocular depth estimation, and multi-object tracking predictions.
Method
Model
MOTA ↑
HOTA ↑
IDS ↓
DINOv2 [ 17 ]
ViT-B
68.99
78.96
10550
V-JEPA4A (random)
ViT-B
71.96
80.86
10047
V-JEPA4A
ViT-B
79.76
92.42
9134
V-JEPA4A (random, DINOv3)
ViT-L
74.91
81.11
9906
V-JEPA4A (DINOv3 teacher)
ViT-L
80.59
93.16
9018
Appendix
Table 9: Extended multi-object tracking metrics on BDD100k [ 26 ] . All methods use ground-truth detection boxes to isolate re-identification quality, consistent with IDS metric in the main paper. MOTA and HOTA are compound metrics that additionally incorporate detection performance.
Method
Model
Patch size
Input resolution
Sem. seg. mIoU [%] ↑
Depth RMSE [m] ↓
MOT IDS ↓
DINOv2 [ 17 ]
ViT-L
1×14×14
224×224
69.3
3.57
10283
DINOv3 [ 22 ]
ViT-L
1×16×16
512×512
72.0
4.10
9877
V-JEPA4A (random)
ViT-B
1×14×14
518×518
48.4
6.90
10047
V-JEPA4A
ViT-B
1×14×14
518×518
64.6
4.52
9134
V-JEPA4A (random, DINOv3)
ViT-L
1×16×16
512×512
51.9
4.24
9906
V-JEPA4A (DINOv3)
ViT-L
1×16×16
512×512
73.2
3.75
9018
Appendix
Table 10: Benchmark results for self-supervised pre-trained vision transformers. For depth estimation and semantic segmentation, the pretrained models were integrated as frozen vision encoders in a DPT model. For multi-object tracking, the ground truth detection boxes were used to pool feature representations for similarity-based matching across frames. Therefore, we report IDS to isolates association quality. Semantic segmentation was evaluated on Cityscapes [ 8 ] , depth prediction on KITTI-2015 [ 10 ] , and multi-object tracking on BDD100k [ 26 ] . All results were obtained on the respective validation sets.
Recipe
NuScenes
OmniDrive
Impromptu VLA
Y/N Acc.
Cat. Acc.
Cat./Cnt. L1
Drive dec.
Act. plan.
Counterf. reas.
VRU det. Acc.
Behav. justif.
Act. plan. Acc.
Waypoint L1
DSF
0.54
0.62
0.34
0.15
0.30
0.15
0.18
0.55
0.30
0.80
+ TU
0.61
0.64
0.34
0.21
0.33
0.19
0.23
0.59
0.32
0.60
+ FTok
0.59
0.71
0.33
0.20
0.36
0.36
0.22
0.61
0.58
0.58
+ VIT
0.60
0.73
0.31
0.22
0.34
0.38
0.24
0.60
0.54
0.62
+ IIV
0.62
0.72
0.38
0.16
0.35
0.41
0.25
0.61
0.59
0.39
Appendix
Table 11: Ablation of Qwen3-VJEPA4A training recipes across VLM driving benchmarks. Bold = best, underline = second best within each column.
Figure 7: Qualitative comparison of a baseline VLM (random-masking backbone) and our saliency-guided V-JEPA4A backbone on a representative driving scene. Saliency-based pre-training produces more actionable, safety-critical scene descriptions: V-JEPA4A explicitly identifies the red traffic light, pedestrians at the crosswalk, and the cyclist in the ego lane, whereas the baseline gives a vague enumeration that omits traffic-light state and pedestrian right-of-way.
Figure 8: Qualitative trajectory prediction examples from Qwen3-VJEPA4A on NuScenes. Predicted trajectories are shown in red ; ground-truth future paths in green .
End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.
Linhan Wang, Zichong Yang, Chen Bai +6
Virginia Tech · Purdue University · XPENG Motors +1
Video Joint Embedding Predictive Architecture (V-JEPA) learns powerful spatiotemporal representations from video through self-supervised latent feature prediction. However, V-JEPA is built around random-mask completion and deterministic regression, making it fundamentally ill-suited for autonomous driving planning that demands future-directed prediction tightly coupled with action. To address this, we rethink the V-JEPA paradigm and present WA-JEPA, a V-JEPA-native world-action model designed for autonomous driving planning. Instead of random spatiotemporal masking, WA-JEPA employs hybrid future-masked pre-training, where the model infers future latents from observed context. Departing from deterministic regression, we recast future prediction as conditional flow matching over latent futures, which substantially improves the model's ability to generate plausible future latents for downstream planning. Finally, a joint future-action predictor is proposed to denoise future scene tokens and ego trajectories together in a unified spatiotemporal latent space, allowing action supervision to directly shape planning-relevant world representations. Pre-trained on nuPlan videos and fine-tuned on NAVSIM, WA-JEPA reaches 91.7 EPDMS on NAVSIM-v2, surpassing the strongest end-to-end and world-action baselines by 1.6 and 1.3 EPDMS, and, without HUGSIM-specific fine-tuning, attains the best HD-Score of 0.4462 on the closed-loop HUGSIM benchmark under the same evaluation protocol. These results validate V-JEPA-native world-action modeling as a powerful and scalable paradigm for autonomous driving planning. Code is available at https://github.com/AFARI-Research/WA-JEPA.
Xinlin Wang, Yujiao Xiang, Yuheng Zhou +11
Afari Intelligent Drive · University of Electronic Science and Technology of China · Southeast University +2
Ego-centric driving videos available online provide an abundant source of visual data for autonomous driving, yet their lack of annotations makes it difficult to learn representations that capture both semantic structure and 3D geometry. Recent advances in large feedforward spatial models demonstrate that point maps and ego-motion can be inferred in a single forward pass, suggesting a promising direction for scalable driving perception. We therefore propose a label-free, teacher-guided framework for learning autonomous driving representations directly from unposed videos. Unlike prior self-supervised approaches that focus primarily on frame-to-frame consistency, we posit that safe and reactive driving depends critically on temporal context. To this end, we leverage a feedforward architecture equipped with a lightweight autoregressive module, trained using multi-modal supervisory signals that guide the model to jointly predict current and future point maps, camera poses, semantic segmentation, and motion masks. Multi-modal teachers provide sequence-level pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation from raw YouTube videos without poses, labels, or LiDAR. The resulting encoder not only transfers effectively to downstream autonomous driving planning on the NAVSIM benchmark, surpassing multi-camera and LiDAR baselines with only a single monocular camera, but also yields strong performance when evaluated on a range of semantic, geometric, and qualitative motion prediction tasks. These geometry and motion-aware features position LFG as a compelling video-centric foundation model for autonomous driving.
Matthew Strong, Wei-Jer Chang, Quentin Herau +4
Applied Intuition · Stanford University · UC Berkeley