Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Driving

Christopher Lang; Alexander Braun; Abhinav Valada

cs.CVAug 17, 2026

Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Driving

Authors: Christopher Lang, Alexander Braun, Abhinav Valada

Organizations: Robert Bosch GmbH, Stuttgart, Germany · Robot Learning Lab, University of Freiburg, Freiburg, Germany

Abstract

Video self-supervised learning through masked spatiotemporal prediction has emerged as a promising paradigm for learning feature representations from unlabeled data. However, existing methods typically rely on random masking, which indiscriminately removes regions irrespective of their semantic or temporal relevance. In ego-centric driving videos, this can weaken the pretext signal since safety-critical cues such as pedestrians, vehicles, lane boundaries, and dynamic interactions often occupy only a small portion of the frame, yet are central to downstream perception. We introduce V-JEPA4A, a domain-specialized variant of V-JEPA for autonomous driving that is pre-trained on publicly available driving videos with a novel saliency-driven masking policy. It accounts for semantically and temporally relevant context. The proposed policy preserves and predicts context according to semantic importance and temporal relevance, yielding more informative representation learning while retaining the efficiency of masked prediction. We evaluate the resulting encoders on four driving benchmarks spanning tracking, semantic segmentation, and depth estimation. The results demonstrate that V-JEPA4A reduces identity switches on BDD100k MOT by 25% over V-JEPA with random masking, achieves 73.2 mIoU on Cityscapes, and 3.75 RMSE on KITTI-2015 depth, while incurring only ~14% additional pre-training iteration overhead.

Figures & tables

Figure 1: Radar chart comparing V-JEPA4A with image- and video-based self-supervised baselines across five driving-relevant downstream evaluations. All axes report normalized scores relative to the best baseline, where the dashed reference ring denotes baseline parity. For lower-is-better metrics such as RMSE, IDS, and trajectory L2 error, scores are inverted before normalization, so larger values consistently indicate better performance.

Figure 2: Illustration of the V-JEPA4A training pipeline. Our saliency-guided mask generators first process a frame sequence to produce spatiotemporal masking tubes. The student encoder $E_{S}$ observes only the unmasked context tokens, while the target encoder $E_{T}$ processes the full (unmasked) clip to provide regression targets. $E_{T}$ can be operated in two modes: ( i ) EMA-updated , where its weights are a slow exponential moving average of $E_{S}$ (following the original V-JEPA [ 4 ] ), or ( ii ) frozen , where $E_{T}$ is initialized from a pre-trained checkpoint and kept fixed throughout training (following SALT [ 16 ] ). The lightweight predictor $P$ maps student context embeddings to predictions of the masked target embeddings, and the combined $\mathcal{L}_{\mathrm{MSE}}+\lambda\,\mathcal{L}_{\mathrm{var}}$ objective drives learning.

Panel of Figure 3 — Figure 3: Mask overlays for different masking strategies: (a) Random, (b) Sobel, (c) Optical Flow, (d) Motion-Compensated Differencing ( MCD ). Red regions are held out for the student encoder during pre-training.

Method	Model	Patch size	Input resolution	Sem. seg. mIoU [%] $\uparrow$	Depth RMSE [m] $\downarrow$	MOT [IDS] $\downarrow$
DINOv2 [ 17 ]	ViT-B	1x14x14	224x224	66.5	5.31	10550
DINOv2 [ 17 ]	ViT-L	1x14x14	224x224	69.3	3.57	10283
DINOv3 [ 22 ]	ViT-L	1x16x16	512x512	72.0	4.10	9877
OpenCLIP [ 20 ]	ViT-B	1x14x14	224x224	54.7	5.15	14098
V-JEPA [ 4 ]	ViT-L	2x16x16	224x224	68.2	4.58	12002
VideoMAE [ 9 ]	ViT-B	1x14x14	518x518	42.1	8.01	14208

Table 1: Benchmark results for self-supervised pre-trained vision transformers. For depth estimation and semantic segmentation, the pretrained models were integrated as frozen vision encoders in a DPT model. For multi-object tracking, the ground truth detection boxes were used to pool feature representations for similarity-based matching across frames. Therefore, we report

IDS

to isolates association quality. Semantic segmentation was evaluated on Cityscapes [ 8 ] , depth prediction on KITTI-2015 [ 10 ] , and multi-object tracking on BDD100k [ 26 ] . All results were obtained on the respective validation sets.

Panel of Figure 4 — Figure 4: Representative downstream-task visualizations over six-frame sequences. Each composite overlays semantic segmentation, monocular depth estimation, and multi-object tracking predictions on BDD100K scene b1c9c847-3bda4659 (top) and a nuScenes scene 0321 (bottom).

Method	NuScenes MQA			OmniDrive				ImpromptuVLA
	Y/N Acc.	Cat. Count	Rel. dist. L1	Safe	Red light	Coll.	Driv. area	1s	2s	3s	4s
OPT-6.7B [ 15 ]	0.82	0.32	4.21	–	–	–	–	–	–	–	–
Omni-L (7B) [ 23 ]	–	–	–	72.1	59.2	34.3	49.1	–	–	–	–
Impromptu-VLA (7B) [ 7 ]	–	–	–	–	–	–	–	0.10	0.31	0.65	1.11
Qwen3-VJEPA (random)	0.71	0.29	4.07	64.7	31.7	25.2	58.6	0.11	0.38	0.73	1.30
Qwen3-VJEPA4A (MCD)	0.78	0.40	2.80	80.3	62.4	29.8	54.0	0.11	0.29	0.58	1.02

Table 2: Comparison of model performances across three driving-related visual question answer benchmarks. Bold = best, underline = second best within each column. The top section reports benchmark-specific reference results taken from the cited works. The controlled comparison is between the Qwen3-based models in the lower section, which only differ in the V-JEPA4A pre-training for the vision encoder.

			BDD100k	Cityscapes	KITTI
	Run	Configuration	IDS $\downarrow$	mIoU $\uparrow$	RMSE $\downarrow$
Masking strategy (ratio = 0.6)	1	MCD (baseline)	9134	64.6	4.52
	2	Random	10047	48.4	6.90
	3	Sobel	10358	48.2	6.14
	4	Flow	9712	53.7	4.24
	5	Sobel+Flow	9902	52.4	5.10

Table 3: Single-factor ablations. The baseline (Run 1, shaded) uses ViT-B,

518{\times}518

, 8 frames,

1{\times}14{\times}14

patches, MCD masking at ratio 0.6, temporal PE, and an EMA-updated target encoder. Each group varies one factor.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Split	# Seqs	FPS	Resolution	Usage
Dataset	train/val/test
KITTI [ 10 ]	28 / 0 / 11	170	10	$1242\times 375$	PT / FT
BDD100k [ 26 ]	70 / 10 / 20	1 100	30	$1280\times 720$	PT / FT
nuScenes [ 5 ]	700 / 150 / 150	1 000	12	$1600\times 900$	PT / FT
ImpromptuVLA [ 7 ] †	56 / 14 / 0	1.1	2	Variable	PT / FT
Cityscapes [ 8 ]	23 / 1 / 2	75	2.5	$2048\times 1024$	FT

Appendix

Table 4: Overview of the datasets used in pre-training ( PT ) and downstream fine-tuning ( FT ). FPS values correspond to the raw video frame rate; we subsample where necessary to obtain a uniform temporal resolution. The splits refer to labeled sequences, while the number of sequences counts all available video sequences, including those without labels.

Dataset	Clips	Frames/clip	FPS	Hours
BDD100K	58,610	16	2.0	130.24
nuScenes	25,042	16	2.0	55.65
KITTI	284	16	2.0	2.40
ImPromptU VLA †	15,715	16	2.0	34.92
† Excludes nuScenes and KITTI clips already listed above.

Appendix

Table 5: Pretraining data Hours are clip-hours computed as clips

\times

frames per clip / FPS.

Group	Hyperparameter	Value
Shared ViT encoder architecture
	Image size	$518{\times}518$
	Patch size	$14$
	Architecture	ViT-B
	Class token	No
	Positional embedding	Multi-resolution RoPE (MRoPE)

Appendix

Table 6: V-JEPA4A encoder architecture and pre-training hyperparameters.

	Mask comp. [ms]
Random	31.4
Sobel	59.7
Optical Flow	78.8
Mot.-Comp. Diff.	88.4
Sobel $+$ Flow	93.5

Appendix

Table 7: Latency overhead of saliency-guided masking strategies relative to random masking (ViT-B/14,

518{\times}518

T{=}8

frames).

Task	Hyperparameter	Value
(A) Cityscapes semantic segmentation (DPT)
	Num. classes	$19$ ( ignore_index $=255$ )
	Loss	Cross-entropy
	Num. frames / tubelet	$1$ / $1$
	Batch size	$32$
	Learning rate $\eta$	$10^{-6}\!\to\!2{\times}10^{-4}\!\to\!10^{-5}$

Appendix

Table 8: Downstream task hyperparameters. All tasks use the shared frozen V-JEPA4A encoder from Tab. 6 ; only task-specific heads and adapters are trained.

Panel of Figure 5 — Figure 5: Qualitative comparison of the four saliency-guided mask generators. Each row shows one generator: (a–b) , (c–d) Optical flow, (e–f) Motion-Compensated Differencing ( MCD ), and (g–h) Combined Sobel + Optical Flow. The left column displays the resulting mask overlay on the input frame, while the right column shows the corresponding saliency heatmap.

Panel of Figure 6 — Figure 6: Additional downstream-task visualizations over six-frame sequences. The appendix figure collects the remaining BDD100K scenes b1d10d08-ec660956 and b21c86ac-2eb7ba16, together with nuScenes scenes 0253, 0397, and 0518. Each composite overlays semantic segmentation, monocular depth estimation, and multi-object tracking predictions.

Method	Model	MOTA $\uparrow$	HOTA $\uparrow$	IDS $\downarrow$
DINOv2 [ 17 ]	ViT-B	68.99	78.96	10550
V-JEPA4A (random)	ViT-B	71.96	80.86	10047
V-JEPA4A	ViT-B	79.76	92.42	9134
V-JEPA4A (random, DINOv3)	ViT-L	74.91	81.11	9906
V-JEPA4A (DINOv3 teacher)	ViT-L	80.59	93.16	9018

Appendix

Table 9: Extended multi-object tracking metrics on BDD100k [ 26 ] . All methods use ground-truth detection boxes to isolate re-identification quality, consistent with IDS metric in the main paper. MOTA and HOTA are compound metrics that additionally incorporate detection performance.

Method	Model	Patch size	Input resolution	Sem. seg. mIoU [%] $\uparrow$	Depth RMSE [m] $\downarrow$	MOT IDS $\downarrow$
DINOv2 [ 17 ]	ViT-L	$1{\times}14{\times}14$	$224{\times}224$	69.3	3.57	10283
DINOv3 [ 22 ]	ViT-L	$1{\times}16{\times}16$	$512{\times}512$	72.0	4.10	9877
V-JEPA4A (random)	ViT-B	$1{\times}14{\times}14$	$518{\times}518$	48.4	6.90	10047
V-JEPA4A	ViT-B	$1{\times}14{\times}14$	$518{\times}518$	64.6	4.52	9134
V-JEPA4A (random, DINOv3)	ViT-L	$1{\times}16{\times}16$	$512{\times}512$	51.9	4.24	9906
V-JEPA4A (DINOv3)	ViT-L	$1{\times}16{\times}16$	$512{\times}512$	73.2	3.75	9018

Appendix

Table 10: Benchmark results for self-supervised pre-trained vision transformers. For depth estimation and semantic segmentation, the pretrained models were integrated as frozen vision encoders in a DPT model. For multi-object tracking, the ground truth detection boxes were used to pool feature representations for similarity-based matching across frames. Therefore, we report

IDS

Recipe	NuScenes			OmniDrive			Impromptu VLA
	Y/N Acc.	Cat. Acc.	Cat./Cnt. L1	Drive dec.	Act. plan.	Counterf. reas.	VRU det. Acc.	Behav. justif.	Act. plan. Acc.	Waypoint L1
DSF	0.54	0.62	0.34	0.15	0.30	0.15	0.18	0.55	0.30	0.80
+ TU	0.61	0.64	0.34	0.21	0.33	0.19	0.23	0.59	0.32	0.60
+ FTok	0.59	0.71	0.33	0.20	0.36	0.36	0.22	0.61	0.58	0.58
+ VIT	0.60	0.73	0.31	0.22	0.34	0.38	0.24	0.60	0.54	0.62
+ IIV	0.62	0.72	0.38	0.16	0.35	0.41	0.25	0.61	0.59	0.39

Appendix

Table 11: Ablation of Qwen3-VJEPA4A training recipes across VLM driving benchmarks. Bold = best, underline = second best within each column.

Panel of Figure 7 — Figure 7: Qualitative comparison of a baseline VLM (random-masking backbone) and our saliency-guided V-JEPA4A backbone on a representative driving scene. Saliency-based pre-training produces more actionable, safety-critical scene descriptions: V-JEPA4A explicitly identifies the red traffic light, pedestrians at the crosswalk, and the cyclist in the ego lane, whereas the baseline gives a vague enumeration that omits traffic-light state and pedestrian right-of-way.

Panel of Figure 8 — Figure 8: Qualitative trajectory prediction examples from Qwen3-VJEPA4A on NuScenes. Predicted trajectories are shown in red ; ground-truth future paths in green .

Video Representation Learning

Video representation learning develops features that capture appearance, motion, and temporal structure for recognition, prediction, and control.

See all 143 papers

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Jan 29, 2026Linhan Wang, Zichong Yang, Chen Bai +6End-to-End Autonomous Driving NavSim
Virginia Tech · Purdue University · XPENG Motors +1
WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving
Aug 21, 2026Xinlin Wang, Yujiao Xiang, Yuheng Zhou +11World Models for Autonomous Driving Autonomous Driving Planning
Afari Intelligent Drive · University of Electronic Science and Technology of China · Southeast University +2
Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos
Feb 25, 2026Matthew Strong, Wei-Jer Chang, Quentin Herau +4Autonomous Driving Autonomous Driving Perception
Applied Intuition · Stanford University · UC Berkeley

Jan 29, 2026J/K move · Enter open · S save

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang+5

Virginia Tech · Purdue University · XPENG Motors+1

End-to-End Autonomous Driving NavSim Joint-Embedding Predictive Architecture Video Representation Learning Autonomous Driving Planning

Open paper AskPDF arXiv

End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.

Abstract

Video Representation Learning

Explore similar work

Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving

Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos