End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.
Figures & tables
Figure 1: ScaleDrive learns transferable driving representations through self-supervised predictive pretraining on large-scale raw driving videos. The pretrained encoder is kept frozen and transferred across heterogeneous end-to-end planners and closed-loop driving environments, consistently improving downstream planning performance.
Method
Inputs
Enc. Frozen
NC
DAC
EP
TTC
Comf.
PDMS
Transfuser Chitta et al. (2022)
L&C
✗
97.7
92.8
79.2
92.8
100.0
84.0
HydraMDP Li et al. (2024a)
L&C
✗
98.3
96.0
78.7
94.6
100.0
86.5
DiffusionDrive Liao et al. (2025a)
L&C
✗
98.2
96.2
82.2
94.7
100.0
88.1
World4Drive † Zheng et al. (2025)
C
✗
97.4
94.3
79.9
92.8
100.0
85.1
Drive-JEPA † Wang et al. (2026b)
C
✗
98.7
96.2
82.9
95.5
100.0
89.0
ScaleDrive †
C
✓
99.0
96.3
83.0
96.1
100.0
89.1
Table 1: Planning performance on NAVSIMv1. “Enc. Frozen” indicates whether the visual encoder is kept frozen during downstream planner training. † denotes training without perception-task labels.
Method
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
EPDMS
DiffusionDrive Liao et al. (2025a)
98.2
95.9
99.4
99.8
87.5
97.3
96.8
98.3
87.7
84.5
DiffusionDriveV2 Zou et al. (2025)
97.7
96.6
99.2
99.8
88.9
97.2
96.0
97.8
91.0
87.5
SparseDriveV2 Sun et al. (2026)
98.1
98.1
99.6
99.8
91.1
97.3
96.9
98.2
78.4
90.1
Latent-WAM Wang et al. (2026a)
98.1
97.3
99.6
99.8
87.7
97.3
97.6
98.1
87.3
89.3
ScaleDrive
99.3
98.9
99.7
99.6
85.4
98.9
95.9
98.2
82.1
90.5
Table 2: Planning performance on NAVSIMv2. ScaleDrive achieves strong overall performance under the more comprehensive closed-loop evaluation metrics.
Method
Trajectory Modeling
Representation
NC
DAC
TTC
EP
PDMS
LTF
BEV
97.4
92.8
92.4
79.0
83.8
LTF(V-JEPA2)
Regression
w. V-JEPA2
97.8
91.7
92.4
78.8
83.2(-0.6)
LTF(ScaleDrive)
w. ScaleDrive
98.2
94.0
93.6
81.6
86.2 (+2.4)
DiffusionDrive
BEV
98.1
94.8
94.1
81.1
86.6
DiffusionDrive(V-JEPA2)
Diffusion
w. V-JEPA2
98.6
95.1
95.1
81.8
87.4(+0.8)
DiffusionDrive(ScaleDrive)
w. ScaleDrive
98.6
95.8
95.3
82.3
88.1 (+1.5)
Table 3: Transfer across end-to-end driving policies . Downstream training and evaluation settings are kept fixed within each policy.
Figure 2: (A) Data scaling performance of ScaleDrive. (B) Command-wise gains under the imbalanced training driving action distribution.
Route Completion ↑
HD-Score ↑
Planner
Representation
Easy
Medium
Hard
Extreme
Avg.
Easy
Medium
Hard
Extreme
Avg.
LTF
BEV
0.684
0.407
0.369
0.255
0.410
0.528
0.246
0.198
0.081
0.246
LTF
w. V-JEPA2
0.738
0.400
0.298
0.279
0.412
0.630
0.228
0.141
0.113
0.251
LTF
w. ScaleDrive
0.742
0.496
0.329
0.259
0.448
0.638
0.361
0.180
0.119
0.315
ChainFlow
DINOv2
0.834
0.433
0.258
0.273
0.430
0.705
0.313
0.116
0.122
0.296
ChainFlow
w. V-JEPA2
0.782
0.443
0.315
0.252
0.432
0.640
0.342
0.168
0.108
0.302
Table 4: Cross-benchmark generalization on HUGSIM.
Planner
Representation
Driving Score ↑
Success Rate ↑
Efficiency ↑
Comfort ↑
TFv6
BEV
95.20
86.80
-
-
TFv6
w. V-JEPA2
95.00
85.91
202.33
15.45
TFv6
w. ScaleDrive
96.65 (+1.45)
90.45 (+3.65)
210.22
18.03
Table 5: Cross-benchmark generalization on Bench2Drive.
Figure 3: Video representation probing. (A–B) Ego-motion estimation with ADE ↓ and FDE ↓ ; (C–D) monocular depth estimation with AbsRel ↓ and δ1↑ .
Figure 4: Qualitative result of representations. (A) t-SNE visualization. (B) PCA visualization of dense video features across representative driving scenes.
Figure 5: Planning results on NAVSIM, HUGSIM and Bench2Drive.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter
Value
Model
Backbone
ViT-Large with 3D RoPE
Resolution
256×256
Frames per clip
8
Patch size
16×16
Tubelet size
2
Predictor dimension
256
Appendix
Table 6: ScaleDrive video latent prediction pretraining configuration.
Category
Hyperparameter
Value
Model
Backbone
ViT-Large with 3D RoPE
Resolution
256×256
Front-view frames
4
Surround-view frames
1
Patch size
16×16
Tubelet size
2
Appendix
Table 7: ScaleDrive planning training configuration on NAVSIM.
Planner
Backbone
Decoder
Video representation integration
LTF
TransFuser
Regression
Attention keys and values
DiffusionDrive
TransFuser
Diffusion
Attention keys and values
DrivoR
DINOv2
Regression
Front-view feature replacement
ChainFlow
DINOv2
Autoregressive
Front-view feature replacement
Appendix
Table 8: Cross-planner configurations. V-JEPA2 and ScaleDrive use the same integration strategy within each planner.
Category
Hyperparameter
Value
Model
Backbone
ViT-Large with 3D RoPE
Resolution
256×256
Frames per clip
4
Patch size
16×16
Tubelet size
2
Planning
Planner architecture
TFv6
Appendix
Table 9: Training configuration on Bench2Drive.
Category
Hyperparameter
Depth Estimation
Ego-Motion Estimation
Model
Backbone
ViT-Large with 3D RoPE
ViT-Large with 3D RoPE
Resolution
256×256
256×256
Tubelet size
2
2
Encoder token grid
2×16×16
2×16×16
Probe
Prediction target
Depth map
Waypoints
Feature aggregation
Temporal mean pooling
3 learnable queries
Appendix
Table 10: Training configuration for ScaleDrive representation probing.
Pretraining Data
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
w/o. Pretrain
99.0
98.0
88.4
96.5
92.4
NAVSIM
98.6
98.1
90.4
95.2
92.7
OpenDV
99.3
98.4
90.6
97.0
93.8
Appendix
Table 11: Comparison with NAVSIM navtrain dataset pretraining.
Method
Pretraining Data
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
V-JEPA2
–
99.03
98.18
88.62
96.10
92.40
ScaleDrive
0.5M
98.99
97.92
89.65
96.12
92.77
1M
99.10
97.99
89.78
96.57
93.05
3M
99.14
98.10
90.32
96.51
93.30
6M
99.16
98.28
90.67
96.58
93.61
Appendix
Table 12: Detailed data-scaling results on NAVSIM.
Masking
PDMS ↑
EPDMS ↑
Ego ADE ↓
Ego FDE ↓
Patch only
93.61
89.96
0.193
0.284
Patch + Temporal-slice
93.79
90.48
0.172
0.246
Appendix
Table 13: Ablation on temporal-slice masking. All configurations share the same pretraining data and training settings. Ego ADE/FDE are measured with the frozen-encoder ego-motion probe.
Figure 6: Visual representation comparison.
Figure 7: ScaleDrive planning results.(A) NAVSIM and (B) HUGSIM.
End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.
Linhan Wang, Zichong Yang, Chen Bai +6
Virginia Tech · Purdue University · XPENG Motors +1
Pretrained foundation models have become an important basis for end-to-end autonomous driving. In contrast to vision-language models pretrained primarily on static image-text pairs, video generative models capture temporal dynamics and motion priors that are naturally suited for driving. We present DriveWAM, a driving world-action model that adapts a pretrained video diffusion transformer into an autoregressive video-action policy. DriveWAM organizes video and action streams into a unified temporal token sequence and trains them under a joint flow-matching objective, preserving the pretrained video-generation architecture while adapting its large-scale video priors to action generation. To incorporate high-level scene understanding, we introduce scene-evolving driving guidance, where a frozen VLM produces chunk-specific semantic intent to guide video-action generation. To keep long-horizon rollout bounded, we further introduce selective KV memory, which maintains bounded modality-aware video and action memory pools through relevance-redundancy cache selection at inference time. Experiments on NAVSIM and the PhysicalAI-Autonomous-Vehicles benchmark show that DriveWAM achieves strong planning performance, and a data-scaling study from 4k to 100k driving clips further confirms the scaling potential of world-action modeling for end-to-end autonomous driving.
Chen Shi, Jinrui Xu, Shaoshuai Shi +3
The Chinese University of Hong Kong, Shenzhen · Voyager Research, Didi Chuxing
Vision-Language-Action (VLA) models have emerged as a promising framework for end-to-end autonomous driving. However, existing VLAs typically rely on sparse action supervision, which underutilizes their powerful scene understanding and reasoning capabilities. Recent attempts to incorporate dense visual supervision via world modeling often overemphasize pixel-level image reconstruction, neglecting semantically meaningful scene representation learning. In this work, we propose LVDrive, a Latent Visual representation enhanced VLA framework for autonomous driving. LVDrive introduces a future scene prediction task into the VLA paradigm, where future representations are learned entirely in a high-level latent space under auxiliary supervision from a pretrained vision backbone. Departing from inefficient autoregressive generation, we jointly model future scene and motion prediction within a unified embedding space, processed in a single forward pass to conduct the future-aware reasoning. We further design a two-stage trajectory decoding strategy that explicitly leverages the learned latent future representations to refine trajectory generation. Extensive experiments on the challenging Bench2Drive benchmark demonstrate that LVDrive achieves significant improvements in closed-loop driving performance, outperforming both action supervised methods and image-reconstruction-based world model approaches.
Xiaodong Mei, Diankun Zhang, Hongwei Xie +3
The Hong Kong University of Science and Technology · Xiaomi EV