End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.
Figures & tables
Figure 1: ScaleDrive learns transferable driving representations through self-supervised predictive pretraining on large-scale raw driving videos. The pretrained encoder is kept frozen and transferred across heterogeneous end-to-end planners and closed-loop driving environments, consistently improving downstream planning performance.
Method
Inputs
Enc. Frozen
NC
DAC
EP
TTC
Comf.
PDMS
Transfuser Chitta et al. (2022)
L&C
✗
97.7
92.8
79.2
92.8
100.0
84.0
HydraMDP Li et al. (2024a)
L&C
✗
98.3
96.0
78.7
94.6
100.0
86.5
DiffusionDrive Liao et al. (2025a)
L&C
✗
98.2
96.2
82.2
94.7
100.0
88.1
World4Drive † Zheng et al. (2025)
C
✗
97.4
94.3
79.9
92.8
100.0
85.1
Drive-JEPA † Wang et al. (2026b)
C
✗
98.7
96.2
82.9
95.5
100.0
89.0
ScaleDrive †
C
✓
99.0
96.3
83.0
96.1
100.0
89.1
Table 1: Planning performance on NAVSIMv1. “Enc. Frozen” indicates whether the visual encoder is kept frozen during downstream planner training. † denotes training without perception-task labels.
Method
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
EPDMS
DiffusionDrive Liao et al. (2025a)
98.2
95.9
99.4
99.8
87.5
97.3
96.8
98.3
87.7
84.5
DiffusionDriveV2 Zou et al. (2025)
97.7
96.6
99.2
99.8
88.9
97.2
96.0
97.8
91.0
87.5
SparseDriveV2 Sun et al. (2026)
98.1
98.1
99.6
99.8
91.1
97.3
96.9
98.2
78.4
90.1
Latent-WAM Wang et al. (2026a)
98.1
97.3
99.6
99.8
87.7
97.3
97.6
98.1
87.3
89.3
ScaleDrive
99.3
98.9
99.7
99.6
85.4
98.9
95.9
98.2
82.1
90.5
Table 2: Planning performance on NAVSIMv2. ScaleDrive achieves strong overall performance under the more comprehensive closed-loop evaluation metrics.
Method
Trajectory Modeling
Representation
NC
DAC
TTC
EP
PDMS
LTF
BEV
97.4
92.8
92.4
79.0
83.8
LTF(V-JEPA2)
Regression
w. V-JEPA2
97.8
91.7
92.4
78.8
83.2(-0.6)
LTF(ScaleDrive)
w. ScaleDrive
98.2
94.0
93.6
81.6
86.2 (+2.4)
DiffusionDrive
BEV
98.1
94.8
94.1
81.1
86.6
DiffusionDrive(V-JEPA2)
Diffusion
w. V-JEPA2
98.6
95.1
95.1
81.8
87.4(+0.8)
DiffusionDrive(ScaleDrive)
w. ScaleDrive
98.6
95.8
95.3
82.3
88.1 (+1.5)
Table 3: Transfer across end-to-end driving policies . Downstream training and evaluation settings are kept fixed within each policy.
Figure 2: (A) Data scaling performance of ScaleDrive. (B) Command-wise gains under the imbalanced training driving action distribution.
Route Completion ↑
HD-Score ↑
Planner
Representation
Easy
Medium
Hard
Extreme
Avg.
Easy
Medium
Hard
Extreme
Avg.
LTF
BEV
0.684
0.407
0.369
0.255
0.410
0.528
0.246
0.198
0.081
0.246
LTF
w. V-JEPA2
0.738
0.400
0.298
0.279
0.412
0.630
0.228
0.141
0.113
0.251
LTF
w. ScaleDrive
0.742
0.496
0.329
0.259
0.448
0.638
0.361
0.180
0.119
0.315
ChainFlow
DINOv2
0.834
0.433
0.258
0.273
0.430
0.705
0.313
0.116
0.122
0.296
ChainFlow
w. V-JEPA2
0.782
0.443
0.315
0.252
0.432
0.640
0.342
0.168
0.108
0.302
Table 4: Cross-benchmark generalization on HUGSIM.
Planner
Representation
Driving Score ↑
Success Rate ↑
Efficiency ↑
Comfort ↑
TFv6
BEV
95.20
86.80
-
-
TFv6
w. V-JEPA2
95.00
85.91
202.33
15.45
TFv6
w. ScaleDrive
96.65 (+1.45)
90.45 (+3.65)
210.22
18.03
Table 5: Cross-benchmark generalization on Bench2Drive.
Figure 3: Video representation probing. (A–B) Ego-motion estimation with ADE ↓ and FDE ↓ ; (C–D) monocular depth estimation with AbsRel ↓ and δ1↑ .
Figure 4: Qualitative result of representations. (A) t-SNE visualization. (B) PCA visualization of dense video features across representative driving scenes.
Figure 5: Planning results on NAVSIM, HUGSIM and Bench2Drive.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter
Value
Model
Backbone
ViT-Large with 3D RoPE
Resolution
256×256
Frames per clip
8
Patch size
16×16
Tubelet size
2
Predictor dimension
256
Appendix
Table 6: ScaleDrive video latent prediction pretraining configuration.
Category
Hyperparameter
Value
Model
Backbone
ViT-Large with 3D RoPE
Resolution
256×256
Front-view frames
4
Surround-view frames
1
Patch size
16×16
Tubelet size
2
Appendix
Table 7: ScaleDrive planning training configuration on NAVSIM.
Planner
Backbone
Decoder
Video representation integration
LTF
TransFuser
Regression
Attention keys and values
DiffusionDrive
TransFuser
Diffusion
Attention keys and values
DrivoR
DINOv2
Regression
Front-view feature replacement
ChainFlow
DINOv2
Autoregressive
Front-view feature replacement
Appendix
Table 8: Cross-planner configurations. V-JEPA2 and ScaleDrive use the same integration strategy within each planner.
Category
Hyperparameter
Value
Model
Backbone
ViT-Large with 3D RoPE
Resolution
256×256
Frames per clip
4
Patch size
16×16
Tubelet size
2
Planning
Planner architecture
TFv6
Appendix
Table 9: Training configuration on Bench2Drive.
Category
Hyperparameter
Depth Estimation
Ego-Motion Estimation
Model
Backbone
ViT-Large with 3D RoPE
ViT-Large with 3D RoPE
Resolution
256×256
256×256
Tubelet size
2
2
Encoder token grid
2×16×16
2×16×16
Probe
Prediction target
Depth map
Waypoints
Feature aggregation
Temporal mean pooling
3 learnable queries
Appendix
Table 10: Training configuration for ScaleDrive representation probing.
Pretraining Data
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
w/o. Pretrain
99.0
98.0
88.4
96.5
92.4
NAVSIM
98.6
98.1
90.4
95.2
92.7
OpenDV
99.3
98.4
90.6
97.0
93.8
Appendix
Table 11: Comparison with NAVSIM navtrain dataset pretraining.
Method
Pretraining Data
NC ↑
DAC ↑
EP ↑
TTC ↑
PDMS ↑
V-JEPA2
–
99.03
98.18
88.62
96.10
92.40
ScaleDrive
0.5M
98.99
97.92
89.65
96.12
92.77
1M
99.10
97.99
89.78
96.57
93.05
3M
99.14
98.10
90.32
96.51
93.30
6M
99.16
98.28
90.67
96.58
93.61
Appendix
Table 12: Detailed data-scaling results on NAVSIM.
Masking
PDMS ↑
EPDMS ↑
Ego ADE ↓
Ego FDE ↓
Patch only
93.61
89.96
0.193
0.284
Patch + Temporal-slice
93.79
90.48
0.172
0.246
Appendix
Table 13: Ablation on temporal-slice masking. All configurations share the same pretraining data and training settings. Ego ADE/FDE are measured with the frozen-encoder ego-motion probe.
Figure 6: Visual representation comparison.
Figure 7: ScaleDrive planning results.(A) NAVSIM and (B) HUGSIM.