cs.CVNov 2, 2026

Scaling Action-Free Video Latent Prediction for Transferable Representations in End-to-End Autonomous Driving

Authors: Xingtai Gui, Xubin Zhou, Yucheng Zhou, Dongqian Guo, Jiahao Gong, Feiyang Tan, Jianbing Shen

Organizations: SKL-IOTSC, Department of AI, University of Macau · Harbin Institute of Technology · Afari Intelligent Drive

Abstract

End-to-end autonomous driving policies require effective visual representations that capture both spatial scene structure and temporal dynamics. Video latent prediction has emerged as an effective paradigm for learning spatiotemporal visual representations, yet it remains unclear whether scaling this objective with action-free driving videos can yield representations that benefit end-to-end planning. We study action-free video latent prediction as a scalable pretraining objective and introduce ScaleDrive, a driving-oriented framework pretrained exclusively on large-scale internet open-world driving videos, without action annotations or benchmark-specific data. We keep the pretrained video encoder frozen during downstream end-to-end planning training to isolate the transferability of representations learned from internet driving videos. Ego-motion and depth probes show that scaling action-free video pretraining progressively strengthens the temporal and geometric information encoded in the frozen representation. We systematically study the visual representation transfer across different end-to-end driving policies and closed-loop benchmarks, where ScaleDrive consistently improves downstream performance and demonstrates that scaling action-free video latent prediction yields representations that generalize across both planning policy architectures and evaluation scenarios and protocols.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

    Jan 29, 2026Linhan Wang, Zichong Yang, Chen Bai +6End-to-End Autonomous DrivingNavSim

  2. DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving

    May 27, 2026Chen Shi, Jinrui Xu, Shaoshuai Shi +3Video Diffusion ModelsEnd-to-End Autonomous Driving

  3. LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model

    May 21, 2026Xiaodong Mei, Diankun Zhang, Hongwei Xie +3VLMs for Autonomous DrivingWorld Models