cs.CVFeb 25, 2026

Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos

Authors: Matthew Strong, Wei-Jer Chang, Quentin Herau, Jiezhi Yang, Yihan Hu, Chensheng Peng, Wei Zhan

Organizations: Applied Intuition · Stanford University · UC Berkeley

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Ego-centric driving videos available online provide an abundant source of visual data for autonomous driving, yet their lack of annotations makes it difficult to learn representations that capture both semantic structure and 3D geometry. Recent advances in large feedforward spatial models demonstrate that point maps and ego-motion can be inferred in a single forward pass, suggesting a promising direction for scalable driving perception. We therefore propose a label-free, teacher-guided framework for learning autonomous driving representations directly from unposed videos. Unlike prior self-supervised approaches that focus primarily on frame-to-frame consistency, we posit that safe and reactive driving depends critically on temporal context. To this end, we leverage a feedforward architecture equipped with a lightweight autoregressive module, trained using multi-modal supervisory signals that guide the model to jointly predict current and future point maps, camera poses, semantic segmentation, and motion masks. Multi-modal teachers provide sequence-level pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation from raw YouTube videos without poses, labels, or LiDAR. The resulting encoder not only transfers effectively to downstream autonomous driving planning on the NAVSIM benchmark, surpassing multi-camera and LiDAR baselines with only a single monocular camera, but also yields strong performance when evaluated on a range of semantic, geometric, and qualitative motion prediction tasks. These geometry and motion-aware features position LFG as a compelling video-centric foundation model for autonomous driving.

Figures & tables

Figure 2: LFG architecture. Starting from unposed single-view driving clips, a pretrained π3\pi^{3} backbone encodes NN observed frames into latent scene tokens. A lightweight causal autoregressive transformer rolls out MM future tokens, which a shared decoder maps to point maps, camera poses, semantic segmentation, confidence maps, and motion masks for all N+MN{+}M frames. Multi-modal teachers provide pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation that transfers effectively to downstream planning.
Figure 3: π3\pi^{3} -to-LFG distillation. We transfer geometric knowledge from the pretrained π3\pi^{3} teacher to LFG by supervising point maps, confidence maps, and camera poses for all observed and future frames. While the teacher has access to the full sequence, the student sees only the first NN frames and must predict both current and future geometry, enabling LFG to learn temporally consistent scene structure and future ego-motion from partial observations.
Figure 4: Semantic distillation. A pretrained SegFormer teacher, trained on Cityscapes, provides soft semantic pseudo-labels for each frame. LFG predicts semantic maps for both observed and future frames using only the first MM inputs, learning temporally consistent scene semantics through teacher–student supervision aligned with the model’s geometric features.
Figure 5: Motion mask generation pipeline. We first detect human and vehicle instances in the first frame using Grounded SAM2, then track their 2D trajectories across time with CoTracker3. Using teacher π3\pi^{3} point maps, tracked pixels are backprojected into 3D and per-instance 3D displacements are measured over the sequence. Instances whose motion exceeds a threshold for at least Kmin⁡K_{\min} frames are labeled as dynamic, and their masks are rasterized into dense per-pixel motion masks Mt\mathbf{M}_{t} , which supervise the motion head.
Figure 6: Segmentation quality on current and future frames. We show results of segmentation on the 1st frame, as well as future frames. LFG decouples dynamic motion from its own movement.
MethodOverallPred.
 PAmIoUmDiceFWPAmIoUmDiceFW
Static baseline––––0.8880.4200.5020.810
SegFormer0.9260.6770.7440.7230.9260.6800.7470.725
MaskFormer0.9220.7600.8290.760––––
LFG0.9470.7680.8270.7700.9420.7510.8140.759
Table 1: Semantic segmentation metrics (overall vs. predicted).
DatasetMethodOverallPredicted
  AbsRelRMSEAbsRelRMSE
KITTI-360π3\pi^{3}0.26 ± 0.084.37 ± 0.650.26 ± 0.074.37 ± 0.66
LFG0.27 ± 0.074.38 ± 0.640.31 ± 0.114.38 ± 0.68
VGGT–4.46 ± 0.82–4.46 ± 0.82
DA3–4.43 ± 0.81–4.44 ± 0.81
Waymoπ3\pi^{3}0.19 ± 0.126.68 ± 3.100.19 ± 0.126.70 ± 3.13
Table 2: Depth estimation results for overall and predicted frames.
Figure 7: Qualitative comparison of full point cloud reconstructions of LFG vs. π3\pi^{3} . The current camera poses are in blue , and future poses in red . LFG point maps retain overall geometric quality, even on future frames, and the predicted camera motion remains precise. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens.
DatasetMethodATERotTrans
KITTI-360π3\pi^{3}0.431.320.31
LFG1.002.300.31
Waymoπ3\pi^{3}0.020.980.44
LFG0.081.000.44
Table 3: Trajectory estimation results. RelPos is split into rotation (deg) and translation (m).
Figure 8: Failure Case of Pseudo-GT on motion . Qualitative comparison of motion predictions (LFG vs Pseudo) with corresponding RGB frames. In this scene, the pseudo ground truth incorrectly predicts a moving car on the far left when it is parked. LFG correctly predicts the static parked car (left) and the dynamic vehicle in front of it.
MethodInput1%10%100% Data
DiffusionDrive3Cam+L64.972.688.1
DINOv31Cam60.075.881.4
PPGeo1Cam61.565.674.6
π3\pi^{3}1Cam56.277.582.8
LFG (Ours)1Cam66.381.485.2
Table 4: Data-efficiency comparison (PDMS↑) on NAVSIM. LFG’s pretrained encoder yields superior data efficiency, demonstrating strong performance in the low-data regime and outperforming other pretrained encoders across all label fractions.
MethodInputNCDACTTCC.EPPDMS
BEV Baselines
UniAD6Cam97.891.992.9100.078.883.4
TransFuser3Cam+L97.792.892.0100.079.284.0
Hydra-MDP3Cam+L96.994.094.0100.078.784.7
DiffusionDrive3Cam+L96.895.494.7100.082.088.1
LFG (Ours)1Cam ∗98.293.794.4100.079.185.2
Table 5: NAVSIM planning benchmark: single-camera LFG vs BEV-based baselines. Higher is better for all metrics.
MethodInput1%10%100%
DiffusionDrive-DINOv23Cam+L57.374.481.5
DiffusionDrive-DINOv21Cam57.573.079.7
LFG (Ours)1Cam66.381.485.2
Table 6: DiffusionDrive comparison on NAVSIM (PDMS ↑\uparrow ).
Setting1%10%100%
Original setting66.381.485.2
+ 2 ×\times pretraining data76.682.384.8
+ Longer prediction horizon80.584.484.8
- Seg, Motion64.877.184.6
- Autoregressive head66.377.784.2
Table 7: Component and scaling ablations on NAVSIM (PDMS ↑\uparrow ).
MethodInput1%10%100% Data
PPGeo1Cam61.565.674.6
PPGeo ∗1Cam59.870.076.4
LFG (Ours)1Cam66.381.485.2
Table A1: Comparing PPGeo with different pretraining data-source. PPGeo ∗ indicates that the model is pretrained on the same OpenDV dataset used by LFG.
Figure A1 : Qualitative comparison of semantic segmentation across RGB, LFG, and SegFormer for current frames 1 and 2 (with ground-truth input) and future frames 4 and 6 . Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens.
Figure A2 : Qualitative comparison of motion predictions (LFG vs Pseudo GT motion) with corresponding RGB frames. We show results on non-future frames to demonstrate the motion map precision on a few images. The ego vehicle is moving on a road with three nearby vehicles moving.
Figure A3 : Qualitative comparison of depth prediction for six frames (first three frames are blue , the future three frames are red . LFG is able to decouple static and dynamic objects as it continues along the road, and future work will improve the sharpness of the last frames’ predictions. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens.
Figure A4 : Additional qualitative comparison of semantic segmentation across RGB, LFG, and SegFormer for current frames 1 and 2 (with ground-truth input) and future frames 4 and 6 . Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens. LFG on current frames enjoys crisper predictions even than its teacher.
Figure A5 : Additional qualitative comparison of semantic segmentation across RGB, LFG, and SegFormer for current frames 1 and 2 (with ground-truth input) and future frames 4 and 6 . Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens. LFG retains accurate predictions of cars, road, buildings, sky, traffic lights and signs, and even a person.
Figure A6 : More qualitative comparison of motion predictions (LFG vs Pseudo) with corresponding RGB frames. In this scene, LFG predicts the moving car across the intersection, but also the close pedestrians, demonstrating that the pretrained point decoders of π3\pi^{3} improve the predictions.
Figure A7 : Qualitative comparison of depth prediction for six frames. LFG is able to decouple static and dynamic objects as it continues along the road, and future work will improve the sharpness of the last frames’ predictions. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens
Figure A8 : More qualitative comparison of depth prediction for six frames. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens
More in this topic

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 536 papers

Explore similar work

CardsList
Aug 31, 2026

Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.

Xin Zhou, Zongchuang Zhao, Zhibo Yang +13

Qwen Team, Huazhong University of Science and Technology

VLMs for Autonomous DrivingAutonomous Driving PerceptionAutonomous Driving Planning3D Scene Understanding
AskPDF
Aug 17, 2026

Mask What Matters: Saliency-Guided Video Self-Supervised Learning for Autonomous Driving

Video self-supervised learning through masked spatiotemporal prediction has emerged as a promising paradigm for learning feature representations from unlabeled data. However, existing methods typically rely on random masking, which indiscriminately removes regions irrespective of their semantic or temporal relevance. In ego-centric driving videos, this can weaken the pretext signal since safety-critical cues such as pedestrians, vehicles, lane boundaries, and dynamic interactions often occupy only a small portion of the frame, yet are central to downstream perception. We introduce V-JEPA4A, a domain-specialized variant of V-JEPA for autonomous driving that is pre-trained on publicly available driving videos with a novel saliency-driven masking policy. It accounts for semantically and temporally relevant context. The proposed policy preserves and predicts context according to semantic importance and temporal relevance, yielding more informative representation learning while retaining the efficiency of masked prediction. We evaluate the resulting encoders on four driving benchmarks spanning tracking, semantic segmentation, and depth estimation. The results demonstrate that V-JEPA4A reduces identity switches on BDD100k MOT by 25% over V-JEPA with random masking, achieves 73.2 mIoU on Cityscapes, and 3.75 RMSE on KITTI-2015 depth, while incurring only ~14% additional pre-training iteration overhead.

Christopher Lang, Alexander Braun, Abhinav Valada

Robert Bosch GmbH, Stuttgart, Germany · Robot Learning Lab, University of Freiburg, Freiburg, Germany

Video Representation LearningSelf-Supervised Visual Representation LearningAutonomous Driving PerceptionMasked Image Modeling
AskPDF
May 21, 2026

LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model

Vision-Language-Action (VLA) models have emerged as a promising framework for end-to-end autonomous driving. However, existing VLAs typically rely on sparse action supervision, which underutilizes their powerful scene understanding and reasoning capabilities. Recent attempts to incorporate dense visual supervision via world modeling often overemphasize pixel-level image reconstruction, neglecting semantically meaningful scene representation learning. In this work, we propose LVDrive, a Latent Visual representation enhanced VLA framework for autonomous driving. LVDrive introduces a future scene prediction task into the VLA paradigm, where future representations are learned entirely in a high-level latent space under auxiliary supervision from a pretrained vision backbone. Departing from inefficient autoregressive generation, we jointly model future scene and motion prediction within a unified embedding space, processed in a single forward pass to conduct the future-aware reasoning. We further design a two-stage trajectory decoding strategy that explicitly leverages the learned latent future representations to refine trajectory generation. Extensive experiments on the challenging Bench2Drive benchmark demonstrate that LVDrive achieves significant improvements in closed-loop driving performance, outperforming both action supervised methods and image-reconstruction-based world model approaches.

Xiaodong Mei, Diankun Zhang, Hongwei Xie +3

The Hong Kong University of Science and Technology · Xiaomi EV

VLMs for Autonomous DrivingWorld ModelsLatent Visual ReasoningVision-Language-Action ModelsWorld Models for Autonomous DrivingAutonomous Driving Planning
AskPDF