cs.CVFeb 25, 2026
SaveLearning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos
Organizations: Applied Intuition · Stanford University · UC Berkeley
Abstract
Ego-centric driving videos available online provide an abundant source of visual data for autonomous driving, yet their lack of annotations makes it difficult to learn representations that capture both semantic structure and 3D geometry. Recent advances in large feedforward spatial models demonstrate that point maps and ego-motion can be inferred in a single forward pass, suggesting a promising direction for scalable driving perception. We therefore propose a label-free, teacher-guided framework for learning autonomous driving representations directly from unposed videos. Unlike prior self-supervised approaches that focus primarily on frame-to-frame consistency, we posit that safe and reactive driving depends critically on temporal context. To this end, we leverage a feedforward architecture equipped with a lightweight autoregressive module, trained using multi-modal supervisory signals that guide the model to jointly predict current and future point maps, camera poses, semantic segmentation, and motion masks. Multi-modal teachers provide sequence-level pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation from raw YouTube videos without poses, labels, or LiDAR. The resulting encoder not only transfers effectively to downstream autonomous driving planning on the NAVSIM benchmark, surpassing multi-camera and LiDAR baselines with only a single monocular camera, but also yields strong performance when evaluated on a range of semantic, geometric, and qualitative motion prediction tasks. These geometry and motion-aware features position LFG as a compelling video-centric foundation model for autonomous driving.
Figures & tables
Figure 2: LFG architecture. Starting from unposed single-view driving clips, a pretrained backbone encodes observed frames into latent scene tokens. A lightweight causal autoregressive transformer rolls out future tokens, which a shared decoder maps to point maps, camera poses, semantic segmentation, confidence maps, and motion masks for all frames. Multi-modal teachers provide pseudo-supervision, enabling LFG to learn a unified pseudo-4D representation that transfers effectively to downstream planning.
Figure 3: -to-LFG distillation. We transfer geometric knowledge from the pretrained teacher to LFG by supervising point maps, confidence maps, and camera poses for all observed and future frames. While the teacher has access to the full sequence, the student sees only the first frames and must predict both current and future geometry, enabling LFG to learn temporally consistent scene structure and future ego-motion from partial observations.
Figure 4: Semantic distillation. A pretrained SegFormer teacher, trained on Cityscapes, provides soft semantic pseudo-labels for each frame. LFG predicts semantic maps for both observed and future frames using only the first inputs, learning temporally consistent scene semantics through teacher–student supervision aligned with the model’s geometric features.
Figure 5: Motion mask generation pipeline. We first detect human and vehicle instances in the first frame using Grounded SAM2, then track their 2D trajectories across time with CoTracker3. Using teacher point maps, tracked pixels are backprojected into 3D and per-instance 3D displacements are measured over the sequence. Instances whose motion exceeds a threshold for at least frames are labeled as dynamic, and their masks are rasterized into dense per-pixel motion masks , which supervise the motion head.
Figure 6: Segmentation quality on current and future frames. We show results of segmentation on the 1st frame, as well as future frames. LFG decouples dynamic motion from its own movement.
| Method | Overall | Pred. | ||||||
|---|---|---|---|---|---|---|---|---|
| PA | mIoU | mDice | FW | PA | mIoU | mDice | FW | |
| Static baseline | – | – | – | – | 0.888 | 0.420 | 0.502 | 0.810 |
| SegFormer | 0.926 | 0.677 | 0.744 | 0.723 | 0.926 | 0.680 | 0.747 | 0.725 |
| MaskFormer | 0.922 | 0.760 | 0.829 | 0.760 | – | – | – | – |
| LFG | 0.947 | 0.768 | 0.827 | 0.770 | 0.942 | 0.751 | 0.814 | 0.759 |
Table 1: Semantic segmentation metrics (overall vs. predicted).
| Dataset | Method | Overall | Predicted | ||
|---|---|---|---|---|---|
| AbsRel | RMSE | AbsRel | RMSE | ||
| KITTI-360 | 0.26 ± 0.08 | 4.37 ± 0.65 | 0.26 ± 0.07 | 4.37 ± 0.66 | |
| LFG | 0.27 ± 0.07 | 4.38 ± 0.64 | 0.31 ± 0.11 | 4.38 ± 0.68 | |
| VGGT | – | 4.46 ± 0.82 | – | 4.46 ± 0.82 | |
| DA3 | – | 4.43 ± 0.81 | – | 4.44 ± 0.81 | |
| Waymo | 0.19 ± 0.12 | 6.68 ± 3.10 | 0.19 ± 0.12 | 6.70 ± 3.13 | |
Table 2: Depth estimation results for overall and predicted frames.
Figure 7: Qualitative comparison of full point cloud reconstructions of LFG vs. . The current camera poses are in blue , and future poses in red . LFG point maps retain overall geometric quality, even on future frames, and the predicted camera motion remains precise. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens.
| Dataset | Method | ATE | Rot | Trans |
|---|---|---|---|---|
| KITTI-360 | 0.43 | 1.32 | 0.31 | |
| LFG | 1.00 | 2.30 | 0.31 | |
| Waymo | 0.02 | 0.98 | 0.44 | |
| LFG | 0.08 | 1.00 | 0.44 |
Table 3: Trajectory estimation results. RelPos is split into rotation (deg) and translation (m).
Figure 8: Failure Case of Pseudo-GT on motion . Qualitative comparison of motion predictions (LFG vs Pseudo) with corresponding RGB frames. In this scene, the pseudo ground truth incorrectly predicts a moving car on the far left when it is parked. LFG correctly predicts the static parked car (left) and the dynamic vehicle in front of it.
| Method | Input | 1% | 10% | 100% Data |
|---|---|---|---|---|
| DiffusionDrive | 3Cam+L | 64.9 | 72.6 | 88.1 |
| DINOv3 | 1Cam | 60.0 | 75.8 | 81.4 |
| PPGeo | 1Cam | 61.5 | 65.6 | 74.6 |
| 1Cam | 56.2 | 77.5 | 82.8 | |
| LFG (Ours) | 1Cam | 66.3 | 81.4 | 85.2 |
Table 4: Data-efficiency comparison (PDMS↑) on NAVSIM. LFG’s pretrained encoder yields superior data efficiency, demonstrating strong performance in the low-data regime and outperforming other pretrained encoders across all label fractions.
| Method | Input | NC | DAC | TTC | C. | EP | PDMS |
|---|---|---|---|---|---|---|---|
| BEV Baselines | |||||||
| UniAD | 6Cam | 97.8 | 91.9 | 92.9 | 100.0 | 78.8 | 83.4 |
| TransFuser | 3Cam+L | 97.7 | 92.8 | 92.0 | 100.0 | 79.2 | 84.0 |
| Hydra-MDP | 3Cam+L | 96.9 | 94.0 | 94.0 | 100.0 | 78.7 | 84.7 |
| DiffusionDrive | 3Cam+L | 96.8 | 95.4 | 94.7 | 100.0 | 82.0 | 88.1 |
| LFG (Ours) | 1Cam ∗ | 98.2 | 93.7 | 94.4 | 100.0 | 79.1 | 85.2 |
Table 5: NAVSIM planning benchmark: single-camera LFG vs BEV-based baselines. Higher is better for all metrics.
| Method | Input | 1% | 10% | 100% |
|---|---|---|---|---|
| DiffusionDrive-DINOv2 | 3Cam+L | 57.3 | 74.4 | 81.5 |
| DiffusionDrive-DINOv2 | 1Cam | 57.5 | 73.0 | 79.7 |
| LFG (Ours) | 1Cam | 66.3 | 81.4 | 85.2 |
Table 6: DiffusionDrive comparison on NAVSIM (PDMS ).
| Setting | 1% | 10% | 100% |
|---|---|---|---|
| Original setting | 66.3 | 81.4 | 85.2 |
| + 2 pretraining data | 76.6 | 82.3 | 84.8 |
| + Longer prediction horizon | 80.5 | 84.4 | 84.8 |
| - Seg, Motion | 64.8 | 77.1 | 84.6 |
| - Autoregressive head | 66.3 | 77.7 | 84.2 |
Table 7: Component and scaling ablations on NAVSIM (PDMS ).
| Method | Input | 1% | 10% | 100% Data |
|---|---|---|---|---|
| PPGeo | 1Cam | 61.5 | 65.6 | 74.6 |
| PPGeo ∗ | 1Cam | 59.8 | 70.0 | 76.4 |
| LFG (Ours) | 1Cam | 66.3 | 81.4 | 85.2 |
Table A1: Comparing PPGeo with different pretraining data-source. PPGeo ∗ indicates that the model is pretrained on the same OpenDV dataset used by LFG.
Figure A1 : Qualitative comparison of semantic segmentation across RGB, LFG, and SegFormer for current frames 1 and 2 (with ground-truth input) and future frames 4 and 6 . Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens.
Figure A2 : Qualitative comparison of motion predictions (LFG vs Pseudo GT motion) with corresponding RGB frames. We show results on non-future frames to demonstrate the motion map precision on a few images. The ego vehicle is moving on a road with three nearby vehicles moving.
Figure A3 : Qualitative comparison of depth prediction for six frames (first three frames are blue , the future three frames are red . LFG is able to decouple static and dynamic objects as it continues along the road, and future work will improve the sharpness of the last frames’ predictions. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens.
Figure A4 : Additional qualitative comparison of semantic segmentation across RGB, LFG, and SegFormer for current frames 1 and 2 (with ground-truth input) and future frames 4 and 6 . Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens. LFG on current frames enjoys crisper predictions even than its teacher.
Figure A5 : Additional qualitative comparison of semantic segmentation across RGB, LFG, and SegFormer for current frames 1 and 2 (with ground-truth input) and future frames 4 and 6 . Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens. LFG retains accurate predictions of cars, road, buildings, sky, traffic lights and signs, and even a person.
Figure A6 : More qualitative comparison of motion predictions (LFG vs Pseudo) with corresponding RGB frames. In this scene, LFG predicts the moving car across the intersection, but also the close pedestrians, demonstrating that the pretrained point decoders of improve the predictions.
Figure A7 : Qualitative comparison of depth prediction for six frames. LFG is able to decouple static and dynamic objects as it continues along the road, and future work will improve the sharpness of the last frames’ predictions. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens
Figure A8 : More qualitative comparison of depth prediction for six frames. Dashed red outlines denote predicted frames with no ground-truth image input , produced solely from the model’s future tokens
More in this topic
Autonomous Driving
Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.
See all 536 papers