VLD: Visual Learned Distance as an Interface for Reinforcement Learning Navigation
Organizations: ETH Zurich · EPFL · Stanford University · UC Berkeley
Abstract
Training end-to-end policies from image data to directly predict navigation actions for robotic systems has proven inherently difficult. Existing approaches often suffer from either the sim-to-real gap during policy transfer or limited action-labeled training data. To address this problem, we introduce Visual Learned Distance (VLD), a scalable framework for image-goal navigation that decouples perception learning from policy learning. We first train a self-supervised distance-to-goal predictor on about 3,000 hours of internet-scale and simulated video, providing a distance signal that a reinforcement learning (RL) policy can minimize. The RL policy is trained separately in simulation using privileged geometric distances perturbed by GeoNoise, a learned noise model that reproduces the viewpoint- and overlap-dependent errors of the trained distance predictor. At deployment, the policy consumes VLD predictions zero-shot, without retraining, inheriting ``where to go'' from large-scale visual training while retaining the robust navigation behaviors learned in simulation. This error modeling is key to the transfer: with VLD inputs, GeoNoise-trained policies reach success on HM3D and on Gibson, whereas policies trained with generic temporally correlated noise collapse to and . On a physical robot, the VLD-driven policy succeeds in of trials, compared with for a strong end-to-end image-goal baseline that outperforms it in simulation. Our decoupled design thus provides an alternative and scalable path toward reliable navigation policies that transfer from simulation to the real world.
Figures & tables
| Model | Habitat | Real-world | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| HM3D | Gibson | In-the-wild | Embodiment | |||||||
| 20 | 50 | 100 | 20 | 50 | 100 | 50 | 100 | 50 | 100 | |
| ViNT | 0.40 | 0.35 | 0.28 | 0.53 | 0.41 | 0.39 | 0.35 | 0.25 | 0.41 | 0.32 |
| ViNT-Tuned | 0.67 | 0.56 | 0.48 | 0.78 | 0.65 | 0.64 | 0.40 | 0.29 | 0.48 | 0.37 |
| VIP | 0.29 | 0.19 | 0.17 | 0.39 | 0.30 | 0.29 | 0.28 | 0.20 | 0.43 | 0.33 |
| VIP-Nav | 0.55 | 0.45 | 0.38 | 0.65 | 0.51 | 0.50 | 0.32 | 0.23 | 0.46 | 0.39 |
| Privileged input | Learned predictor | ||||||||
| HM3D | Gibson | HM3D | Gibson | ||||||
| Training signal | Predictor | SR | SPL | SR | SPL | SR | SPL | SR | SPL |
| GT distance (no noise) | – | 94.7 | 0.582 | 95.8 | 0.610 | – | – | – | – |
| GeoNoise (ours) | VLD | 90.2 | 0.512 | 92.2 | 0.597 | 58.7 | 0.283 | 76.2 | 0.422 |
| GeoNoise + conf. | VLD | 88.9 | 0.499 | 89.9 | 0.581 | 52.1 | 0.225 | 68.2 | 0.386 |
| OU noise | VLD | 94.4 | 0.543 | 94.5 | 0.646 | 4.9 | 0.011 | 11.5 | 0.040 |
| Method | Overall | |||||
|---|---|---|---|---|---|---|
| VLD + GeoNoise (ours) | 5/5 | 4/5 | 4/5 | 4/5 | 5/5 | 22/25 (88%) |
| FGPrompt-EF [ 48 ] | 0/5 | 2/5 | 3/5 | 5/5 | 5/5 | 15/25 (60%) |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Avg. Velocity | Total Hrs | Type |
|---|---|---|---|
| HM3D | m/s | 1150h | Synthetic |
| YouTube | m/s | 1720h | In-the-wild |
| Ego4D | – m/s | 50h | In-the-wild |
| SCAND | – m/s | 9h | Embodiment |
| Grand Tour | m/s | 5h | Embodiment |
| CityWalker | m/s | 15h | Embodiment |
| VLD Model | |
|---|---|
| Total # Parameters (approx.) | 33M |
| Trainable # Parameters | 11M |
| Image Encoder | DINOv2-Small |
| Image Input Res. | |
| Image Token Dimension | |
| Attn. Hidden Dim. | |
| Training | |
|---|---|
| # Steps | 2M |
| Batch Size | 64 |
| Learning Rate | |
| Optimizer | AdamW |
| LR Schedule | Cosine |
| Warmup Steps | 10K |
| Model | HM3D (↑) | Gibson (↑) | ||||
|---|---|---|---|---|---|---|
| 20 | 50 | 100 | 20 | 50 | 100 | |
| VLD (single) | 0.82 | 0.70 | 0.61 | 0.84 | 0.73 | 0.71 |
| VLD (multi) | 0.82 | 0.71 | 0.63 | 0.86 | 0.75 | 0.73 |
| Objective | HM3D (↑) | Gibson (↑) | ||||
|---|---|---|---|---|---|---|
| 20 | 50 | 100 | 20 | 50 | 100 | |
| VLD (NLL) | 0.82 | 0.70 | 0.61 | 0.84 | 0.73 | 0.71 |
| VLD (MSE) | 0.71 | 0.67 | 0.61 | 0.78 | 0.70 | 0.68 |
| VLD (QRL) | 0.17 | 0.20 | 0.16 | 0.06 | 0.07 | 0.07 |
| VLD (VIP) | 0.73 | 0.53 | 0.43 | 0.79 | 0.59 | 0.56 |
| Category | Accuracy (%, ↑) |
|---|---|
| In–In (both in goal room) | 58.61 |
| Out–In (one in goal room) | 65.48 |
| Out–Out (neither in goal room) | 53.32 |
| Failure Type | HM3D (%) | Gibson (%) |
|---|---|---|
| Time-out failure | 71 | 62 |
| Near-match failure | 28 | 29 |
| Bad or uninformative goal | 1 | 9 |
| Method | G1 | G2 | G3 | G4 | G5 | Overall |
|---|---|---|---|---|---|---|
| Success rate (SR ) | ||||||
| VLD + GeoNoise | 1.00 | 0.80 | 0.80 | 0.80 | 1.00 | 0.88 |
| FGPrompt-EF | 0.00 | 0.40 | 0.60 | 1.00 | 1.00 | 0.60 |
| Average steps over successful trials | ||||||
| VLD + GeoNoise | 58.6 | 138.4 | 298.6 | 342.6 | 80.8 | – |
| FGPrompt-EF | – | 326.0 | 271.6 | 75.4 | 106.4 | – |