Training end-to-end policies from image data to directly predict navigation actions for robotic systems has proven inherently difficult. Existing approaches often suffer from either the sim-to-real gap during policy transfer or limited action-labeled training data. To address this problem, we introduce Visual Learned Distance (VLD), a scalable framework for image-goal navigation that decouples perception learning from policy learning. We first train a self-supervised distance-to-goal predictor on about 3,000 hours of internet-scale and simulated video, providing a distance signal that a reinforcement learning (RL) policy can minimize. The RL policy is trained separately in simulation using privileged geometric distances perturbed by GeoNoise, a learned noise model that reproduces the viewpoint- and overlap-dependent errors of the trained distance predictor. At deployment, the policy consumes VLD predictions zero-shot, without retraining, inheriting ``where to go'' from large-scale visual training while retaining the robust navigation behaviors learned in simulation. This error modeling is key to the transfer: with VLD inputs, GeoNoise-trained policies reach 58.7% success on HM3D and 76.2% on Gibson, whereas policies trained with generic temporally correlated noise collapse to 4.9% and 11.5%. On a physical robot, the VLD-driven policy succeeds in 88% of trials, compared with 60% for a strong end-to-end image-goal baseline that outperforms it in simulation. Our decoupled design thus provides an alternative and scalable path toward reliable navigation policies that transfer from simulation to the real world.
Figures & tables
Figure 1: Overview of our framework. (A) Training: we separately train a temporal Visual Learned Distance (VLD) function on diverse real-world and synthetic video, and an RL navigation policy in simulation that receives the privileged distance-to-goal perturbed by GeoNoise, a learned model of VLD’s errors. (B) Deployment: the trained policy consumes VLD predictions in place of the noisy privileged distance, without retraining, to navigate in simulated and real-world environments.
Figure 2: Visual Learned Distance (VLD) architecture. A shared frozen DINOv2 encoder turns the current view into query tokens and the goal image into key/value memory tokens. A Transformer decoder attends from the observation queries to the goal tokens, and MLP heads on the CLS output predict the temporal distance t^ and a confidence score c^ .
Model
Habitat
Real-world
HM3D
Gibson
In-the-wild
Embodiment
20
50
100
20
50
100
50
100
50
100
ViNT
0.40
0.35
0.28
0.53
0.41
0.39
0.35
0.25
0.41
0.32
ViNT-Tuned
0.67
0.56
0.48
0.78
0.65
0.64
0.40
0.29
0.48
0.37
VIP
0.29
0.19
0.17
0.39
0.30
0.29
0.28
0.20
0.43
0.33
VIP-Nav
0.55
0.45
0.38
0.65
0.51
0.50
0.32
0.23
0.46
0.39
Table 1: Ordinal consistency (Kendall’s τ , ↑ ) on held-out Habitat and real-world trajectories at horizons of 20, 50, and 100 steps. VLD is trained on synthetic data, real-world data, or all sources.
Figure 3: Goal cues and distance predictions. Each observation is evaluated independently against the final-frame goal image. Left: normalized predicted distances and Kendall’s τ ; middle: VLD confidence; right: the goal image; below: observed frames. Red annotations link emerging goal cues to increases in confidence. Confidence is auxiliary; only distance supplies goal information to the main navigation policy.
Privileged input
Learned predictor
HM3D
Gibson
HM3D
Gibson
Training signal
Predictor
SR
SPL
SR
SPL
SR
SPL
SR
SPL
GT distance (no noise)
–
94.7
0.582
95.8
0.610
–
–
–
–
GeoNoise (ours)
VLD
90.2
0.512
92.2
0.597
58.7
0.283
76.2
0.422
GeoNoise + conf.
VLD
88.9
0.499
89.9
0.581
52.1
0.225
68.2
0.386
OU noise
VLD
94.4
0.543
94.5
0.646
4.9
0.011
11.5
0.040
Table 2: Navigation in simulation. Left: each policy evaluated with the privileged distance signal it is trained with. Right: image-goal navigation, where the distance input of the same policy is replaced zero-shot by the listed learned predictor with fixed policy weights; the policy in the last distance-based row is trained directly on VLD, and FGPrompt-EF processes the goal image end-to-end. “+ conf.” denotes policies that also receive a confidence input. SR in %, SPL in [0,1] .
Figure 4: Noise signals on one trajectory. GeoNoise (green) follows VLD’s (orange) broad scale and trend; OU-noised distances (red) differ. GT (blue) is temporal ground truth; additional examples are in Appendix L .
Method
S→G1
G1→G2
G2→G3
G3→G4
G4→G5
Overall
VLD + GeoNoise (ours)
5/5
4/5
4/5
4/5
5/5
22/25 (88%)
FGPrompt-EF [ 48 ]
0/5
2/5
3/5
5/5
5/5
15/25 (60%)
Table 3: Real-world navigation on a TurtleBot4-Light: successful trials out of five per task (25 per method). Tasks are executed in sequence from the start position S . Action counts are reported in Appendix N .
Figure 5: Real-world trajectory comparison. Both policies observe the target object (a printer) in one of their views. The VLD-driven policy (top, blue) immediately moves toward the goal as the predicted distance drops sharply, while FGPrompt-EF (bottom, green) initially moves in the opposite direction.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Negative mining teaches the model to separate unrelated scenes. Given two observations from completely different environments (left: kitchen; right: spaceship), the desired behavior is to output a distance close to tdmax , indicating that the images cannot correspond to nearby states on any trajectory. Only the model trained with negative mining learns this behavior, predicting a high distance with high confidence; the model trained without it collapses toward an uncertain mid-range prediction.
Dataset
Avg. Velocity
Total Hrs
Type
HM3D
2.5 m/s
1150h
Synthetic
YouTube
0.5 m/s
1720h
In-the-wild
Ego4D
0.5 – 1.5 m/s
50h
In-the-wild
SCAND
1.5 – 2 m/s
9h
Embodiment
Grand Tour
1 m/s
5h
Embodiment
CityWalker
1.5 m/s
15h
Embodiment
Appendix
Table 4: Datasets used for training and validating the temporal distance function. We report average velocity, total hours, and dataset type.
Figure 7: Projection success ratio (PSR) visualization. Each row shows the current view, the goal-image points projected into the current view (green: depth-consistent matches, red: failed matches), the goal image, and a top-down map of the environment. The legend in the second panel reports the PSR r , the numbers of matched and failed points, and the predicted distance d^ and confidence c^ of GeoNoise.
Figure 8: Adapted ViNT Model Architecture [ 38 ] . We modify the original ViNT model by removing the action prediction head and retaining only the temporal distance prediction branch. The model encodes past observations and goal images with two EfficientNet-B0 encoders, fuses the resulting features into a sequence of tokens, and processes them with a Transformer to predict the (temporal) distance to the goal.
Figure 9: Encoder-based alternative. Observation and goal tokens are concatenated and processed jointly with a Transformer encoder, following the ViNT design but using frozen DINOv2 encoders.
Figure 10: VIP-style alternatives. Observations and goals are encoded separately via some image encoder that can either be frozen or trainable, and distance is computed directly in the embedding space (via ℓ2 , linear, or quasimetric neural networks).
VLD Model
Total # Parameters (approx.)
33M
Trainable # Parameters
11M
Image Encoder
DINOv2-Small
Image Input Res.
224×224
Image Token Dimension
384
Attn. Hidden Dim.
384
Appendix
Table 5: Hyperparameters for the Single-View VLD model.
Training
# Steps
2M
Batch Size
64
Learning Rate
5×10−5
Optimizer
AdamW
LR Schedule
Cosine
Warmup Steps
10K
Appendix
Table 6: Training setup for the Single-View VLD model.
Figure 11: Multi-view Visual Learned Distance (VLD) architecture. Similar to the single-view variant (Figure 2 ), but processes a short temporal window of past views. Each frame is encoded with DINOv2, followed by self-attention across temporal tokens before aggregation and decoding.
Model
HM3D (↑)
Gibson (↑)
20
50
100
20
50
100
VLD (single)
0.82
0.70
0.61
0.84
0.73
0.71
VLD (multi)
0.82
0.71
0.63
0.86
0.75
0.73
Appendix
Table 7: Single-view vs. multi-view inputs. Ordinal consistency (Kendall’s τ , ↑ higher is better) of VLD models on Habitat datasets. We report results at 20, 50, and 100-step horizons.
Objective
HM3D (↑)
Gibson (↑)
20
50
100
20
50
100
VLD (NLL)
0.82
0.70
0.61
0.84
0.73
0.71
VLD (MSE)
0.71
0.67
0.61
0.78
0.70
0.68
VLD (QRL)
0.17
0.20
0.16
0.06
0.07
0.07
VLD (VIP)
0.73
0.53
0.43
0.79
0.59
0.56
Appendix
Table 8: Comparison of learning objectives. Ordinal consistency (Kendall’s τ , ↑ higher is better) of VLD models trained with different objectives on Habitat datasets. Results shown for 20, 50, and 100-step horizons.
Figure 12: Evaluating distance accuracy via spatial point sampling. For each scene, we sample points both inside and outside the room containing the goal (red). Following the criterion illustrated here, a point is considered in-room if there exists at least one camera orientation at that location from which the goal image exhibits non-zero visual overlap; otherwise, it is labeled out-room . A notable side effect of this definition is that certain hallway locations in this figure, adjacent to the goal room, may also be classified as “in-room,” simply because the goal becomes marginally visible from specific orientations at those positions.
Figure 13: Habitat qualitative trajectories. Each top row shows a full trajectory plot (normalized predicted distances, GT distance, and Kendall’s τ ) with all models overlaid (VLD, ViNT-Tuned, VIP-Nav). Each bottom row displays the sequence of observations along the respective trajectory.
Figure 14: Real-world qualitative trajectories (Part 1). The top rows of examples 1–3 show diverse “in-the-wild” and embodiment scenes with normalized predicted distances and Kendall’s τ overlaid for all models (VLD, ViNT-Tuned, VIP-Nav). Each bottom row displays the sequence of observations along the respective trajectory.
Figure 15: Real-world qualitative trajectories (Part 2). Examples 4–6 show additional robot trajectories with normalized distance predictions and auxiliary confidence. Observation strips provide visual context for changes in the predictions. Together with Figure 14 , these show both progress trends and local deviations on real imagery.
Category
Accuracy (%, ↑)
In–In (both in goal room)
58.61
Out–In (one in goal room)
65.48
Out–Out (neither in goal room)
53.32
Appendix
Table 9: Distance accuracy on HM3D validation scenes. The metric reports the fraction of pairwise comparisons in which VLD correctly identifies the closer viewpoint relative to the goal.
Figure 16: Four example trajectories illustrating the behavior of different noise models. Ground-truth (GT) temporal distance (a linearly decreasing signal) is shown in blue . VLD predictions are shown in orange . The GeoNoise —the learned geometric-overlap noise model—produces the curves shown in green , while OU-noised distances are shown in red , obtained by applying an Ornstein–Uhlenbeck process directly to the privileged ground-truth distances.
Figure 17: Exploration driven by VLD. The policy uses VLD as a scalar navigation signal, rotating and evaluating multiple viewpoints before selecting an exploration direction. Each step shows: (i) the agent’s current observation (blue arrow) and the goal position and orientation (red arrow), (ii) the goal image, (iii) a top-down map with the taken trajectory in blue (green shows the optimal path), and (iv) the plot of VLD-predicted distance and confidence over past steps.
Figure 18: Successful navigation. Once partial visual overlap occurs, predicted distance decreases and the policy approaches the goal. The confidence trace is shown for analysis. Each step shows: (i) the agent’s current observation (blue arrow) and the goal position and orientation (red arrow), (ii) the provided goal image, (iii) a top-down map with executed (blue) and optimal trajectories (green), and (iv) VLD distance and confidence plots.
Failure Type
HM3D (%)
Gibson (%)
Time-out failure
71
62
Near-match failure
28
29
Bad or uninformative goal
1
9
Appendix
Table 10: Distribution of navigation failure modes of the VLD-driven policy, as percentages of unsuccessful episodes.
Figure 19: Failure cases. Failures arise from long-horizon exploration without explicit memory (time-outs), visually deceptive near-matches, or ambiguous goal definitions. Each example shows: (i) the agent’s current observation (blue arrow) and the goal position and orientation (red arrow), (ii) the goal image, (iii) the top-down map with the executed (gradient color from dark blue at the start, to red color at the steps close to max steps limit) and optimal paths (green), and (iv) the VLD distance/confidence trace.
Figure 20: Goal images used in the real-world navigation experiments.
Method
G1
G2
G3
G4
G5
Overall
Success rate (SR ↑ )
VLD + GeoNoise
1.00
0.80
0.80
0.80
1.00
0.88
FGPrompt-EF
0.00
0.40
0.60
1.00
1.00
0.60
Average steps over successful trials
VLD + GeoNoise
58.6
138.4
298.6
342.6
80.8
–
FGPrompt-EF
–
326.0
271.6
75.4
106.4
–
Appendix
Table 11: Real-world navigation results. Five repetitions at each of five goals give 25 attempts per method. Average steps count actions in successful trials only. Human action counts provide a reference without exploration. Bold indicates the best SR, including ties.
Figure 21: Example real-world navigation trajectory using VLD. Even when the goal object is not directly visible, the predicted visual distance provides directional guidance that allows the agent to progressively move toward the target.