Image-goal planning with a frozen visual world model depends on how predicted observations are scored against the goal. Mean-squared error (MSE) between embeddings from a pretrained visual encoder can provide a useful signal near a goal, yet favor states across an obstacle that are visually similar to the goal and difficult to reach. We fit a small residual adapter on top of frozen I-JEPA embeddings to improve this latent goal cost. Starting near each goal, a planner collects trajectories from progressively wider regions of initial states. The adapter learns from the order of observations along successful trajectories, with additional comparisons inferred from failed rollouts. No task-specific progress labels are used for fitting, so the procedure can be applied to other goal-reaching tasks. DotWall and PointMaze are used in this work. In paired 50-case evaluations with identical starts, goals, world models, and planner settings, success increases from 54% to 98% in DotWall and from 82% to 96% in PointMaze. On 16 unseen DotWall layout-goal cases, the adapter fitted on one fixed layout improves agreement between latent-cost ordering and shortest legal-path ordering from 71.9% to 85.8%. The results indicate that planning experience can improve a frozen visual representation's goal cost without retraining the encoder or the world model.
Figures & tables
Fig. 1: Rendered observations from the two environments. The red ball starts at its shown position and must reach the green goal. In DotWall, the legal route passes through a doorway; in PointMaze, it goes around an obstacle.
Fig. 2: World-model training. Frozen I-JEPA encodes the current and next images. The action-conditioned world model predicts the next visual embeddings and proprioceptive state, which are compared with the observed targets.
Fig. 3: Image-goal planning with the learned world model. CEM proposes action sequences, the model predicts their outcomes in latent space, and the MSE between a predicted terminal latent and the encoded goal scores each sequence. Only the first selected action is executed before replanning.
Fig. 4: Raw latent-cost heatmaps for one goal in each environment, compared with shortest legal-path distance. Darker colors indicate lower values; each panel has its own color scale. The costs are computed from rendered observations, not imagined world-model rollouts. Both encoders show departures from legal-path ordering, although their relative quality varies by goal.
Fig. 5: The encoder and world model remain frozen. The adapter transforms predicted and goal embeddings only on the latent goal-cost path used by CEM.
Fig. 6: The first three DotWall start bands around one goal. Each stage adds starts farther from the goal. Bands determine starting positions, not the region an episode must remain within.
Fig. 7: Latent-cost fields for one goal in each environment. Columns show shortest legal-path distance, raw I-JEPA latent MSE, and adapted latent MSE. Each position represents the cost of a rendered observation with the ball at that position relative to the goal image (star). DotWall goal (14,22) was held out from adapter fitting; Each panel has an independent color scale.
Fig. 8: Matched DotWall example: raw-cost planning times out in the left room; adapted-cost planning crosses the doorway and reaches the goal.
Fig. 9: Matched closed-loop planning success with raw and adapted I-JEPA latent costs. (a) DotWall; directly blocked cases are a subset of opposite-room cases. (b) PointMaze; opposite-arm cases are a subset of detours. Counts above bars are successes over evaluated cases.
Fig. 10: Matched PointMaze detour: raw-cost planning remains in the lower arm until timeout, while adapted-cost planning goes around the central wall and reaches the goal.
Measure
Raw I-JEPA
Adapted
Map-wide pair ordering
71.9%
85.8%
Opposite-room pair ordering
48.5%
77.1%
Mean Spearman ρ
0.594
0.864
Doorway-local uphill steps
39.2%
15.6%
TABLE I: Latent-cost geometry on 16 unseen DotWall layout–goal cases.
Fig. 11: Two unseen DotWall layouts (rows), with shortest legal-path distance and adapted latent cost (columns). The adapter was fitted only on the standard (32,30) layout. Each panel uses its own color scale.
Fig. 12: Detailed world-model training architecture. A frozen I-JEPA encoder maps the current and next observations to visual embeddings. The model combines the current embeddings and proprioceptive state with a macro-action comprising five successive 2D actions. An action-conditioned transformer predicts the next visual embeddings and proprioceptive state; both predictions are trained against their observed targets using MSE.
Fig. 13: Schematic of the failed-episode signal in DotWall. Green routes S1 – S4 reach the goal; the red route fails. Dots are visited observations and circles mark two sampled failed observations. Dashed links represent latent comparisons with each successful route, not traversable paths. Highlighted blue links are the three lowest costs for A . All displayed values are illustrative.
Image-goal navigation with latent world models requires not only accurate future prediction, but also a planning cost that reliably ranks candidate action sequences. We define the cost as the cosine distance between the predicted future embedding and the goal embedding, and show that poor cost ordering can mislead sampling-based planners such as Cross-Entropy Method (CEM). To address this, we propose a latent world model built on a frozen DINO-family encoder and train it with two complementary objectives. An autoregressive rollout loss reduces the gap between training and multi-step planning rollouts, while a Monotone Cost Ranking (MCR) loss directly encourages increasingly perturbed action sequences to receive higher planning costs. We also study InfoNCE-based action-contrastive training and find that temporal permutation negatives distort the latent geometry and degrade planning performance. On the GNM navigation dataset, our method outperforms Navigation World Models (NWM), DINO-WM, OmniVLA, and NoMaD, achieving state-of-the-art image-goal navigation performance while reducing orientation error by 2.7× over the same-encoder DINO WM baseline. We also deploy the model zero-shot on a physical robot, where it follows goal-directed paths in unseen indoor and outdoor environments.
Latent world models are judged by how well they predict, so when planning fails at long horizons the natural reading is that the predictor degrades. On a reproduction of LeWorldModel on TwoRoom we show the binding constraint is the planner's objective instead. The predictor is not the limit: its imagined state seventy-five environment steps ahead is still only 0.189 as wrong as assuming the world froze, while the planner never imagines beyond twenty-five. The objective is. Cross-entropy-method planning minimises squared latent distance, which tracks true distance at r = 0.426, saturates by about eighty arena units and decreases beyond a hundred and twenty, so moving away from the goal can lower the cost. The information is present throughout: a ridge probe recovers position from the frozen embedding at R^2 0.9922. The pathology is the method's, not one reimplementation's. It is present in the authors' released weights, and across four checkpoints long-horizon success rank-orders exactly with metric quality and inversely with prediction accuracy. Replacing only the objective, with nothing retrained and no GPU, lifts goals reached at offset 100 from 26.0% to 98.0%, equals the 98.0% at offset 25, and reaches 92.0% under a third of the budget: planning stops depending on the horizon. The best cost is not the most accurate. A head learned from frame separation alone predicts spatial distance worse than a position probe (r = 0.819 against 0.9897) yet plans better, charging 24% more to cross the environment's dividing wall where squared latent distance charges 4% less. It has learned reachability, not proximity.
Learning good representations is essential for latent planning with world models. While pretrained visual encoders produce strong semantic visual features, they are not tailored to planning and contain information irrelevant -- or even detrimental -- to planning. Inspired by the perceptual straightening hypothesis in human visual processing, we introduce temporal straightening to improve representation learning for latent planning. Using a curvature regularizer that encourages locally straightened latent trajectories, we jointly learn an encoder and a predictor of a Joint-Embedding Predictive Architecture (JEPA) world model. We show that reducing curvature this way makes the Euclidean distance in latent space a better proxy for the geodesic distance and improves the conditioning of the planning objective. We demonstrate empirically that temporal straightening makes gradient-based planning more stable and yields significantly higher success rates across a suite of goal-reaching tasks. Our code is available at https://agenticlearning.ai/temporal-straightening.
Ying Wang, Oumayma Bounou, Gaoyue Zhou +4
New York University · Brown University · University of Toronto