Learning to Plan from Random Exploration

Deqian Kong; Guangyan Sun; Sheng Cheng; Sirui Xie; Bo Pang; Jianwen Xie; Tony Geng; Caiwen Ding

cs.LGSep 29, 2026

Learning to Plan from Random Exploration

Authors: Deqian Kong, Guangyan Sun, Sheng Cheng, Sirui Xie, Bo Pang, Jianwen Xie, Tony Geng, Caiwen Ding, +1 more

Organizations: UCLA · University of Minnesota · Amazon AGI · Salesforce Research · Lambda · Rice University

Abstract

Random exploration reveals how an environment can be traversed before a goal is specified. Can this experience support long-range planning without policy-improvement training? Our random-walk analysis explains what temporal relations contain: short horizons reveal geodesic geometry in the diffusion limit, while longer horizons reveal connectivity between regions before mixing removes these distinctions. We learn these relations with a conditional energy-based model that estimates temporal log-density ratios through horizon-conditioned embeddings. The model is trained on observation pairs by noise-contrastive estimation, without action or reward labels. The planner queries these learned relations at different horizons as it moves toward the goal. At test time, a separate local dynamics model predicts candidate action outcomes, and the temporal model evaluates their progress toward the goal by selecting or aggregating estimated improvements across horizons. The agent executes one action and replans with both models fixed. Experiments demonstrate long-range maze planning from random exploration using states and images. Learned score fields, embedding probes, and planned routes exhibit properties of a multiscale cognitive map. We further demonstrate egocentric navigation from random exploration and manipulation planning from suboptimal data.

Figures & tables

Figure 1: Emergent cognitive maps from random exploration. (A) Score $G_{\theta}(x,y_{g},\tau)$ over positions $x$ in Large, with goal $y_{g}$ fixed (star). Short horizons ( $\tau=16$ ) peak near the goal, while long horizons ( $\tau=512$ ) spread through connecting passages. (B) t-SNE of $h_{\theta}(x,\tau)$ in Giant at $\tau=1$ and $4096$ . At $\tau=1$ , embeddings preserve local geodesic geometry, while at $\tau=4096$ , they reflect connectivity between distant regions. 3D views appear in Fig. 5 . (C) GP and PAP routes through Large and Giant. Colour denotes the selected horizon for GP and the dominant score-contributing horizon for PAP. Matching markers pair starts ( $\circ$ ) and goals ( $\star$ ).

	Official tasks						Random pairs
	Navigate		Random Data				Random Data
	Large	Giant	Large		Giant		Large		Giant
Method	SR $\uparrow$	SR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Random policy (floor)	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Shortest-path oracle	1.00	1.00	1.00	0.94	1.00	0.98	1.00	0.95	1.00	0.95
GCIQL ( Kostrikov et al., 2022 )	0.34	0.00	0.00	0.00	0.00	0.00	0.15	0.15	0.07	0.06

Table 1: PointMaze goal reaching from states (A) and images (B) on official tasks and random start–goal pairs. The GCIQL, QRL, and HIQL Navigate results are quoted from Park et al. (2025) .

Figure 2: Official-task success versus maximum planning horizon. Color distinguishes Large and Giant and line style distinguishes GP and PAP. Lines show three-seed means and bands one standard deviation. Planning uses every integer horizon from

1

\tau_{\max}

Table 4

Figure 3: An egocentric plan in Habitat Apartment 1 by the model that plans from the image and the position (horizons capped at 512, no constraint on consecutive headings). Left: the route, coloured by the selected horizon, from the start ( $\circ$ ) to the goal ( $\star$ ). Right: the agent’s view at ten decisions and the horizon selected there.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Learned embeddings $h(x,\tau)$ on the Large (top) and Giant (bottom) state mazes; the left column colours each cell by its position and the other columns keep those colours. Every panel is rotated and reflected onto the maze by Procrustes, since t-SNE leaves the orientation undetermined.

$\tau$	1	16	256	4096
near	0.95	0.91	0.58	0.57
far	$-$ 0.01	0.88	0.97	0.70

Appendix

Table 4: Spearman correlation between embedding distance and geodesic distance for near (

\leq 8

steps) and far pairs, Large maze.

Figure 5: 3D t-SNE of state embeddings on Large (top) and Giant (bottom). At $\tau=1$ , 3D views reveal connectivity obscured in 2D. Colours match the maze regions at left; axes are not physical coordinates.

Figure 6: Three held-out cube-single episodes planned from pixels, one per row, each showing eight evenly spaced executed steps with the goal image last. The cube travels 0.455, 0.277, and 0.325 m, respectively; in every row the planner reaches the cube, grasps it, and carries it to the goal.

Explore similar work

How Long, Not How Close: A Learned Temporal Metric for Planning in Latent World Models
Oct 4, 2026Lama Moukheiber, Haotian Xue, Yongxin ChenLatent World Models
Georgia Institute of Technology
Plan First, Diffuse Later: Extrinsic Graph Guidance for Long-Horizon Diffusion Planning
May 16, 2026Yaniv Hassidof, Adir Morgan, Yilun Du +1Diffusion Planning Long-Horizon Planning
Technion · Harvard
Reinforced Planning with Latent World Models
Aug 19, 2026Armin Sommer, Jannik SchillingWorld Model Planning Classical Planning
Pantheon Industries

Oct 4, 2026, cs.LGJ/K move · Enter open · S save

How Long, Not How Close: A Learned Temporal Metric for Planning in Latent World Models

Lama Moukheiber, Haotian Xue, Yongxin Chen

Georgia Institute of Technology

Latent World Models

Open paper AskPDF arXiv

Latent world models plan by rolling a frozen predictor forward under candidate action sequences and ranking the candidates by the latent distance between their imagined end state and the goal. However, this ranking breaks down when the goal lies several plans away, because the latent distance measures how closely an end state resembles the goal rather than how far it remains from reaching it. To address this, we propose TEMPO, a temporal-distance planning objective that leaves the world model untouched, learns only from the demonstrations already used to train it, and adds negligible cost to the planner's search. TEMPO learns a small map of the frozen latent in which the distance between two states of an episode reflects the number of environment steps between them, and blends this distance into the planner's cost. It requires no rewards, policies or success labels and, being a cost rather than a model, applies to frozen world models with one latent vector per state that plan by a latent distance. We evaluate TEMPO on eleven simulated environments (e.g., maze navigation, tabletop pushing, robotic arm control and three-dimensional manipulation) with the LeWM and PLDM planners. With a small MLP that adds at most 0.3% to a plan's arithmetic, TEMPO improves both planners at every goal distance, including the one-plan setting of their evaluations, raises LeWM from 36% to 99% on TwoRoom three plans from the goal, and remains competitive on a broad range of 2D and 3D navigation, reaching and manipulation tasks.

Abstract

Explore similar work

How Long, Not How Close: A Learned Temporal Metric for Planning in Latent World Models

Plan First, Diffuse Later: Extrinsic Graph Guidance for Long-Horizon Diffusion Planning

Reinforced Planning with Latent World Models