VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching

Yunzhe Hu; Dong Xu

cs.LGOct 19, 2026

VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching

Authors: Yunzhe Hu, Dong Xu

Organizations: School of Computing and Data Science, The University of Hong Kong

Abstract

Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based methods allow for planning by scaling test-time compute to solve complex tasks. Combining both holds appealing promise for pre-training on suboptimal demonstrations and zero-shot adaptation via goal specification and planning. Yet existing methods assume effective planning requires accurate next-state prediction in pixel or latent space. In this work, we show that value prediction is more useful for planning, whereas accurate state prediction is often unnecessary. Our key insight is that offline goal reaching naturally admits a closed-form Monte Carlo return, enabling value regression to replace temporal-difference bootstrapping. We use this return as the sole supervision for latent world models: imagined latents are trained to predict goal-reaching value rather than match future states, implicitly preventing representation collapse. These planning-oriented representations let imperfect world models imagine on the level sets of value field beyond dataset transitions. We then present VPFlow, a value-predictive planner that pairs the world models with a one-step flow policy and plans in its noise space via evolutionary search. It enables policy improvement with imagined generalized advantage and reduces out-of-distribution queries common in action-space planning. On 30 state and pixel OGBench datasets, VPFlow surpasses model-free baselines by at least 30.3 and 10.1 percentage points in success rate, respectively, and outperforms latent planners LeWorldModel, PLDM, and TD-MPC2. It achieves this despite a latent prediction error orders of magnitude larger than with state prediction, while minimizing this error lowers success rate. This demonstrates that value-predictive representations, not exact state prediction, are what offline planning requires.

Figures & tables

Figure 1: State-predictive and value-predictive action-conditioned latent world models. Left : Joint-embedding predictive architecture (JEPA) aligns predicted and future latents with a joint encoder and explicit regularization, e.g. , variance-invariance-covariance regularization (VICReg) or sketched isotropic Gaussian regularization (SIGReg). Middle : TD-MPC learns values from one-step temporal-difference (TD) targets, with source and target encoders, stop-gradient, and exponential moving average (EMA) heuristics. Right : VPFlow trains the latent dynamics and encoders jointly via value prediction: predicted latent states seek accurate goal-reaching value estimates, not future states. Closed-form hindsight Monte Carlo (MC) returns provide the temporal signal and act as an implicit regularizer.

Figure 2: The value equivalence principle view of world modeling. The value-predictive representations of the same timestep constitute a level set of the value field.

Figure 3: VPFlow objectives. VPFlow alternately trains (a) the goal-conditioned value function $V_{\phi}$ , latent dynamics $f_{\phi}$ , and encoder $E_{\phi}$ jointly, and (b) a distilled one-step flow policy $\pi_{\theta}$ . The value function regresses the closed-form Monte Carlo returns $R_{h}$ on imagined rollouts. The policy is improved by maximizing the imagined value from the learned dynamics model, regularized by conditional flow matching to stay close to the data manifold.

Input	Task Type	Environment	GCBC	GCIVL	GCIQL	QRL	CRL	VPFlow
State	Locomotion	PointMaze (4 datasets)	16.0	38.2	27.8	60.0	29.8	87.8
	Locomotion	AntMaze (4 datasets)	19.8	31.8	35.0	53.0	61.8	68.0
	Manipulation	Cube (4 datasets)	2.0	22.5	27.8	1.5	8.2	42.2
		Scene (1 dataset)	5.0	42.0	51.0	5.0	19.0	78.0
		Puzzle (4 datasets)	0.5	9.0	36.8	0.2	2.0	51.0
		Overall (17 datasets)	9.3	26.4	32.9	27.3	25.1	63.2

Table 1: Success rate (

\%

\uparrow

) on OGBench compared against model-free baselines. Entries are mean success rates over the datasets in each environment, with the number of datasets in parentheses. The best and second best results in each row are in bold and underlined, respectively.

Figure 4: Success rate of planners with latent world models on the only pixel dataset where the baselines do not completely fail.

Panel of Figure 5 — Figure 5: Comparison of value-predictive and state-predictive representations. Success rate (left axis) and next-latent prediction error (right axis, log scale) during training, for value-predictive (solid) and state-predictive (dashed) models. The error is the mean Euclidean distance $\|\hat{z}_{t+h}-z_{t+h}\|_{2}$ between the predicted and encoded next latents. On state and pixel inputs, the value-predictive representation achieves higher success even when its state prediction error is larger by orders of magnitude.

Panel of Figure 8 — Figure 8: Rollouts in the environment and learned world models, with their value fields. With value prediction, the agent maintains a lower value error and reaches the goal successfully despite substantially larger latent prediction error.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: World model architecture. A Transformer with cross-attention and causal mask enables parallel rollouts of future latents induced by action chunks.

B

is the batch size,

H

the chunk length,

d_{a}

the action dimension,

d_{z}

the latent dimension,

d

the Transformer width.

Panel of Figure 10 — Figure 10: Noise-space evolutionary search with $\varepsilon$ -ES on pointmaze-giant-navigate-v0 . Gray points denote candidate noises, orange points denote elites, and the blue star marks the selected noise at each generation. The right panel shows the corresponding advantage scores.

Setting

LeWM

PLDM

TD-MPC2 b

Appendix

Table 3: Configurations of latent planning baselines on pixel-based OGBench. Merged cells indicate shared settings; – denotes an absent component or an inapplicable setting.

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Aug 25, 2026

LeFlow: Generative Latent Flow Planning for World Models

Latent world models are inherently strong encoders that transform image pixel to latent embedding, yet existing world models still rely on online trajectory optimization for action planning: for every state-goal pair, an iterative optimizer is run from scratch to search for optimal action sequences, treating the world model as a black-box simulator. This approach pays the full iterative optimization cost anew at every replanning step and reuses no planning experience across queries. In this work, we ask whether planning itself can be amortized once a latent world model has been learned. We present LeFlow, which learns a reusable latent trajectory prior operating directly in the latent dynamics space from the world model. LeFlow recasts planning as conditional latent trajectory generation: a rectified-flow model imagines a future latent path between the current and goal embeddings, an inverse dynamics decoder turns latent transitions into action chunks, and the frozen world model verifies each candidate by autoregressive rollout. Across four major goal-conditioned pixel-control benchmarks, LeFlow replaces iterative action-space optimization with amortized latent planning and fixed-budget rollout selection, achieving consistent success-rate gains with an order-of-magnitude reduction in planning time. Our results argue that latent world models should support not only prediction but reusable planning priors. Our code is available at https://github.com/hsiangwei0903/LeFlow.

Hsiang-Wei Huang, Jianxu Shangguan, Junbin Lu +1

University of Washington, United States

Latent-Space Planning World Model-Based Planning Latent World Models

Ask PDF

Sep 19, 2026

FIRM-WM: State-factorized factual-interventional recurrent modeling for reward-free visual planning

Reward-free latent world models can learn from offline videos and solve new image--goal tasks by optimizing actions through predicted latent futures. This setting places two demands on the planning state: its coordinates must be comparable with a goal image. Moreover, its dynamics must retain velocity, motion trend, contact, and other history--dependent information beyond those goal coordinates. Offline training creates a second mismatch: each recorded trajectory reveals one factual future, whereas a sampling--based planner compares many actions that were not taken from the same state. We introduce FIRM-WM (Factual--Interventional Recurrent World Model), a compact pixel world model designed around these two gaps. Its recurrent state separates a typed, goal--comparable configuration from a 128-dimensional dynamic fiber used for prediction but excluded from the terminal goal cost. Broad factual trajectories provide state coverage, while common--reset intervention branches provide observed outcomes for alternative action sequences. Before executing each branch, we reset the environment and restore the same recorded values exposed by the environment's state--setting interface. Under matched CEM planning and three independent full-pipeline seeds, FIRM-WM reaches 99.0

\pm

1.0% on TwoRoom, 92.7

\pm

2.1% on Reacher, and 88.0

\pm

3.0% on OGBench-Cube, compared with 89.0%, 88.0%, and 70.0% for LeWM. The deployed model uses 2.98--3.42M parameters and records 2.13--11.60

\times

lower planning time on these tasks.

Yilun Wu, Yunjian Zhang, Aobo Li +3

Saint Petersburg State University, Saint Petersburg, Russia · Shenzhen Kaihong Digital Industry Development Co., Ltd., Shenzhen, China · Chinese Academy of Sciences (CAS) – Shenzhen Institute of Advanced Technology +2

World Models for Robotics Latent World Models Action-Conditioned World Models Goal-Conditioned Planning

Ask PDF

Aug 19, 2026

Reinforced Planning with Latent World Models

Humans solve complex problems by constructing plans and mentally simulating their outcomes with an internal model of the world. Machine learning has made substantial progress in learning world models that predict the consequences of action sequences, yet the procedures used to plan with these models remain largely hand-designed. Most planners rely on fixed search or optimization rules; approaches that learn aspects of search typically imitate a predefined optimizer or use planning to inform an amortized policy, rather improving multi-step plans. We introduce Reinforced Planning, a method that learns the plan-update itself by reinforcing update rules that produce better plans, using gradients propagated through a differentiable world model. We instantiate Reinforced Planning in RP1, which learns a critic over imagined outcomes via temporal-difference learning and a neural plan-improvement operator trained via imagined rollouts with a pretrained world model. RP1 can be trained fully offline without environment interaction; environment episodes are used only for checkpoint selection. Across visual navigation, arm reaching, and robotic manipulation on two world-model backbones, RP1 matches or exceeds existing planners, achieving near-perfect success in several settings while using

1{,}000\times

fewer world-model rollouts than the strongest alternative (CEM) and planning up to

67\times

faster under concurrent planners inference.

Armin Sommer, Jannik Schilling

Pantheon Industries

World Model-Based Planning Robotic RL Model-Based Planning Model-Based RL

Ask PDF

Abstract

Offline RL

Explore similar work

LeFlow: Generative Latent Flow Planning for World Models

FIRM-WM: State-factorized factual-interventional recurrent modeling for reward-free visual planning

Reinforced Planning with Latent World Models