Latent world-model planning has shown promising results in goal-conditioned robotic manipulation, but remains challenging for long-horizon planning due to limited temporal guidance and inefficient action search. A distant visual goal provides little guidance for preparatory actions, while uninformed action sampling makes local optimization inefficient. We propose PILOT (Policy-Initialized Local Optimization with Temporal subgoals), which uses a video model to generate intermediate visual subgoals and a learned policy to initialize the cross-entropy method (CEM). CEM refines candidate action sequences toward each subgoal using an action-conditioned world model in a receding-horizon control loop. In our primary real-robot evaluation across four UR5e manipulation tasks, PILOT achieves 68.8% success, compared with 55.0% without policy initialization, 13.8% for policy-only execution, and 0.0% for direct final-goal planning. Simulation results show that these gains persist over longer horizons: PILOT exceeds direct execution of the same policy by 30 percentage points on long Push-T tasks and by 22 percentage points on LIBERO-10. Policy initialization also accelerates local optimization: on Push-T, CEM matches the cost reached by generic initialization after 10 iterations in only 4 iterations and reduces the final planning cost by 24.1% under the same sampling budget. These results show that combining temporal subgoals with policy-guided action search improves both long-horizon task execution and local planning efficiency. Videos are available on the webpage: https://pilot-planner.github.io/.
Figures & tables
Fig. 1: Motivation for PILOT. Visual subgoals decompose long-horizon planning into local planning stages, while a learned policy prior focuses action search on plausible trajectories.
Fig. 2: Overview of PILOT. A video prediction model generates visual subgoals from the current observation and task instruction. For each active subgoal, a learned policy initializes CEM, which refines candidate action sequences using an action-conditioned world model and a latent-space subgoal cost.
Fig. 3: Real-world UR5e evaluation setup. We evaluate PILOT on four pick-and-place tasks using visual observations from a fixed RGB camera.
Fig. 4: PILOT on the four real-world tasks. Top row of each panel: initial observation and video-predicted visual subgoals. Bottom row: robot observations after the allotted control steps for each subgoal.
Method
Cup
Pencil Case
Bowl-Box
Bowl-Bowl
Avg. (95% CI)
Flat WM
0/20
0/20
0/20
0/20
0.0% (0.0–4.6)
Policy only
4/20
3/20
2/20
2/20
13.8% (7.9–23.0)
PILOT w/o prior
8/20
5/20
14/20
17/20
55.0% (44.1–65.4)
PILOT
11/20
13/20
13/20
18/20
68.8% (57.9–77.8)
TABLE I: Real-world task success over 20 trials per task. Avg. pools all 80 trials; parentheses give 95% Wilson confidence intervals.
Fig. 5: Effect of policy-prior initialization on local planning efficiency. (a) Policy-guided initialization achieves higher task success under limited online planning budgets. (b) The prior provides a lower-cost initialization and maintains lower planning cost during CEM refinement.
Fig. 6: PILOT on Push-T. Top: initial observation and video-predicted visual subgoals. Bottom: simulator observations after the allotted control steps for each subgoal.
Method
Short
Medium
Long
Flat WM
33.3%
0.0%
0.0%
Flat WM + prior
46.7%
0.0%
0.0%
Policy only (5 demos)
0.0%
0.0%
0.0%
Policy only (10 demos)
13.3%
3.3%
6.7%
Policy only (20 demos)
26.7%
16.7%
10.0%
Policy only (30 demos)
40.0%
33.3%
23.3%
TABLE II: Push-T success rate by task length (30 held-out initial states per bucket). Methods with a policy prior use the 30-demonstration policy.
Fig. 7: Push-T planning efficiency. (a) Success on medium-length tasks versus mean time per planning call as the CEM budget varies, for generic initialization and for priors fine-tuned on 10, 20, and 30 demonstrations; lines are saturating-exponential fits. (b) CEM objective over iterations with and without policy initialization on identical planning problems from 90 held-out initial states; costs are normalized to generic initialization at iteration one, and shading shows standard errors.
Fig. 8: PILOT on the LIBERO-10 task put the black bowl in the bottom drawer of the cabinet and close it . Top: initial observation and video-predicted visual subgoals. Bottom: simulator observations after the allotted control steps for each subgoal.
Fig. 9: LIBERO-10 success rate (5 held-out initial states per task, aggregated over the ten tasks). PILOT (oracle) uses expert subgoals.
Video models provide rich priors for generalizable robotic manipulation, but plausible visual futures are not always executable. We introduce DreamCan, a hierarchical policy that grounds video models through sparse visual planning: a high-level planner adapted from a pre-trained video model and a low-level visuomotor policy. Rather than predicting dense trajectories, the high-level planner generates sparse visual subgoals that balance reachability and task utility. From offline, action-free robot videos, we estimate these quantities and use dynamic programming to select subgoal sequences without manual annotations. The resulting subgoals are more reliably attained than uniformly spaced alternatives, and the full system outperforms language, dense-video, and uniform-subgoal planners, reaching high planning success rates on Language-Table tasks and 91.2% success on LIBERO-Long with a 0.1B-parameter low-level policy. Additional suboptimal videos and test-time compute further improve success. The system also performs challenging real-world bimanual tasks, including box folding and packing with unseen object combinations. More results are available at https://dreamcan.github.io/.
Chongkai Gao, Wenlong Huang, Haonan Chen +5
Stanford University · National University of Singapore · Harvard University +1
Long-horizon compositional manipulation has become increasingly important for real-world robot deployment, where a single task involves multiple coordinated subtasks. Existing world-action models (WAMs) jointly predict short-horizon visual futures and actions, but typically lack explicit subtask-level reasoning. We propose Visual Goal-conditioned Action Reasoning (ViGAR), a hierarchical framework that factorizes manipulation into a visual subgoal planner and a subgoal executor. Given the current observation and global instruction, the subgoal planner predicts a visual subgoal for the next subtask. The subgoal executor then jointly generates future visual trajectories and actions conditioned on the predicted subgoal. Both components share a pretrained world-model representation, enabling task-level planning and action generation to benefit from common physical knowledge. Moreover, our framework naturally supports in-context learning: using a global goal image as context can induce different subtask decompositions and behaviors without parameter updates. On the RoboTwin Clean2Random benchmark, ViGAR achieves 82.00% and 67.02% success rates under the Clean and Random settings, respectively, surpassing the strongest baseline by 12.86 percentage points in average success rate. Real-world robot experiments on five compositional and two in-context learning tasks further confirm the effectiveness of ViGAR.
World-action models use predicted visual futures to condition robot actions, yet execution feedback can invalidate parts of a prediction while leaving its task structure useful. We propose Revisable Temporal Planning (RTP), which maintains the visual future as a persistent action condition and revises it after feedback. Its central mechanism is a learned revision bridge: it resumes an intermediate state saved during visual generation and adapts its continuation to current observations. Visual and action supervision connect this revision to subsequent control. Time-aware history supplies observed evidence, and an adaptive policy selects retention, bridge revision, or fresh replanning from new noise before decoding the next action. On RoboMME and RMBench, RTP achieves task-averaged success rates of 48.6% and 84.8%, respectively. Matched comparisons support learned continuation; estimated checkpoint-source and action-prefix effects are positive but less precisely resolved. These results connect feedback-driven visual-plan revision to closed-loop task performance. Project Page: https://PLACEHOLDER.github.io/RTP/