Video models provide rich priors for generalizable robotic manipulation, but plausible visual futures are not always executable. We introduce DreamCan, a hierarchical policy that grounds video models through sparse visual planning: a high-level planner adapted from a pre-trained video model and a low-level visuomotor policy. Rather than predicting dense trajectories, the high-level planner generates sparse visual subgoals that balance reachability and task utility. From offline, action-free robot videos, we estimate these quantities and use dynamic programming to select subgoal sequences without manual annotations. The resulting subgoals are more reliably attained than uniformly spaced alternatives, and the full system outperforms language, dense-video, and uniform-subgoal planners, reaching high planning success rates on Language-Table tasks and 91.2% success on LIBERO-Long with a 0.1B-parameter low-level policy. Additional suboptimal videos and test-time compute further improve success. The system also performs challenging real-world bimanual tasks, including box folding and packing with unseen object combinations. More results are available at https://dreamcan.github.io/.
Figures & tables
Fig. 2 : The problems of language planning, dense video planning, and uniform subgoal planning on the Push-T task.
Fig. 3 : Subgoal annotation for Box Folding . Left: demonstration frames. Middle: reachability, task utility, their combination, and the dynamic programming cost curves used for selection at the initial frame. Note that reachability and DP curves will change with the starting state. Right: full cost curves J during the subgoal selection process. We illustrate four subgoals for clarity; the task uses twelve in our experiments.
Fig. 4 : Illustration of the subgoals generated from our methods on all simulation tasks in this paper.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
FVD ↓
LT
LL
LT
LL
LT
LL
LT
LL
Dense Visual Planning
27.54
18.52
0.859
0.834
0.115
0.128
233
370
Uniform Subgoal Planning
27.51
18.06
0.855
0.826
0.118
0.143
258
380
Ours
28.95
18.86
0.870
0.838
0.103
0.127
154
363
TABLE I : Visual plan generation quality comparisons on Language Table (LT) and LIBERO-Long (LL) tasks.
VLP [ 7 ]
DreamCan (Ours)
make a line
56%
86%
all to region
78%
100%
canonical shape
15%
52%
TABLE II : Long-horizon task planning success rates on three Language Table tasks. We reproduced the canonical shape result by ourselves because the original VLP didn’t do it.
Method
PushT
LIBERO-Long
Language Table
E2E Policy
58.0%
79.2%
–
Language Planning
62.0%
75.6%
56.0%
Dense Visual Planning
70.0%
84.0%
71.0%
Uniform Subgoal Planning
72.0%
82.4%
69.0%
VLA-OS [ 31 ] (0.9B, non-pretrained)
72.0%
76.0%
–
π0 [ 32 ] (3.3B, pretrained)
76.0%
85.2%
–
TABLE III : Comparison of different planning methods on PushT, LIBERO-Long, and Language Table make a line task. Numbers for our methods are the average success rates of 50 evaluation episodes. The VLA and WAM results on PushT were run by us. LIBERO-Long results are from their original papers. Bold numbers indicate the best results or our results.
Fig. 5 : Top: subgoals found by uniform sampling and ours. Bottom: success rates on the Square task with more suboptimal data for different planners.
Fig. 6 : The real-world experiment setup, three real-world tasks, and subgoal sequences.
Fig. 7 : Test time search results on the Language Table make a line task. Each point is evaluated over 100 testing episodes.
Fig. 9 : Language steering experiments. Left: given different language instructions, our planner can generate different corresponding planning results. Right: given only the black instruction, our planner can generate different rollouts, while a VLM instructed by the green text can give a task progress value function that selects the correct planning rollout.
DP Terms
S
RS
US
RU
Ours
SR
81.8%
74.2%
78.0%
83.8%
91.2%
TABLE IV : The ablation study of subgoal discovery terms.
Fig. 10 : Left: learned reachability follows the shortest path, not elapsed time. Right: our reachability skips a failed grasp.
Method
Box Folding
Box Packing (Unseen)
Make a Line
End-to-End Policy
5%
0%
10%
Dense Video Planning
0%
5%
0%
Uniform Subgoal Planning
10%
20%
25%
Ours
40%
65%
55%
TABLE V : Real-world experiment results. We evaluate each task with 20 episodes.