Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation

Tzu-Yu Chuang; Ching-Hsiang Chang; Yi-Hsiu Lee; Yi-Ting Chen; Min Sun; YuanFu Yang

cs.ROOct 7, 2026

Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation

Authors: Tzu-Yu Chuang, Ching-Hsiang Chang, Yi-Hsiu Lee, Yi-Ting Chen, Min Sun, YuanFu Yang

Organizations: National Yang Ming Chiao Tung University · National Tsing Hua University

Abstract

Generative world models provide rich predictions of how manipulation scenes may evolve toward task objectives, yet those futures do not directly expose the compact task variables required by control. When training supervises future prediction alone, terminal goal accuracy is not an explicit learning objective, even when geometric recovery is available. We present Entity-Level Goal Readout, a learned prediction-to-execution interface that makes the executable terminal goal an explicit output of a 3D trace world model. It combines object-centric pose prediction with translation grounded in observed depth to produce a compact goal in SE(3). A shared Pose-Native Executor consumes this fixed goal with online object-pose feedback for closed-loop control without rerunning the world model. Across five manipulation tasks, the pipeline achieves a mean success rate of 79.69%. Goal diagnostics directly measure terminal goal accuracy, while controlled translation perturbations characterize how execution degrades under goal error. Zero-shot deployment on a Franka arm achieves 73.33% success on nominal StackCube, 66.67% with distractors, and 75.00% on PickPlate with a target unseen during policy training. These results support treating the prediction-to-execution interface as an explicit learned component of world-model planning rather than incidental post-processing in the control pipeline itself. Project page: https://claire0730.github.io/executable-goals/

Figures & tables

Fig. 1: From prediction to an executable goal. A single RGB-D observation and task instruction condition a 3D trace world model. The Entity-Level Goal Readout converts the resulting world-model representation into a compact object-relative goal in $SE(3)$ for the Pose-Native Executor. The StackCube example illustrates the prediction-to-execution interface from a rich predicted future to closed-loop manipulation.

Fig. 2: Qualitative motivation. Cosmos 3 Edge [ 1 ] , Veo 3.1 [ 2 ] , and TesserAct [ 3 ] illustrate interpenetration, hallucinated geometry, duplicated objects or grippers, deformation, and inconsistent RGB, depth, and normals. These prediction defects motivate evaluating the control goals recovered from predicted futures. The separate goal diagnostic in Table I excludes Veo 3.1.

Fig. 3: System overview. Frozen encoders supply a shared world-model representation for trace prediction and Entity-Level Goal Readout. The readout exposes one object-relative goal, anchored at $t=0$ and fixed for the episode. Online tracking and pose estimation update object state to close the loop. The Pose-Native Executor combines pose, goal, proprioception, and previous action to produce an eight-dimensional arm and gripper command at $20$ Hz.

Fig. 4: Entity-Level Goal Readout. (a) Masked pooling and cross attention produce terminal rotation, with dashed paths for training-only auxiliary supervision. (b) The map branch grounds position in valid observed depth. (c) Goal composition combines rotation and position into the executable goal. Image panels illustrate intermediate representations.

Prediction	Method	PickCube	PegInsertionSide	StackCube
RGB video	Cosmos 3 Edge	$141.3\!\pm\!89.3$	$348.2\!\pm\!214.9$	$87.6\!\pm\!61.6$
RGB-DN video	TesserAct, adapted	$456.9\!\pm\!158.3$	$76.9\!\pm\!38.2$	$224.8\!\pm\!89.0$
2D tracks	Track2Act, adapted	$167.5\!\pm\!57.0$	$256.2\!\pm\!33.0$	$143.0\!\pm\!89.0$
4D geometry	Structured-4D, released	$192.2\!\pm\!82.9$	$373.5\!\pm\!115.3$	$31.0\!\pm\!19.3$
4D geometry	Structured-4D, adapted	$166.4\!\pm\!61.3$	$361.4\!\pm\!83.3$	$36.3\!\pm\!20.3$
3D tracks	TraceGen + Rigid Readout ( $K=1$ )	$41.1\!\pm\!21.7$	$42.6\!\pm\!31.6$	$28.9\!\pm\!38.7$

TABLE I: Terminal goal position error in mm, reported as mean

\pm

standard deviation over valid readouts. Source-specific standard deviation conventions are retained and documented in the supplementary appendix. Lower is better. Evaluation banks and validity conventions are specified in Sec. IV .

	Success (%)
Method	PickCube	LiftPegUpright	PegInsertionSide	StackCube	PushCube	Mean
Native executor
DP3 [ 14 ]	36.00	84.00	12.00	20.00	93.00	49.00
RoboFlow4D [ 24 ]	14.00	–	–	12.00	45.00	–
Structured-4D, released	0.00	0.00	16.00	84.00	0.00	20.00
Structured-4D, adapted	33.00	16.00	20.00	89.00	92.00	50.00

TABLE II: Closed-loop task success. External systems use native executors. The shared-executor block uses the same policy weights and paired scenes, but different goal pipelines and planner checkpoints. Native-executor results are separate system-level evaluations. Rates are percentages. Mean is the unweighted average of all five displayed task rates, rounded to two decimal places. It is omitted for incomplete rows. Bold and underline mark the best and second-best results in each column across both blocks, ranked using the reported values. Dashes denote unreported results. Oracle Goal references are shown separately in Fig. 5 (a).

Panel of Figure 7 — Fig. 5: Task success and goal error distributions. (a) Light blue, blue, and orange denote rigid $K=1$ , rigid $K=4$ , and the final pipeline. Gray segments show empirical Oracle Goal references. (b,c) Seed 999 box plots show the median and interquartile range (IQR), $1.5$ IQR whiskers, and all outliers on logarithmic axes. Groups have $256$ finite records except StackCube rigid positions ( $252/256$ ). (c) Raw quaternion errors are shown for PegInsertionSide. Comparison conditions are in Sec. IV-B .

Panel of Figure 8 — Fig. 6: Position Quality and Geometry Probe. Median / 90th-percentile errors in mm. Lower is better. Final denotes the routed pipeline. No geo. removes geometry from the entity-position predictor at test time. It is not an ablation of Final. Segments connect quantiles, not confidence bounds. All groups use $256$ scenes, with $252$ finite StackCube rigid positions and $256$ otherwise.

Panel of Figure 9 — Fig. 7: Execution sensitivity to goal position. The Oracle Goal is translated with the planner bypassed. Each point pools $256$ scenes at seeds 999 and 997 ( $512$ episodes), with scenes and directions paired across magnitudes. Markers show measured success, error bars show pointwise $95\%$ Wilson intervals, and lines guide the eye. PegInsertionSide uses $10$ mm clearance.

Planner	Parameters	Memory	Time	Output
		(GB)	(s)
Cosmos 3 Edge	9.13B	19.33	35.63	81 RGB frames
TesserAct	5.7B	16.22	$292.0\!\pm\!4.9$	49 RGB-DN frames
Track2Act	469M	1.91	3.80	400 2D tracks
Structured-4D	1.1B	5.71	4.45	4D point cloud
Ours	677.6M	5.60	1.27	3D traces + goal

TABLE III: Planning cost on an RTX 5090 after model warmup using the shared timing audit. TesserAct reports mean

\pm

standard deviation over 18 clips.

Task	Setting	Successes / Trials	Rate (%)
StackCube	nominal	$22/30$	73.33
PickPlate	unseen target	$9/12$	75.00
StackCube + distractors	intervention	$8/12$	66.67

TABLE IV: Real robot success on a Franka arm with an RTX 3090.

Panel of Figure 12 — Fig. 8: Real robot deployment. Rollouts show (a) nominal StackCube, (b) StackCube with distractors, and (c) PickPlate, where the cube is placed in the orange plate. PickPlate is unseen during policy training. Frames progress from left to right.

Explore similar work

Achieve What You Imagined: Learning to Align Actions with Visual Plans
Sep 27, 2026Yuheng Qiao, Ziran Wei, Xiaohan Wang +5Contact-Rich Dexterous Manipulation Action Expert
KTH · Beihang University · The Hong Kong University of Science and Technology (Guangzhou) +2
DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation
Aug 13, 2026DreamX Team, Rui Chen, Xiangxiang Chu +7Efficient World-Action Model Robotic Manipulation
World Models for Robotic Manipulation: A Survey
May 27, 2026Fangyuan Wang, Ziyuan Wang, Guorui Pei +15World Models Robotic Manipulation
Department of Mechanical Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China · Department of Mechanical Engineering and Automation, Harbin Institute of Technology, Shenzhen, China · School of Advanced Engineering, Great Bay University, Dongguan, Guangdong, China +9

Sep 27, 2026, cs.ROJ/K move · Enter open · S save

Achieve What You Imagined: Learning to Align Actions with Visual Plans

Yuheng Qiao, Ziran Wei, Xiaohan Wang, Daqiang Guo+4

KTH · Beihang University · The Hong Kong University of Science and Technology (Guangzhou)+2

Contact-Rich Dexterous Manipulation Action Expert World Models Physical Plausibility

Open paper AskPDF arXiv

World-action models can jointly predict future visual observations and robot actions. However, discrepancies may exist between their visual predictions and the consequences implied by generated actions. We observe that WAMs can often generate visually plausible task-completion outcomes before producing action sequences that reliably achieve them. Consequently, we treat the WAM-generated visual prediction as a goal-conditioned visual proposal rather than a directly executable plan. We use a frozen action-conditioned world model to predict action-conditioned consequences and construct feedback based on consistency between the two future predictions and alignment with the terminal goal. Leveraging this feedback, we employ Flow Policy Optimization (FPO) to optimize the action head of the WAM. This framework avoids online robot interaction and additional training of task-specific reward models. Across four real-world UR5 manipulation tasks, our method increases the mean success rate from 43.4% to 75.1%, compared with 61.4% for

π_{0.5}

. These results show that cross-model prediction discrepancy can provide useful feedback for improving robot policies under the evaluated manipulation tasks. Website: https://imagine-to-achieve.github.io/

Abstract

Explore similar work

Achieve What You Imagined: Learning to Align Actions with Visual Plans

DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

World Models for Robotic Manipulation: A Survey