Goal Images Are Control Targets: Spatially-Grounded Goal-Image Synthesis for Robotic Manipulation

Riddhi Chatterjee; Ritish Shrirao; Kushagra Mehrotra; Viswanath Gopalakrishnan

cs.CVOct 17, 2026

Goal Images Are Control Targets: Spatially-Grounded Goal-Image Synthesis for Robotic Manipulation

Authors: Riddhi Chatterjee, Ritish Shrirao, Kushagra Mehrotra, Viswanath Gopalakrishnan

Organizations: International Institute of Information Technology Bangalore, India

Abstract

Goal-conditioned manipulation policies are steered by a goal image, a picture of the desired post-manipulation scene synthesized from the current observation and a language instruction. The prevailing assumption is that better image-editing quality yields better goals, so goal generators are built and compared using general image-editing metrics. We show this assumption is false. Across two independent goal-conditioned policies on CALVIN, the image editors that top aggregate editing-quality leaderboards (Gemini 3 Pro Image and GPT Image 1.5) produce worse control targets than our proposed far smaller finetuned open models, and the gap widens sharply for the multi-step (long-horizon) tasks robots actually face. A per-dimension analysis explains why. Downstream policy success is governed almost entirely by manipulation adherence (selecting the right object and the right final state; Spearman $ρ\approx 0.9$ with success), while photorealism and aesthetic fidelity, precisely where large proprietary editors excel, are uncorrelated with success ( $ρ\approx 0$ ). This dissociation holds on both CALVIN and the real-world-aligned SIMPLER-Bridge benchmark. We then present SGE-Goal, a spatially-grounded decomposed editing framework that predicts initial and final 2D edit-region masks before goal-image synthesis. SGE-Goal turns open base models (FLUX.1 Kontext, FLUX.2 klein) into the strongest goal generators we evaluate on CALVIN, and second only to Gemini 3 Pro Image on SIMPLER-Bridge. Base models fine-tuned with SGE-Goal also hold up under chaining far better than the best proprietary editor, sustaining roughly seven times the success rate at the longest (five-step) chains. We release code, datasets, checkpoints, and a human-aligned diagnostic evaluator. Our results argue that goal generators must be selected by downstream utility, not by editing-quality leaderboards.

Figures & tables

Figure 1: Goal images are control targets, not artworks. General-purpose editors try optimizing for editing quality in general, but for robotic manipulation a goal image must primarily (i) manipulate the correct object to the correct final state and (ii) avoid spurious changes that mislead the policy. SGE-Goal achieves this through explicit spatial grounding, while strong general editors introduce unmentioned changes, miss the instruction, or hallucinate objects.

Figure 2: Downstream policy success vs. chain length for two pre-trained policies on CALVIN. SGE-Goal goal images (green) yield the best success rates at every chain length and retain performance as chain length increases, while the strongest general editors collapse: at chain length 5, SGE-Goal (FLUX.1 Kontext [Dev]) reaches

30.9\%

vs. Gemini 3 Pro’s

4.4\%

under GR-MG. Long-horizon manipulation is where goal-image utility matters most. Values are previewed here and tabulated in full in Tab. 2 ; base

{=}

untuned editor.

Figure 3: Automated data-extraction pipeline. Each robot video is processed by a chain of vision foundation models to produce spatially-grounded training tuples $(I_{o},I_{e},M_{i},M_{f},T)$ without manual annotation.

Figure 4: SGE-Goal synthesizes the goal image in three steps with explicit 2D spatial grounding via the initial mask $M_{i}$ and final mask $M_{f}$ .

	Adher.	Obj. Pres.	Sc. Art.	Realism	Avg.	I2I-Bench
	(R14–15)	(R1–4)	(R5–12)	(R13)	(R1–15)
Align. (%)	85.5	90.3	92.8	100.0	91.7	71.2

Table 1: Agreement with human judgment (%,

\uparrow

higher is better). Our evaluator is far more human-aligned than existing automated frameworks; this makes it a trustworthy diagnostic, though downstream policy success remains the ground truth for utility.

Figure 5: Automatically extracted training data is high quality. Human quality ratings ( $1$ – $5$ ) for the four components of each extracted tuple: original image $I_{o}$ , edited image $I_{e}$ , initial mask $M_{i}$ , final mask $M_{f}$ , across the three datasets ( $50$ annotators, $500$ tuples/dataset). Diamonds mark per-component means; the lowest is $4.44/5$ .

Method	GR-MG [ 19 ]					GHIL-Glue [ 13 ]
Method	1	2	3	4	5	1	2	3	4	5
GT goal (oracle)	73.6	–	–	–	–	39.0	–	–	–	–
SGE-Goal (FK)	73.1	59.7	47.3	36.8	30.9	29.2	12.0	3.7	0.8	0.0
SGE-Goal (F2K)	71.8	53.8	34.8	21.4	17.3	22.8	8.1	1.5	0.1	0.0
Gemini 3 Pro Image	63.3	36.9	18.1	8.1	4.4	13.2	2.4	0.0	0.0	0.0
Clover	63.2	34.2	12.8	3.6	1.9	17.3	3.2	0.2	0.0	0.0

Table 2: Downstream policy success (SR%,

\uparrow

) on CALVIN , varying only the goal generator, across all chain lengths for two pre-trained, frozen goal-conditioned policies. SGE-Goal yields the best policy at every chain length under both policies, matches the GR-MG oracle at CL

{=}1

while approaching the GHIL-Glue oracle, and degrades most gracefully. Columns are chain lengths

1

–

5

under each policy. FK

{=}

FLUX.1 Kontext [Dev], F2K

{=}

FLUX.2 [klein], CL

{=}

chain length, base

{=}

untuned editor. Best (excl. oracle) per column in bold; SGE-Goal rows shaded; “–” not run.

Figure 6: Chained multi-step prediction by SGE-Goal. The generated goal at each step becomes the input for the next. Scene structure is preserved across all five steps and only the instructed objects move, limiting the drift that destroys long-horizon performance for entangled editors.

Method	Spoon	Carrot	Stack	Eggpl.	Mean
Gemini 3 Pro Image	16.7	29.2	0.0	4.2	12.5
SGE-Goal (F2K)	16.7	16.7	0.0	8.3	10.4
SGE-Goal (FK)	8.3	20.8	0.0	4.2	8.3
SuSIE	16.7	8.3	4.2	0.0	7.3
Qwen Image Edit	8.3	4.2	0.0	0.0	3.1
GPT Image 1.5	4.2	0.0	0.0	4.2	2.1

Table 3: Downstream policy success on SIMPLER-Bridge (SR%,

\uparrow

higher is better). A real-world-aligned WidowX/BridgeData-V2 simulator ( Sec. 4.1 ) under the frozen GHIL-Glue [ 13 ] goal-conditioned policy, across four atomic tasks (24 episodes each). SGE-Goal is the strongest open generator and second overall, behind only Gemini 3 Pro Image; both SGE-Goal variants surpass SuSIE despite SuSIE’s home-field advantage, since GHIL-Glue is SuSIE’s own low-level policy. Best per column in bold; SGE-Goal rows shaded; base

{=}

untuned editor; F2K

{=}

FLUX.2 klein, FK

{=}

FLUX.1 Kontext.

Conditioning	CL1	CL2	CL3	CL4	CL5
Image ${+}$ text (I ${+}$ T)	73.1	59.7	47.3	36.8	30.9
Text only (T)	59.0	37.8	13.1	2.0	0.2

Table 4: Visual goals beat text goals at every chain length. GR-MG success rate (SR%,

\uparrow

) under image

{+}

text (I

{+}

T) vs. text-only (T) goal conditioning; the visual goal’s benefit grows with chain length. I

{+}

T uses an SGE-Goal (FLUX.1 Kontext) goal image. This corroborates and extends MUTEX [ 29 ] , which on LIBERO-100 reports (single step) image-only

61.6\%>

{+}

59.2\%>

text-only

50.1\%

. CL

{=}

chain length.

Inference (A10G)		Data generation (A6000)
Component	Time	Stage	Time
Initial mask ( $M_{i}$ )	2.47	EVF-SAM + BLIP-2	4.46
Final mask ( $M_{f}$ )	5.16	SAM (first frame)	1.07
Goal image ( $\hat{I}_{e}$ )	2.92	XMem tracking	7.76
Total	10.55	LoFTR + KMeans	8.74

Table 5: Runtime (seconds,

\downarrow

lower is better). End-to-end goal-image inference is

{\sim}10.6

s on a single A10G GPU, entirely on open models, with no proprietary API. The one-time, offline data-generation pipeline (A6000) is separate from inference.

Figure 7: Qualitative goal-image comparison. For each instruction, the goal image produced by SGE-Goal (Ours, FLUX.1 Kontext) and five baselines. SGE-Goal moves the correct object to the correct final state while preserving the rest of the scene, whereas proprietary editors (Gemini 3 Pro Image, GPT Image 1.5) render more photorealistic frames but often miss the target object or alter unrelated content. As we show next, this photorealism does not translate into downstream success, while adherence does ( Tab. 7 ).

	Our eval, rank ( $\downarrow$ )			I2I-Bench, score ( $\uparrow$ )
Method	Br	Ca	Li	Br	Ca	Li
Gemini 3 Pro Image	2.07	2.65	1.81	0.785	0.707	0.675
GPT Image 1.5	5.15	5.09	3.04	0.801	0.716	0.774
Qwen Image Edit	4.14	4.17	2.06	0.646	0.642	0.680
SGE-Goal (FLUX.2 klein)	2.99	3.41	4.98	0.655	0.626	0.484
SGE-Goal (FLUX.1 Kont.)	3.81	3.18	5.54	0.649	0.616	0.479

Table 6: Image-editing quality on BridgeDataV2 (Br), CALVIN (Ca), LIBERO (Li): our human-aligned evaluator (weighted rank,

\downarrow

) and I2I-Bench (weighted score,

\uparrow

). By editing quality, the proprietary editors lead (Gemini tops our evaluator, GPT tops I2I-Bench) with SGE-Goal close behind on Br/Ca, yet both are only mid-pack downstream— the metrics invert ( Tab. 2 ). Best per column in bold; SGE-Goal rows shaded. “Kont.” abbreviates Kontext.

	CALVIN $\|\rho\|$		Bridge
Dimension	GR-MG	GHIL	$\|\rho\|$
Adherence
Source adherence (R14)	0.92	0.95	0.85
Destination adherence (R15)	0.90	0.88	0.66
Photorealism
Photorealism (R13)	0.25	0.07	0.02

Table 7: Adherence/photorealism dissociation. Strength of correlation (absolute Spearman

|\rho|

; higher means more predictive) between goal-image adherence or photorealism and downstream policy success. CALVIN values are averaged over chain lengths

1

–

5

for the GR-MG and GHIL-Glue (GHIL) policies; SIMPLER-Bridge is atomic. Adherence strongly predicts success on every benchmark, whereas photorealism does not. The full 15-dimension version is in the supplement.

	CALVIN (SR% at chain length)										SIMPLER- Bridge
	GR-MG					GHIL-Glue
	1	2	3	4	5	1	2	3	4	5
FLUX.1 Kontext
SGE-Goal (full)	73.1	59.7	47.3	36.8	30.9	29.2	12.0	3.7	0.8	0.0	8.3
w/o Final Mask	60.1	28.4	12.9	5.7	3.5	13.8	0.3	0.0	0.0	0.0	5.2
w/o Masks	50.7	21.3	7.5	2.8	1.1	7.3	0.0	0.0	0.0	0.0	5.2

Table 8: Ablation by downstream success rate (SR%,

\uparrow

higher is better). Each spatial component improves the policy on CALVIN, under both the GR-MG and GHIL-Glue policies across all chain lengths (

1

–

5

), and on SIMPLER-Bridge; the masks matter most under long-horizon chaining. “full” uses both masks and the static-scene loss; base

{=}

untuned editor. Best per block in bold; SGE-Goal rows shaded.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Training
Initialization	Octo-Base (released checkpoint)
Fine-tuned parameters	all
Data	CALVIN D training split, $1{,}214$
	annotated windows ( ${\sim}64$ k frames),
	all $34$ tasks ( $35$ – $37$ each)
Cameras	static $256^{2}$ , gripper $128^{2}$

Appendix

Table S1 : Octo-ft fine-tuning and evaluation recipe. The policy is trained only on real CALVIN frames and never sees any generator’s output.

Method	1	2	3	4	5
GT goal (oracle)	36.2	17.1	6.2	1.4	0.3
SGE-Goal (FK)	34.7	14.8	5.0	1.1	0.3
SGE-Goal (F2K)	35.6	16.7	5.4	1.3	0.3
Gemini 3 Pro Image	21.0	6.7	1.4	0.2	0.0
Clover	21.3	3.1	0.2	0.0	0.0
GPT Image 1.5	21.0	2.1	0.3	0.0	0.0

Appendix

Table S2 : A third CALVIN policy: Octo fine-tuned on CALVIN (Octo-ft) , success rate (SR%,

\uparrow

) at chain lengths

1

–

5

, conditioned on the goal image only. Same

1{,}087

test rows, seed and rollout protocol as the GR-MG and GHIL-Glue results in the main paper. The two SGE-Goal variants lead at every chain length and are the only generators close to the ground-truth oracle. FK

{=}

FLUX.1 Kontext, F2K

{=}

FLUX.2 klein, base

{=}

untuned editor. Best (excl. oracle) per column in bold; SGE-Goal rows shaded.

Figure S1 : Downstream success vs. chain length under all three CALVIN policies : the two pre-trained policies of the main paper, (a) GR-MG and (b) GHIL-Glue, and (c) our CALVIN-fine-tuned Octo (Octo-ft), conditioned on the goal image only. Same test rows, chain skeletons and aggregation in all three panels. Under every policy the SGE-Goal goal images (green) give the highest success at every chain length at which any generator succeeds (all reach

0\%

under GHIL-Glue at CL

{=}5

), while the other generators fall towards zero.

Method	Goal image + instruction	Goal image only
GT goal (oracle)	50.1	36.2
SGE-Goal (FK)	49.3	34.7
SGE-Goal (F2K)	48.6	35.6
Gemini 3 Pro Image	45.6	21.0
Clover	48.1	21.3
GPT Image 1.5	45.9	21.0

Appendix

Table S3 : Language masks the goal image. Octo-ft success at CL

{=}1

(SR%,

\uparrow

) when conditioned on the goal image together with the language instruction, and on the goal image alone. With the instruction, the policy solves most tasks from language and the goal generator barely matters (spread

4.0

points); without it, generators separate by

20.0

points. We therefore report goal-image-only results. Best (excl. oracle) per column in bold; SGE-Goal rows shaded.

	1	2	3	4	5
FLUX.1 Kontext
SGE-Goal (full)	34.7	14.8	5.0	1.1	0.3
w/o Final Mask	19.8	6.5	1.2	0.1	0.0
w/o Masks	15.1	2.4	0.2	0.0	0.0
base	19.7	0.9	0.1	0.0	0.0
FLUX.2 klein

Appendix

Table S4 : Mask ablation under Octo-ft (SR%,

\uparrow

), goal image only, at chain lengths

1

–

5

; same test rows and protocol as the GR-MG and GHIL-Glue ablation in the main paper. “full” uses both masks and the static-scene loss; base

{=}

untuned editor. Best per block in bold; SGE-Goal rows shaded.

			$\|\rho\|$ with SR
			CALVIN			SIMPLER- Bridge
	Dimension	Group	GR-MG	GHIL	Octo-ft	SIMPLER- Bridge
R14	Source adherence	Adher.	0.92	0.95	0.87	0.85
R15	Destination adherence	Adher.	0.90	0.88	0.88	0.66
R1	Unmentioned stationary obj.	Consist.	0.53	0.53	0.47	0.49
R2	Unmentioned moved obj.	Consist.	0.23	0.57	0.32	0.17

Appendix

Table S5 : Correlation strength (absolute Spearman

|\rho|

; higher means more predictive) of each dimension with downstream success, for the three CALVIN policies (avg. over CL

{=}1

–

5

) and for SIMPLER-Bridge. Adherence (R14/R15) is the only consistently strong predictor on all four; consistency dimensions are mixed and inconsistent across benchmarks, and photorealism (R13) is weak everywhere. GHIL = GHIL-Glue; Octo-ft = Octo fine-tuned on CALVIN, conditioned on the goal image only.

				CALVIN SR (%)
Method	R14	R15	R13	GR-MG	GHIL	Octo-ft
SGE-Goal (FLUX.1 Kontext)	0.361	0.461	0.433	49.6	9.1	11.2
SGE-Goal (FLUX.2 klein)	0.348	0.469	0.224	39.8	6.5	11.8
Gemini 3 Pro Image	0.350	0.459	0.688	26.2	3.1	5.9
GPT Image 1.5	0.318	0.291	0.662	24.7	1.7	4.7
Clover	0.319	0.362	0.014	23.1	4.1	4.9

Appendix

Table S6 : Raw rates (CALVIN, our evaluator) vs. CALVIN success. Per-method R14 (source adherence), R15 (destination adherence), R13 (photorealism), and mean policy success over CL

{=}1

–

5

for the three CALVIN policies, GR-MG, GHIL-Glue and Octo-ft (our CALVIN-fine-tuned Octo policy, conditioned on the goal image only). SGE-Goal leads on adherence but trails on photorealism, and its two variants are the top two generators under all three policies; the photorealism leader (Qwen) sits near the bottom downstream—only adherence tracks success. Best per column in bold; SGE-Goal rows shaded.

Method	R14	R15	R13	SIMPLER SR (%)
Gemini 3 Pro Image	0.431	0.558	0.915	12.5
SGE-Goal (FLUX.2 klein)	0.517	0.451	0.342	10.4
SGE-Goal (FLUX.1 Kontext)	0.514	0.417	0.591	8.3
SuSIE	0.374	0.373	0.230	7.3
Qwen Image Edit	0.337	0.453	0.898	3.1
GPT Image 1.5	0.336	0.457	0.821	2.1

Appendix

Table S7 : Raw rates (BridgeDataV2, our evaluator) vs. SIMPLER-Bridge success. Per-method R14 (source adherence), R15 (destination adherence), R13 (photorealism), and mean SIMPLER-Bridge policy success. The dissociation holds again: high-photorealism editors collapse downstream (Qwen, GPT, and the FLUX bases all score

{\leq}3.1

), so photorealism does not track success—adherence does. Gemini leads adherence (R15) and is the only generator to outscore SGE-Goal; SGE-Goal leads source adherence (R14) and, despite a modest photorealism rate, ranks a close second overall, above every base editor and open baseline. Best per column in bold; SGE-Goal rows shaded; Clover is CALVIN-only and omitted.

Figure S2 : Full per-dimension dissociation across all downstream success-rate sources. Correlation strength (absolute Spearman

|\rho|

; higher means more predictive) of each of the 15 evaluation dimensions with downstream policy success, for the three CALVIN policies (GR-MG, GHIL-Glue, and Octo-ft conditioned on the goal image only) and for real-world-aligned SIMPLER-Bridge. Adherence (R14, R15; green band) is the only group that is strongly and consistently predictive across all four sources; photorealism (R13; orange band) is weak everywhere, and the remaining consistency and scene dimensions are mixed and do not agree across benchmarks. This is the detailed, fully-labelled version of Tab. S5 .

Figure S3 : Failure cases , all real test-split examples. (a) Image-editing pipeline. (a1) Step 1 grounds a same-coloured distractor (yellow pepper) instead of the occluded target (yellow duck in the box). (a2, a3) Step 2 places the object in the wrong spot: the fork on the pot instead of to its left (a2), and the cloth on the right burner instead of the left one (a3). (a4–a6) Step 3, given the dataset masks: a deformable cloth rendered as a translucent double exposure (a4); a hallucinated magenta block where the grasped blue block was (a5); the target block left in place while the arm is altered (a6). Masks are outlined, with the rest of the image dimmed: magenta = predicted by SGE-Goal, cyan = from the dataset (dataset masks also include the robot arm). Steps 1–2 use the FLUX.2 klein pipeline on BridgeDataV2; Step 3 uses FLUX.1 Kontext. (b) Data-generation pipeline , showing extracted training tuples. (b1) The faucet’s rotation is missed by motion clustering, so both masks contain only the arm. (b2) The edited frame is taken while the duck is still held by the gripper. (b3) A stationary burner grate is labelled dynamic and enters both masks.

Training
Base editors	FLUX.1 Kontext [Dev], FLUX.2 klein
LoRA rank / $\alpha$ / dropout	$16$ / $16$ / $0$
Optimizer	8-bit AdamW
Learning rate	$1{\times}10^{-4}$ , constant ( $200$ warmup)
Batch size	$160$
Gradient accumulation	$1$

Appendix

Table S8 : Full training and inference hyperparameters for SGE-Goal’s LoRA fine-tuning (Steps 2–3). The recipe is shared across all runs; only the listed per-stage, per-editor and per-dataset settings differ, except for the two FLUX.1 Kontext LIBERO runs ( † ): Step 2 with guidance

3.5

, batch size

43

, no warmup and seed

42

; Step 3 with batch size

72

(

2{\times}36

) and no warmup. Each run trains for up to

75

epochs and saves a checkpoint every epoch; the epoch of the checkpoint used is listed at the bottom. Br

{=}

BridgeDataV2, Ca

{=}

CALVIN, Li

{=}

LIBERO; FK

{=}

FLUX.1 Kontext, F2K

{=}

FLUX.2 klein. Here

\tau_{\text{rec}}

gates the auxiliary consistency loss to low-noise samples (

\sigma\!<\!\tau_{\text{rec}}

), whose one-step clean estimate decodes reliably; fraction is the share of those gated samples per batch on which the (decode-based) auxiliary loss is actually computed (

1.0{=}

all).

Figure S4 : Validation loss for SGE-Goal fine-tuning. Total validation loss—the rectified-flow loss plus the weighted auxiliary consistency loss (channel-consistency,

\lambda{=}25

, for Step 2; static-scene-consistency,

\lambda{=}7

, for Step 3)—on the held-out validation split as fine-tuning progresses, for both fine-tuned stages (Step 2 final-mask predictor, Step 3 goal-image synthesizer) and both base editors (FLUX.1 Kontext, FLUX.2 klein), across all three processed datasets. The total validation loss decreases for every run, indicating that the models keep improving on held-out data without overfitting. The

x

-axis is normalized by each run’s length; validation uses the val split of each processed dataset.

	I2I-Bench score ( $\uparrow$ )
Method	Bridge	CALVIN	LIBERO
GPT Image 1.5	0.801	0.716	0.774
Gemini 3 Pro Image	0.785	0.707	0.675
SGE-Goal (FLUX.2 klein)	0.655	0.626	0.484
SGE-Goal (FLUX.1 Kont.)	0.649	0.616	0.479
Qwen Image Edit	0.646	0.642	0.680

Appendix

Table S9 : Weighted average of I2I-Bench Single-Reference dimension scores (

\uparrow

) on BridgeDataV2-teleoperated (Bridge), CALVIN-ABCD (CALVIN), and LIBERO-10-90-Combined (LIBERO). Best per column in bold (proprietary editors top this metric); SGE-Goal rows shaded. “Kont.” abbreviates Kontext.

Figure S5 : Additional subjective comparisons on BridgeDataV2-teleoperated. SGE-Goal (FLUX.2 [klein]) applies consistent, instruction-adherent edits.

Figure S6 : Subjective comparisons on CALVIN-ABCD. SGE-Goal (FLUX.2 [klein]) applies consistent, instruction-adherent edits.

Figure S7 : Subjective comparisons on LIBERO-10-90-Combined. SGE-Goal (FLUX.2 [klein]) applies consistent, instruction-adherent edits.

Configuration	Bridge ( $\downarrow$ )	CALVIN ( $\downarrow$ )
FLUX.1 Kontext [Dev]
SGE-Goal (full)	1.89	1.67
w/o final mask	2.01	2.19
w/o masks	2.97	2.83
base	3.13	3.32
FLUX.2 [klein]

Appendix

Table S10 : Editing-metric ablation under our 15-dimension evaluator (weighted average rank,

\downarrow

lower is better), on BridgeDataV2 and CALVIN.

Configuration	Bridge ( $\uparrow$ )	CALVIN ( $\uparrow$ )
FLUX.1 Kontext [Dev]
SGE-Goal (full)	0.649	0.616
w/o final mask	0.585	0.557
w/o masks	0.526	0.513
base	0.533	0.449
FLUX.2 [klein]

Appendix

Table S11 : I2I-Bench ablation (weighted score,

\uparrow

higher is better) on BridgeDataV2 and CALVIN. Mask guidance helps under the external metric on both datasets.

Figure S8 : Training data samples from BridgeDataV2-teleoperated generated by our automated data extraction pipeline. Each sample consists of the original image $I_{o}$ , initial edit-region mask $M_{i}$ , final edit-region mask $M_{f}$ , edited image $I_{e}$ , and the manipulation instruction $T$ .

Figure S9 : Training data samples from CALVIN-ABCD generated by our automated data extraction pipeline. Each sample consists of the original image $I_{o}$ , initial edit-region mask $M_{i}$ , final edit-region mask $M_{f}$ , edited image $I_{e}$ , and the manipulation instruction $T$ .

Figure S10 : Training data samples from LIBERO-10-90-Combined generated by our automated data extraction pipeline. Each sample consists of the original image $I_{o}$ , initial edit-region mask $M_{i}$ , final edit-region mask $M_{f}$ , edited image $I_{e}$ , and the manipulation instruction $T$ .

Goal-Conditioned Robotic Manipulation

See all 20 papers

Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation
Oct 7, 2026Tzu-Yu Chuang, Ching-Hsiang Chang, Yi-Hsiu Lee +3Robotic Manipulation Robot Task Planning
National Yang Ming Chiao Tung University · National Tsing Hua University
IM-ENGINE: Image Editing for Embodied Data Generation
Sep 5, 2026Yian Wang, Junyi Cao, Xiaowen Qiu +1Synthetic Data Generation for Robotics Robotic Manipulation
University of Massachusetts Amherst · Genesis AI
Imagine2Act: Leveraging Object-Action Motion Consistency from Imagined Goals for Robotic Manipulation
Sep 21, 2025Liang Heng, Jiadong Xu, Yiwen Wang +6Robot Skill Learning Language-Conditioned Robot Manipulation
CFCS, School of Computer Science, Peking University · PrimeBot · AGIBOT

Oct 7, 2026J/K move · Enter open · S save

Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation

Tzu-Yu Chuang, Ching-Hsiang Chang, Yi-Hsiu Lee, Yi-Ting Chen+2

National Yang Ming Chiao Tung University · National Tsing Hua University

Robotic Manipulation Robot Task Planning Goal-Conditioned Robotic Manipulation World Models for Robotics Model-Based Planning

Open paper AskPDF arXiv

Generative world models provide rich predictions of how manipulation scenes may evolve toward task objectives, yet those futures do not directly expose the compact task variables required by control. When training supervises future prediction alone, terminal goal accuracy is not an explicit learning objective, even when geometric recovery is available. We present Entity-Level Goal Readout, a learned prediction-to-execution interface that makes the executable terminal goal an explicit output of a 3D trace world model. It combines object-centric pose prediction with translation grounded in observed depth to produce a compact goal in SE(3). A shared Pose-Native Executor consumes this fixed goal with online object-pose feedback for closed-loop control without rerunning the world model. Across five manipulation tasks, the pipeline achieves a mean success rate of 79.69%. Goal diagnostics directly measure terminal goal accuracy, while controlled translation perturbations characterize how execution degrades under goal error. Zero-shot deployment on a Franka arm achieves 73.33% success on nominal StackCube, 66.67% with distractors, and 75.00% on PickPlate with a target unseen during policy training. These results support treating the prediction-to-execution interface as an explicit learned component of world-model planning rather than incidental post-processing in the control pipeline itself. Project page: https://claire0730.github.io/executable-goals/

Abstract

Goal-Conditioned Robotic Manipulation

Explore similar work

Predicted Futures Are Not Enough: Learning Executable Goals for Robot Manipulation

IM-ENGINE: Image Editing for Embodied Data Generation

Imagine2Act: Leveraging Object-Action Motion Consistency from Imagined Goals for Robotic Manipulation