Goal-conditioned manipulation policies are steered by a goal image, a picture of the desired post-manipulation scene synthesized from the current observation and a language instruction. The prevailing assumption is that better image-editing quality yields better goals, so goal generators are built and compared using general image-editing metrics. We show this assumption is false. Across two independent goal-conditioned policies on CALVIN, the image editors that top aggregate editing-quality leaderboards (Gemini 3 Pro Image and GPT Image 1.5) produce worse control targets than our proposed far smaller finetuned open models, and the gap widens sharply for the multi-step (long-horizon) tasks robots actually face. A per-dimension analysis explains why. Downstream policy success is governed almost entirely by manipulation adherence (selecting the right object and the right final state; Spearman ρ≈0.9 with success), while photorealism and aesthetic fidelity, precisely where large proprietary editors excel, are uncorrelated with success (ρ≈0). This dissociation holds on both CALVIN and the real-world-aligned SIMPLER-Bridge benchmark. We then present SGE-Goal, a spatially-grounded decomposed editing framework that predicts initial and final 2D edit-region masks before goal-image synthesis. SGE-Goal turns open base models (FLUX.1 Kontext, FLUX.2 klein) into the strongest goal generators we evaluate on CALVIN, and second only to Gemini 3 Pro Image on SIMPLER-Bridge. Base models fine-tuned with SGE-Goal also hold up under chaining far better than the best proprietary editor, sustaining roughly seven times the success rate at the longest (five-step) chains. We release code, datasets, checkpoints, and a human-aligned diagnostic evaluator. Our results argue that goal generators must be selected by downstream utility, not by editing-quality leaderboards.
Figures & tables
Figure 1: Goal images are control targets, not artworks. General-purpose editors try optimizing for editing quality in general, but for robotic manipulation a goal image must primarily (i) manipulate the correct object to the correct final state and (ii) avoid spurious changes that mislead the policy. SGE-Goal achieves this through explicit spatial grounding, while strong general editors introduce unmentioned changes, miss the instruction, or hallucinate objects.
Figure 2: Downstream policy success vs. chain length for two pre-trained policies on CALVIN. SGE-Goal goal images (green) yield the best success rates at every chain length and retain performance as chain length increases, while the strongest general editors collapse: at chain length 5, SGE-Goal (FLUX.1 Kontext [Dev]) reaches 30.9% vs. Gemini 3 Pro’s 4.4% under GR-MG. Long-horizon manipulation is where goal-image utility matters most. Values are previewed here and tabulated in full in Tab. 2 ; base = untuned editor.
Figure 3: Automated data-extraction pipeline. Each robot video is processed by a chain of vision foundation models to produce spatially-grounded training tuples (Io,Ie,Mi,Mf,T) without manual annotation.
Figure 4: SGE-Goal synthesizes the goal image in three steps with explicit 2D spatial grounding via the initial mask Mi and final mask Mf .
Adher.
Obj. Pres.
Sc. Art.
Realism
Avg.
I2I-Bench
(R14–15)
(R1–4)
(R5–12)
(R13)
(R1–15)
Align. (%)
85.5
90.3
92.8
100.0
91.7
71.2
Table 1: Agreement with human judgment (%, ↑ higher is better). Our evaluator is far more human-aligned than existing automated frameworks; this makes it a trustworthy diagnostic, though downstream policy success remains the ground truth for utility.
Figure 5: Automatically extracted training data is high quality. Human quality ratings ( 1 – 5 ) for the four components of each extracted tuple: original image Io , edited image Ie , initial mask Mi , final mask Mf , across the three datasets ( 50 annotators, 500 tuples/dataset). Diamonds mark per-component means; the lowest is 4.44/5 .
Method
GR-MG [ 19 ]
GHIL-Glue [ 13 ]
1
2
3
4
5
1
2
3
4
5
GT goal (oracle)
73.6
–
–
–
–
39.0
–
–
–
–
SGE-Goal (FK)
73.1
59.7
47.3
36.8
30.9
29.2
12.0
3.7
0.8
0.0
SGE-Goal (F2K)
71.8
53.8
34.8
21.4
17.3
22.8
8.1
1.5
0.1
0.0
Gemini 3 Pro Image
63.3
36.9
18.1
8.1
4.4
13.2
2.4
0.0
0.0
0.0
Clover
63.2
34.2
12.8
3.6
1.9
17.3
3.2
0.2
0.0
0.0
Table 2: Downstream policy success (SR%, ↑ ) on CALVIN , varying only the goal generator, across all chain lengths for two pre-trained, frozen goal-conditioned policies. SGE-Goal yields the best policy at every chain length under both policies, matches the GR-MG oracle at CL =1 while approaching the GHIL-Glue oracle, and degrades most gracefully. Columns are chain lengths 1 – 5 under each policy. FK = FLUX.1 Kontext [Dev], F2K = FLUX.2 [klein], CL = chain length, base = untuned editor. Best (excl. oracle) per column in bold; SGE-Goal rows shaded; “–” not run.
Figure 6: Chained multi-step prediction by SGE-Goal. The generated goal at each step becomes the input for the next. Scene structure is preserved across all five steps and only the instructed objects move, limiting the drift that destroys long-horizon performance for entangled editors.
Method
Spoon
Carrot
Stack
Eggpl.
Mean
Gemini 3 Pro Image
16.7
29.2
0.0
4.2
12.5
SGE-Goal (F2K)
16.7
16.7
0.0
8.3
10.4
SGE-Goal (FK)
8.3
20.8
0.0
4.2
8.3
SuSIE
16.7
8.3
4.2
0.0
7.3
Qwen Image Edit
8.3
4.2
0.0
0.0
3.1
GPT Image 1.5
4.2
0.0
0.0
4.2
2.1
Table 3: Downstream policy success on SIMPLER-Bridge (SR%, ↑ higher is better). A real-world-aligned WidowX/BridgeData-V2 simulator ( Sec. 4.1 ) under the frozen GHIL-Glue [ 13 ] goal-conditioned policy, across four atomic tasks (24 episodes each). SGE-Goal is the strongest open generator and second overall, behind only Gemini 3 Pro Image; both SGE-Goal variants surpass SuSIE despite SuSIE’s home-field advantage, since GHIL-Glue is SuSIE’s own low-level policy. Best per column in bold; SGE-Goal rows shaded; base = untuned editor; F2K = FLUX.2 klein, FK = FLUX.1 Kontext.
Conditioning
CL1
CL2
CL3
CL4
CL5
Image + text (I + T)
73.1
59.7
47.3
36.8
30.9
Text only (T)
59.0
37.8
13.1
2.0
0.2
Table 4: Visual goals beat text goals at every chain length. GR-MG success rate (SR%, ↑ ) under image + text (I + T) vs. text-only (T) goal conditioning; the visual goal’s benefit grows with chain length. I + T uses an SGE-Goal (FLUX.1 Kontext) goal image. This corroborates and extends MUTEX [ 29 ] , which on LIBERO-100 reports (single step) image-only 61.6%> I + T 59.2%> text-only 50.1% . CL = chain length.
Inference (A10G)
Data generation (A6000)
Component
Time
Stage
Time
Initial mask ( Mi )
2.47
EVF-SAM + BLIP-2
4.46
Final mask ( Mf )
5.16
SAM (first frame)
1.07
Goal image ( I^e )
2.92
XMem tracking
7.76
Total
10.55
LoFTR + KMeans
8.74
Table 5: Runtime (seconds, ↓ lower is better). End-to-end goal-image inference is ∼10.6 s on a single A10G GPU, entirely on open models, with no proprietary API. The one-time, offline data-generation pipeline (A6000) is separate from inference.
Figure 7: Qualitative goal-image comparison. For each instruction, the goal image produced by SGE-Goal (Ours, FLUX.1 Kontext) and five baselines. SGE-Goal moves the correct object to the correct final state while preserving the rest of the scene, whereas proprietary editors (Gemini 3 Pro Image, GPT Image 1.5) render more photorealistic frames but often miss the target object or alter unrelated content. As we show next, this photorealism does not translate into downstream success, while adherence does ( Tab. 7 ).
Our eval, rank ( ↓ )
I2I-Bench, score ( ↑ )
Method
Br
Ca
Li
Br
Ca
Li
Gemini 3 Pro Image
2.07
2.65
1.81
0.785
0.707
0.675
GPT Image 1.5
5.15
5.09
3.04
0.801
0.716
0.774
Qwen Image Edit
4.14
4.17
2.06
0.646
0.642
0.680
SGE-Goal (FLUX.2 klein)
2.99
3.41
4.98
0.655
0.626
0.484
SGE-Goal (FLUX.1 Kont.)
3.81
3.18
5.54
0.649
0.616
0.479
Table 6: Image-editing quality on BridgeDataV2 (Br), CALVIN (Ca), LIBERO (Li): our human-aligned evaluator (weighted rank, ↓ ) and I2I-Bench (weighted score, ↑ ). By editing quality, the proprietary editors lead (Gemini tops our evaluator, GPT tops I2I-Bench) with SGE-Goal close behind on Br/Ca, yet both are only mid-pack downstream— the metrics invert ( Tab. 2 ). Best per column in bold; SGE-Goal rows shaded. “Kont.” abbreviates Kontext.
CALVIN ∣ρ∣
Bridge
Dimension
GR-MG
GHIL
∣ρ∣
Adherence
Source adherence (R14)
0.92
0.95
0.85
Destination adherence (R15)
0.90
0.88
0.66
Photorealism
Photorealism (R13)
0.25
0.07
0.02
Table 7: Adherence/photorealism dissociation. Strength of correlation (absolute Spearman ∣ρ∣ ; higher means more predictive) between goal-image adherence or photorealism and downstream policy success. CALVIN values are averaged over chain lengths 1 – 5 for the GR-MG and GHIL-Glue (GHIL) policies; SIMPLER-Bridge is atomic. Adherence strongly predicts success on every benchmark, whereas photorealism does not. The full 15-dimension version is in the supplement.
CALVIN (SR% at chain length)
SIMPLER- Bridge
GR-MG
GHIL-Glue
1
2
3
4
5
1
2
3
4
5
FLUX.1 Kontext
SGE-Goal (full)
73.1
59.7
47.3
36.8
30.9
29.2
12.0
3.7
0.8
0.0
8.3
w/o Final Mask
60.1
28.4
12.9
5.7
3.5
13.8
0.3
0.0
0.0
0.0
5.2
w/o Masks
50.7
21.3
7.5
2.8
1.1
7.3
0.0
0.0
0.0
0.0
5.2
Table 8: Ablation by downstream success rate (SR%, ↑ higher is better). Each spatial component improves the policy on CALVIN, under both the GR-MG and GHIL-Glue policies across all chain lengths ( 1 – 5 ), and on SIMPLER-Bridge; the masks matter most under long-horizon chaining. “full” uses both masks and the static-scene loss; base = untuned editor. Best per block in bold; SGE-Goal rows shaded.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Training
Initialization
Octo-Base (released checkpoint)
Fine-tuned parameters
all
Data
CALVIN D training split, 1,214
annotated windows ( ∼64 k frames),
all 34 tasks ( 35 – 37 each)
Cameras
static 2562 , gripper 1282
Appendix
Table S1 : Octo-ft fine-tuning and evaluation recipe. The policy is trained only on real CALVIN frames and never sees any generator’s output.
Method
1
2
3
4
5
GT goal (oracle)
36.2
17.1
6.2
1.4
0.3
SGE-Goal (FK)
34.7
14.8
5.0
1.1
0.3
SGE-Goal (F2K)
35.6
16.7
5.4
1.3
0.3
Gemini 3 Pro Image
21.0
6.7
1.4
0.2
0.0
Clover
21.3
3.1
0.2
0.0
0.0
GPT Image 1.5
21.0
2.1
0.3
0.0
0.0
Appendix
Table S2 : A third CALVIN policy: Octo fine-tuned on CALVIN (Octo-ft) , success rate (SR%, ↑ ) at chain lengths 1 – 5 , conditioned on the goal image only. Same 1,087 test rows, seed and rollout protocol as the GR-MG and GHIL-Glue results in the main paper. The two SGE-Goal variants lead at every chain length and are the only generators close to the ground-truth oracle. FK = FLUX.1 Kontext, F2K = FLUX.2 klein, base = untuned editor. Best (excl. oracle) per column in bold; SGE-Goal rows shaded.
Figure S1 : Downstream success vs. chain length under all three CALVIN policies : the two pre-trained policies of the main paper, (a) GR-MG and (b) GHIL-Glue, and (c) our CALVIN-fine-tuned Octo (Octo-ft), conditioned on the goal image only. Same test rows, chain skeletons and aggregation in all three panels. Under every policy the SGE-Goal goal images (green) give the highest success at every chain length at which any generator succeeds (all reach 0% under GHIL-Glue at CL =5 ), while the other generators fall towards zero.
Method
Goal image + instruction
Goal image only
GT goal (oracle)
50.1
36.2
SGE-Goal (FK)
49.3
34.7
SGE-Goal (F2K)
48.6
35.6
Gemini 3 Pro Image
45.6
21.0
Clover
48.1
21.3
GPT Image 1.5
45.9
21.0
Appendix
Table S3 : Language masks the goal image. Octo-ft success at CL =1 (SR%, ↑ ) when conditioned on the goal image together with the language instruction, and on the goal image alone. With the instruction, the policy solves most tasks from language and the goal generator barely matters (spread 4.0 points); without it, generators separate by 20.0 points. We therefore report goal-image-only results. Best (excl. oracle) per column in bold; SGE-Goal rows shaded.
1
2
3
4
5
FLUX.1 Kontext
SGE-Goal (full)
34.7
14.8
5.0
1.1
0.3
w/o Final Mask
19.8
6.5
1.2
0.1
0.0
w/o Masks
15.1
2.4
0.2
0.0
0.0
base
19.7
0.9
0.1
0.0
0.0
FLUX.2 klein
Appendix
Table S4 : Mask ablation under Octo-ft (SR%, ↑ ), goal image only, at chain lengths 1 – 5 ; same test rows and protocol as the GR-MG and GHIL-Glue ablation in the main paper. “full” uses both masks and the static-scene loss; base = untuned editor. Best per block in bold; SGE-Goal rows shaded.
∣ρ∣ with SR
CALVIN
SIMPLER- Bridge
Dimension
Group
GR-MG
GHIL
Octo-ft
R14
Source adherence
Adher.
0.92
0.95
0.87
0.85
R15
Destination adherence
Adher.
0.90
0.88
0.88
0.66
R1
Unmentioned stationary obj.
Consist.
0.53
0.53
0.47
0.49
R2
Unmentioned moved obj.
Consist.
0.23
0.57
0.32
0.17
Appendix
Table S5 : Correlation strength (absolute Spearman ∣ρ∣ ; higher means more predictive) of each dimension with downstream success, for the three CALVIN policies (avg. over CL =1 – 5 ) and for SIMPLER-Bridge. Adherence (R14/R15) is the only consistently strong predictor on all four; consistency dimensions are mixed and inconsistent across benchmarks, and photorealism (R13) is weak everywhere. GHIL = GHIL-Glue; Octo-ft = Octo fine-tuned on CALVIN, conditioned on the goal image only.
CALVIN SR (%)
Method
R14
R15
R13
GR-MG
GHIL
Octo-ft
SGE-Goal (FLUX.1 Kontext)
0.361
0.461
0.433
49.6
9.1
11.2
SGE-Goal (FLUX.2 klein)
0.348
0.469
0.224
39.8
6.5
11.8
Gemini 3 Pro Image
0.350
0.459
0.688
26.2
3.1
5.9
GPT Image 1.5
0.318
0.291
0.662
24.7
1.7
4.7
Clover
0.319
0.362
0.014
23.1
4.1
4.9
Appendix
Table S6 : Raw rates (CALVIN, our evaluator) vs. CALVIN success. Per-method R14 (source adherence), R15 (destination adherence), R13 (photorealism), and mean policy success over CL =1 – 5 for the three CALVIN policies, GR-MG, GHIL-Glue and Octo-ft (our CALVIN-fine-tuned Octo policy, conditioned on the goal image only). SGE-Goal leads on adherence but trails on photorealism, and its two variants are the top two generators under all three policies; the photorealism leader (Qwen) sits near the bottom downstream—only adherence tracks success. Best per column in bold; SGE-Goal rows shaded.
Method
R14
R15
R13
SIMPLER SR (%)
Gemini 3 Pro Image
0.431
0.558
0.915
12.5
SGE-Goal (FLUX.2 klein)
0.517
0.451
0.342
10.4
SGE-Goal (FLUX.1 Kontext)
0.514
0.417
0.591
8.3
SuSIE
0.374
0.373
0.230
7.3
Qwen Image Edit
0.337
0.453
0.898
3.1
GPT Image 1.5
0.336
0.457
0.821
2.1
Appendix
Table S7 : Raw rates (BridgeDataV2, our evaluator) vs. SIMPLER-Bridge success. Per-method R14 (source adherence), R15 (destination adherence), R13 (photorealism), and mean SIMPLER-Bridge policy success. The dissociation holds again: high-photorealism editors collapse downstream (Qwen, GPT, and the FLUX bases all score ≤3.1 ), so photorealism does not track success—adherence does. Gemini leads adherence (R15) and is the only generator to outscore SGE-Goal; SGE-Goal leads source adherence (R14) and, despite a modest photorealism rate, ranks a close second overall, above every base editor and open baseline. Best per column in bold; SGE-Goal rows shaded; Clover is CALVIN-only and omitted.
Figure S2 : Full per-dimension dissociation across all downstream success-rate sources. Correlation strength (absolute Spearman ∣ρ∣ ; higher means more predictive) of each of the 15 evaluation dimensions with downstream policy success, for the three CALVIN policies (GR-MG, GHIL-Glue, and Octo-ft conditioned on the goal image only) and for real-world-aligned SIMPLER-Bridge. Adherence (R14, R15; green band) is the only group that is strongly and consistently predictive across all four sources; photorealism (R13; orange band) is weak everywhere, and the remaining consistency and scene dimensions are mixed and do not agree across benchmarks. This is the detailed, fully-labelled version of Tab. S5 .
Figure S3 : Failure cases , all real test-split examples. (a) Image-editing pipeline. (a1) Step 1 grounds a same-coloured distractor (yellow pepper) instead of the occluded target (yellow duck in the box). (a2, a3) Step 2 places the object in the wrong spot: the fork on the pot instead of to its left (a2), and the cloth on the right burner instead of the left one (a3). (a4–a6) Step 3, given the dataset masks: a deformable cloth rendered as a translucent double exposure (a4); a hallucinated magenta block where the grasped blue block was (a5); the target block left in place while the arm is altered (a6). Masks are outlined, with the rest of the image dimmed: magenta = predicted by SGE-Goal, cyan = from the dataset (dataset masks also include the robot arm). Steps 1–2 use the FLUX.2 klein pipeline on BridgeDataV2; Step 3 uses FLUX.1 Kontext. (b) Data-generation pipeline , showing extracted training tuples. (b1) The faucet’s rotation is missed by motion clustering, so both masks contain only the arm. (b2) The edited frame is taken while the duck is still held by the gripper. (b3) A stationary burner grate is labelled dynamic and enters both masks.
Training
Base editors
FLUX.1 Kontext [Dev], FLUX.2 klein
LoRA rank / α / dropout
16 / 16 / 0
Optimizer
8-bit AdamW
Learning rate
1×10−4 , constant ( 200 warmup)
Batch size
160
Gradient accumulation
1
Appendix
Table S8 : Full training and inference hyperparameters for SGE-Goal’s LoRA fine-tuning (Steps 2–3). The recipe is shared across all runs; only the listed per-stage, per-editor and per-dataset settings differ, except for the two FLUX.1 Kontext LIBERO runs ( † ): Step 2 with guidance 3.5 , batch size 43 , no warmup and seed 42 ; Step 3 with batch size 72 ( 2×36 ) and no warmup. Each run trains for up to 75 epochs and saves a checkpoint every epoch; the epoch of the checkpoint used is listed at the bottom. Br = BridgeDataV2, Ca = CALVIN, Li = LIBERO; FK = FLUX.1 Kontext, F2K = FLUX.2 klein. Here τrec gates the auxiliary consistency loss to low-noise samples ( σ<τrec ), whose one-step clean estimate decodes reliably; fraction is the share of those gated samples per batch on which the (decode-based) auxiliary loss is actually computed ( 1.0= all).
Figure S4 : Validation loss for SGE-Goal fine-tuning. Total validation loss—the rectified-flow loss plus the weighted auxiliary consistency loss (channel-consistency, λ=25 , for Step 2; static-scene-consistency, λ=7 , for Step 3)—on the held-out validation split as fine-tuning progresses, for both fine-tuned stages (Step 2 final-mask predictor, Step 3 goal-image synthesizer) and both base editors (FLUX.1 Kontext, FLUX.2 klein), across all three processed datasets. The total validation loss decreases for every run, indicating that the models keep improving on held-out data without overfitting. The x -axis is normalized by each run’s length; validation uses the val split of each processed dataset.
I2I-Bench score ( ↑ )
Method
Bridge
CALVIN
LIBERO
GPT Image 1.5
0.801
0.716
0.774
Gemini 3 Pro Image
0.785
0.707
0.675
SGE-Goal (FLUX.2 klein)
0.655
0.626
0.484
SGE-Goal (FLUX.1 Kont.)
0.649
0.616
0.479
Qwen Image Edit
0.646
0.642
0.680
Appendix
Table S9 : Weighted average of I2I-Bench Single-Reference dimension scores ( ↑ ) on BridgeDataV2-teleoperated (Bridge), CALVIN-ABCD (CALVIN), and LIBERO-10-90-Combined (LIBERO). Best per column in bold (proprietary editors top this metric); SGE-Goal rows shaded. “Kont.” abbreviates Kontext.
Table S10 : Editing-metric ablation under our 15-dimension evaluator (weighted average rank, ↓ lower is better), on BridgeDataV2 and CALVIN.
Configuration
Bridge ( ↑ )
CALVIN ( ↑ )
FLUX.1 Kontext [Dev]
SGE-Goal (full)
0.649
0.616
w/o final mask
0.585
0.557
w/o masks
0.526
0.513
base
0.533
0.449
FLUX.2 [klein]
Appendix
Table S11 : I2I-Bench ablation (weighted score, ↑ higher is better) on BridgeDataV2 and CALVIN. Mask guidance helps under the external metric on both datasets.
Figure S8 : Training data samples from BridgeDataV2-teleoperated generated by our automated data extraction pipeline. Each sample consists of the original image Io , initial edit-region mask Mi , final edit-region mask Mf , edited image Ie , and the manipulation instruction T .
Figure S9 : Training data samples from CALVIN-ABCD generated by our automated data extraction pipeline. Each sample consists of the original image Io , initial edit-region mask Mi , final edit-region mask Mf , edited image Ie , and the manipulation instruction T .
Figure S10 : Training data samples from LIBERO-10-90-Combined generated by our automated data extraction pipeline. Each sample consists of the original image Io , initial edit-region mask Mi , final edit-region mask Mf , edited image Ie , and the manipulation instruction T .
Generative world models provide rich predictions of how manipulation scenes may evolve toward task objectives, yet those futures do not directly expose the compact task variables required by control. When training supervises future prediction alone, terminal goal accuracy is not an explicit learning objective, even when geometric recovery is available. We present Entity-Level Goal Readout, a learned prediction-to-execution interface that makes the executable terminal goal an explicit output of a 3D trace world model. It combines object-centric pose prediction with translation grounded in observed depth to produce a compact goal in SE(3). A shared Pose-Native Executor consumes this fixed goal with online object-pose feedback for closed-loop control without rerunning the world model. Across five manipulation tasks, the pipeline achieves a mean success rate of 79.69%. Goal diagnostics directly measure terminal goal accuracy, while controlled translation perturbations characterize how execution degrades under goal error. Zero-shot deployment on a Franka arm achieves 73.33% success on nominal StackCube, 66.67% with distractors, and 75.00% on PickPlate with a target unseen during policy training. These results support treating the prediction-to-execution interface as an explicit learned component of world-model planning rather than incidental post-processing in the control pipeline itself. Project page: https://claire0730.github.io/executable-goals/
Tzu-Yu Chuang, Ching-Hsiang Chang, Yi-Hsiu Lee +3
National Yang Ming Chiao Tung University · National Tsing Hua University
Learning-based manipulation requires supervision that is both semantically meaningful and physically executable, but current data pipelines often provide only one of these properties. Human demonstrations capture intent but are costly to collect and constrained by the human-robot embodiment gap, while simulation can scale data generation but often under-specifies functional behavior. We present IM-ENGINE, a simulator-grounded pipeline that uses image editing as an intermediate representation for embodied data generation. Given a rendered scene with known geometry, depth, segmentation, and camera parameters, IM-ENGINE edits the image to inject task-relevant semantics, recovers explicit 3D state using simulator priors and an unchanged anchor object, refines the state in physics, and converts it into robot-executable supervision. We instantiate the pipeline for dexterous grasp synthesis and goal-state generation. For grasping, IM-ENGINE generates a human grasp in image space, recovers the hand-object interaction, retargets it to a robot hand, and refines it into physically validated robot grasps. For goal generation, it edits a rendered scene into a desired outcome, recovers the target-object pose, and refines it into physically valid, semantically meaningful goals and trajectories. This combination of generative semantic priors and simulator grounding enables scalable task-relevant supervision for robot learning.
Relational object rearrangement (ROR) tasks (e.g., insert flower to vase) require a robot to manipulate objects with precise semantic and geometric reasoning. Existing approaches either rely on pre-collected demonstrations that struggle to capture complex geometric constraints or generate goal-state observations to capture semantic and geometric knowledge, but fail to explicitly couple object transformation with action prediction, resulting in errors due to generative noise. To address these limitations, we propose Imagine2Act, a 3D imitation-learning framework that incorporates semantic and geometric constraints of objects into policy learning to tackle high-precision manipulation tasks. We first generate imagined goal images conditioned on language instructions and reconstruct corresponding 3D point clouds to provide robust semantic and geometric priors. These imagined goal point clouds serve as additional inputs to the policy model, while an object-action consistency strategy with soft pose supervision explicitly aligns predicted end-effector motion with generated object transformation. This design enables Imagine2Act to reason about semantic and geometric relationships between objects and predict accurate actions across diverse tasks. Experiments in both simulation and the real world demonstrate that Imagine2Act outperforms previous state-of-the-art policies. More visualizations can be found at https://sites.google.com/view/imagine2act.
Liang Heng, Jiadong Xu, Yiwen Wang +6
CFCS, School of Computer Science, Peking University · PrimeBot · AGIBOT