Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Abstract

Figures & tables

World Action Models

Abstract

World Action Models

Explore similar work

Appendix

Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

Riemann-1.0: An Embodied World Action Model for Physical AI

DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

Riemann-1.0: An Embodied World Action Model for Physical AI

DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

Junyi Hu; Zhewen He; Zhenhua Li; Yi Fang

cs.ROOct 1, 2026

Authors: Junyi Hu, Zhewen He, Zhenhua Li, Yi Fang

Organizations: New York University Abu Dhabi · Chatsign Technology

A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.

Figure 1: Keep the effect, drop the actor. A demonstration records how one arm moved and what happened to the world. Copying the demonstrator (top) ties the task to that motion: a published imitator fails the held-out task, replayed actions miss the target after a 5 cm push and fail on a UR5e, and a replay can only grasp as recorded. PEWAM keeps only the effect, as a program of object keypoints, a terminal state and two contact points (bottom), and re-solves the execution from the live scene: the same episode succeeds, the push is absorbed, a UR5e runs it without retraining, and rotating the contact points turns the grasp.

Figure 2: PEWAM end to end. (1) Compiling a demonstration into an effect program (§ 3.1 ); (2) the training data; (3) one model with a flow-matching time per stream (§ 3.2 ); (4) the closed loop of Algorithm 1 . Bottom: contact as points, and three operators of the same loop.

LIBERO, held-out tasks
method	success
UWM ( Zhu et al., 2025 )	21.1
Instant Policy ( Vosylius and Johns, 2025 )	0 5.6
ATM ( Wen et al., 2024 )	0 0.0
Zero-WAM ( Zhou et al., 2026 )	0 0.0
PEWAM (ours)	44.4

Table 1: Held-out tasks from one demonstration , success rate (%). Left: the three held-out LIBERO-Goal tasks, 90 episodes; each published method reads the same demonstration through its own interface: UWM its video frames, Instant Policy its point clouds, ATM its point tracks and Zero-WAM its video; the reference rows replay its actions or give the same weights its end-effector trajectory (Appendix H.1 ). Right: the Meta-World test classes selected by the rule of Appendix B , from ML10 (shelf-place, sweep-into) and ML45 (bin-picking; shelf, sweep and bin in the table); PEWAM is the mean of three trainings, each over 30 episodes programmed by another episode’s demonstration; published policies are fine-tuned on ten demonstrations of the class ( Shang et al., 2026 ) . On the five-class mean PEWAM is below DAMI (49.8 against 56.8 on ML10, 34.0 against 37.5 on ML45; Appendix J ).

Figure 3: One program per held-out task, executed closed loop. Left of each pair: the program on its demonstration with the robot removed (object track white, terminal state $G^{\star}$ yellow, contact points red and blue). Right: the last frame of the execution, with the object’s path in white and its length in steps above. The last pair is a LIBERO-Object episode with a held-out object (the sauce bottle into the basket). Episodes are the lowest-numbered success of each task in its held-out evaluation; failures and intermediate frames are in Appendix O , and the baselines on the same episodes in Appendix Figure 9 .

Figure 4: Editing the program (bowl task). (a) Terminal edits on a $3\times 3$ grid of contact rotation and terminal shift: realised against commanded shift, filled where the task predicate fires. (b) Contact edits: realised grasp rotation against commanded rotation of the two contact points, median of episode-paired shifts with 95% bootstrap bands. Contacts stored as affine weights (red) are read with the wrong sign. (c) The grasp with contacts rotated by $\mp 45^{\circ}$ , from above.

Panel of Figure 5 — Figure 5: Execution is a free variable. (a) One effect, three grasps: seed 2 takes the bowl from the other side, and all three succeed. (b) The bowl’s distance to its programmed end. Re-solving the remaining program absorbs the push; replaying the demonstration’s actions ends 8.0 cm short.

ingredients	contact / corpus	Panda	IIWA	Kinova3	UR5e	bowl	bottle	bowl	bottle
		first training, bowl / bottle				sum		mean sum
neither	— / —	28/18	26/7	17/16	8/0	79	41	78.5	41.0
contact only	yes / —	27/17	21/13	24/10	2/1	74	41	82.0	48.5
corpus only	— / yes	24/7	15/0	11/2	14/1	64	10	63.5	12.0
both (PEWAM)	yes / yes	27/15	24/17	22/5	16/6	89	43	83.3	27.7

Table 2: One program, four arms, no retraining : successes of 30 on the bowl / bottle tasks for the first training of each recipe, factored over the contact channel and the paired multi-solution corpus. Proprioception is re-expressed in the training arm’s joint space, the identity on the Panda. Mean: sums averaged over trainings (two per recipe, three for PEWAM); every training is in Table 8 .

	bowl	bottle	$\Sigma$ , step 32
replay of the demonstration	29 / 6	28 / 0	9
PEWAM	27 / 9	12 / 14	28
without the gate	26 / 3	13 / 4	10

Table 3: A 5 cm push : successes of 30, undisturbed / pushed at step 28, and the pushed sum over both tasks with the push at step 32. All cells are in Table 14 .

Figure 6: Programs compiled from real human videos. Each program is drawn on the demonstration’s first frame, before the hand enters (white: entities at the start; one colour per entity for its route; yellow: its terminal region $G^{\star}$ ), beside the demonstration’s last frame. Programs 1 and 2 share a scene: the same cube carried to a plate and to a box.

task	goal image	PEWAM
cube to plate	8	10
cube into box	5	10
cube onto tray	5	8
box into bin	4	8
total (of 40)	22	36

Table 4: A real Franka arm programmed by human videos , successes of 10. Each program is compiled from one human video; no robot demonstration of these tasks is used. Left: PEWAM against the same backbone and training data conditioned on the video’s last frame as a goal image. Right: PEWAM on the cube-to-plate program as demonstrated (the trials of the left block), with its terminal state shifted by 10 cm, and with the cube pushed 5 cm mid-task.

changed	measure	PEWAM	variant
no contact channel	four arms, bowl / bottle	89 / 43	64 / 10
no paired corpus	contact following	0.68	0.31
demo-clock pointer	held-out bottle / $\Sigma$	12 / 40	1 / 27
native proprioception	four arms, bowl	89	28
relative effect	held-out $\Sigma$	40	1

Table 5: Core ablations : one component changed per row, on the measure it acts on. All variants: Table 9 .

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

split	simulator, robot	held out at test	test episodes	compared with
LIBERO-Goal (§ 4.1 )	MuJoCo, Panda	3 tasks (new goals)	$3\times 30$	other task interfaces; UWM, Instant Policy, ATM, Zero-WAM
LIBERO-Object (§ 4.1 )	MuJoCo, Panda	3 objects	$3\times 30$	same recipe without LIBERO-Object
Meta-World ML10 / ML45 (§ 4.1 )	MuJoCo, Sawyer	5 classes each (benchmark-defined)	30 per class	observation-only reference; published
RLBench (§ 4.1 )	CoppeliaSim, Panda	24 tasks	$24\times 30$	observation-only reference; Instant Policy
four arms (§ 4.3 )	MuJoCo, IIWA / Kinova3 / UR5e	the arm	30 per task	ablations of contact and corpus
real robot (§ 4.4 )	lab table, Franka Research 3, RGB-D	4 tasks from human videos	$4\times 10$	same backbone with a goal image

Appendix

Table 6: Experimental setup. Each test episode is a closed-loop rollout, scored by the benchmark’s own success check in simulation and by the criterion of Appendix L on the real arm. Every held-out episode is programmed by a single demonstration, and no parameters are updated at test time.

	bowl $\to$ stove	bottle $\to$ cabinet	knob	$\Sigma/90$	3 seeds
PEWAM, effect program	27	12	1	40	40, 38, 33
PEWAM, terminal state only	30	19	0	49	49, 50, 32
goal image instead of program	0	0	0	0
language instead of program	0	0	0	0
no program	0	0	0	0
UWM, demo frames ( Zhu et al., 2025 )	0	0	19	19

Appendix

Table 7: Held-out LIBERO-Goal tasks: successes out of 30 per task in every row. The per-task columns come from the first training, which is the checkpoint every other experiment uses; the last column adds two retrainings of the same configuration with new seeds. The second block retrains the same recipe and backbone and changes only the task interface. The last row queries the model with the demonstrator’s own execution instead of its effect. Published methods read the same held-out demonstration through their own interface (Appendices H.1 – H.4 ).

ingredients	contact / corpus	Panda	IIWA	Kinova3	UR5e	$\Sigma$ bowl	$\Sigma$ bottle
first training
neither	— / —	28/18	26/7	17/16	8/0	79	41
contact only	yes / —	27/17	21/13	24/10	2/1	74	41
corpus only	— / yes	24/7	15/0	11/2	14/1	64	10
both (PEWAM)	yes / yes	27/15	24/17	22/5	16/6	89	43
further trainings

Appendix

Table 8: One program, four arms, no retraining : successes out of 30 on the bowl / bottle tasks, split by whether the recipe has the contact channel and the paired multi-solution corpus. Proprioception is re-expressed in the training arm’s joint space, which is the identity on the Panda. Each recipe is trained twice (seeds 0 and 1) and PEWAM three times; the last block averages the sums over trainings.

	effect program				goal	four arms
	bowl	bottle	knob	$\Sigma$	$\Sigma$	$\Sigma$ bowl	$\Sigma$ bottle	following
which queries to train: earlier backbone’s recipe; re-plan every 8 steps, no fixed sampling seed
all six queries (the earlier backbone)	24	12	4	40	–	–	–	–
all six, goal share 0.25 (W5)	16	13	0	29	29	–	–	–
inverse, goal, inverse dynamics (W0)	29	6	0	35	58	–	–	–
W0 + partial program (W2)	24	13	0	37	56	–	–	–

Appendix

Table 9: Ablation variants, one training each unless marked as a second training (seed 1). Effect program: successes out of 30 on the three held-out tasks and their sum; goal: the terminal-state query’s sum over 90; four arms: sums over Panda, IIWA, Kinova3 and UR5e on the bowl and bottle tasks, as in Table 2 ; following: mean over the

\pm 20^{\circ}

and

\pm 45^{\circ}

commanded rotations, as in Table 10 ; † : mean over

\pm 45^{\circ}

only. “–”: not evaluated under that block’s protocol, or (W1, R1) a query the variant never trains. The first block and the goal column were evaluated earlier, partly on an RTX A6000; every other entry was evaluated on RTX 3090s. The first block uses a different protocol from the others.

	successes at commanded rotation							realised / commanded
	$0$	$+20$	$-20$	$+45$	$-45$	$+90$	$-90$	$+20$	$-20$	$+45$	$-45$	mean
PEWAM, seed 0	27	22	24	24	24	22	26	0.64	0.79	0.47	0.82	0.68
PEWAM, seed 1	26	23	21	24	19	24	22	0.38	0.38	0.49	0.46	0.43
PEWAM, seed 2	27	26	22	27	21	26	16	0.73	0.51	0.63	0.49	0.59
W8 (contact, no corpus)	28	26	20	25	11	24	10	0.50	0.24	0.39	0.12	0.31

Appendix

Table 10: Rotating the two contact points about the object axis (task 4,

n{=}30

). Left: successes at each commanded rotation. Right: the fraction of the commanded rotation that the realised grasp achieved, measured by replaying each rollout and reading the gripper contacts. The three PEWAM rows are the same configuration trained with three seeds; W8 is trained without the paired corpus.

Figure 7: Terminal shift under the two queries, bowl task, 30 episodes: mean and standard deviation of the realised shift, measured as in Figure 4 a. The slope is the least-squares fit through the origin over the shifted settings;

r

is the correlation over all 150 episode–shift pairs. (a) The headline model. (b) The earlier backbone, v3.

recipe	query	$-10$	$-5$	0	$+5$	$+10$ cm	slope	$r$
PEWAM	program	8	27	27	28	22	0.71	0.65
	terminal	1	30	30	30	5	1.01	0.99
neither ingredient (W3)	program	10	27	28	27	20	0.80	0.89
	terminal	3	29	30	30	16	0.95	0.96
corpus only (W6)	program	6	26	25	27	24	0.81	0.67
	terminal	5	30	30	30	18	0.90	0.99

Appendix

Table 11: Terminal edits for the training variants of Table 2 (bowl task, first training of each variant, protocol of Table 1 ): successes of 30 at each commanded shift of

G^{\star}

, and the slope of realised against commanded shift (fitted through the origin, with each episode paired against its unshifted run).

	success	median final distance	bowl raised $>3$ cm
reversed program (target: the demonstration’s start)	2	25.7 cm	4
no program (target: the demonstration’s start)	1	24.8 cm	3
forward program (target: the stove; already there)	30	0.0 cm	–

Appendix

Table 12: The bowl is placed at the program’s end (on the stove, within 2.1 mm) before the episode starts. An episode succeeds if the bowl’s centre ends within 5 cm of the target; 30 episodes, protocol of Table 1 .

cell	program / live keypoints	bowl	bottle	$\Sigma$	median (cm)	max (cm)	lost
					live error, bowl / bottle
P0	state / state	27	12	39	–	–	–
P1	state / pixels	16	9	25	0.9 / 3.2	4.0 / 13.2	0 / 1
P2	pixels / pixels	25	15	40	0.6 / 3.4	2.3 / 10.7	1 / 2

Appendix

Table 13: Live keypoints from rendered RGB-D (pre-registration 28): successes of 30, and the error of the live centre of the manipulated entity against simulator state. Error: median over episodes of the per-episode median and of the per-episode maximum. Lost: episodes in which the mask vanished for at least one step.

	bowl $\to$ stove		bottle $\to$ top
	undisturbed	pushed	undisturbed	pushed
replay of the demonstration’s actions	29	6	28	0
effect program	27	9	12	14
re-plan every 8 steps (push at step 32)	–	18	–	–
re-plan every 16 steps (push at step 32)	–	21	–	–
G2: no nearest-neighbour matching before engagement	28	12	14	17

Appendix

Table 14: A 5 cm push on the headline model: successes of 30, undisturbed and pushed. The push lands at step 28 unless noted otherwise. Variant G2 was designed after the results of the first gate had been seen. Effect-program rows with the gate were evaluated on RTX 3090s, the other rows on an RTX A6000.

Figure 8: Episode 201 of the bowl task, pushed 5 cm at step 28 (dotted). (a) The program frame selected at every re-plan. Undisturbed, the pointer without the gate passes the program’s first motion (dashed, frame 48) and the bowl is lifted. Pushed, the pointer jumps from frame 24 to 42 both with and without the gate; with the gate it moves to the program’s engagement frame 49 at step 44, and without it stays near frame 42. (b) The bowl’s height above its start: when pushed, it is never lifted.

Figure 9: Two episodes (bowl episode 210, bottle episode 410) as each demonstration-reading method executes them; every row shows the outcome recorded in that method’s own evaluation. The episodes were picked by the selection rule of Appendix O , which no knob episode satisfies. UWM has no row because it was recorded only for episode 200.

query	middle drawer	top drawer, bowl inside	bowl $\to$ plate	$\Sigma/90$
effect program	0	0	27	27
terminal state only	0	0	29	29

Appendix

Table 15: The second LIBERO-Goal split, with held-out tasks drawn at random: successes out of 30, one training run, table protocol.

model	query	training	bbq sauce	chocolate	milk	$\Sigma/90$
			(bottle)	pudding (box)	(carton)
PEWAM + LIBERO-Object	effect program	1	20	29	18	67
	effect program	2	19	28	18	65
	terminal state only	1	29	30	30	89
	terminal state only	2	25	30	29	84
PEWAM, no object data	effect program	1	0	0	0	0

Appendix

Table 16: The second held-out split: one skill, three objects the weights never manipulated, 30 episodes each. The upper model was trained twice with the same recipe (seeds 0 and 1). Effect-program rows were evaluated on RTX 3090s and terminal-state rows on V100s. The lower pair never saw LIBERO-Object: across its 90 episodes the object’s median displacement is 0.0 cm, so the object stays where it began, a median 17 cm from the target, against 5 cm for the upper pair.

	door-close	drawer-open	lever-pull	shelf-place	sweep-into	mean
Meta-World’s own ( Yu et al., 2019 )	94.0	22.7	5.3	0.0	24.7	29.3
DP3	100.0	70.3	44.7	12.7	16.7	48.9
Mamba	100.0	32.3	27.7	10.0	9.7	35.9
FreqPolicy	93.0	98.0	18.7	7.0	15.7	46.5
FlowPolicy	100.0	13.3	17.7	0.0	6.7	27.5
DAMI ( Shang et al., 2026 )	100.0	100.0	45.3	10.7	28.0	56.8

Appendix

Table 17: Meta-World ML10, five test classes, success rate in per cent. Published rows are as reported by Shang et al. (2026) . Our rows use 30 episodes per class, one training run and a checkpoint fixed before evaluation; the reference is the same recipe trained on the observation alone. Bold marks a result above every published row; the relative rows are those reported in § 4.1 . The seeds 0–2 rows add two more trainings of that recipe (seeds 1 and 2, trained after the row above them had been read) and give the mean per cent and the lowest and highest count out of 30.

	bin-picking	box-close	hand-insert	door-lock	door-unlock	mean
Meta-World’s own ( Yu et al., 2019 )	0.3	1.7	10.3	42.7	7.7	12.5
DP3	9.3	26.7	6.7	51.0	94.0	37.5
Mamba	9.7	12.7	5.3	22.7	34.7	17.0
FreqPolicy	11.0	25.3	13.7	11.3	6.0	13.5
FlowPolicy	9.0	13.3	6.0	15.0	22.0	13.1
DAMI ( Shang et al., 2026 )	13.7	22.3	13.7	39.3	98.7	37.5

Appendix

Table 18: Meta-World ML45, five test classes, success rate in per cent. The upper block reproduces Table III of Shang et al. (2026) : its own method and the five baselines it runs, including the Meta-World RL baseline, at 20 episodes per class on three seeds, averaging the five best checkpoints after 200 adaptation steps on the class. The lower block is this paper, at 30 episodes on one seed and one checkpoint, with no weight change at test time; the transferred layout programs each episode with another episode’s demonstration. The seeds 0–2 rows add two more trainings of the joint-state recipe (seeds 1 and 2, trained after the row above them had been read) and give the mean per cent and the lowest and highest count out of 30. Bold marks a result above every published row.

Appendix

Table 19: Meta-World ML45, five test classes, 30 episodes each, scored by the benchmark’s own success check. Both models share one recipe and schedule, are trained on the 45 training classes and are compared only with each other; the third row is that recipe trained on the observation alone.

	TurnOnStove	OpenSingleDoor	PnPCounterToMicrowave	$\Sigma/90$
effect program	21	24	24	69
terminal state only	18	15	17	50
reference: observation only	14	14	16	44
demonstrator’s execution as query (inverse dynamics)	9	7	11	27
OpenWAM- $\alpha$ , released LIBERO checkpoint, zero-shot (language)	2	5	3	10

Appendix

Table 20: RoboCasa, the three held-out tasks, 30 episodes each, scored by RoboCasa’s own success check. The first four rows share the PEWAM recipe and the pointer of § 3.3 and were evaluated on one A100; the third row is the same recipe trained on the observation alone. The last row is a released checkpoint trained only on LIBERO, run zero-shot on the same episodes (see the OpenWAM-

\alpha

paragraph).

Figure 10: Programs compiled from ten further real human demonstrations. Each panel draws the program on the first frame of the demonstration (white: entities at the start; one colour per entity for its route; yellow: its terminal region). The inset shows the last frame of the same demonstration.

	increment action			absolute pose
	program	relative	reference	program	relative	reference	IP, 1 demo	IP, 2 demos	replay
LampOn	29	12	30	0	25	0	30	30	30
PutRubbishInBin	27	0	2	8	1	0	30	30	30
PushButton	26	6	28	20	12	12	29	30	30
ToiletSeatDown	20	12	14	19	15	11	30	29	30
TakeLidOffSaucepan	17	15	10	0	20	27	29	30	30

Appendix

Table 21: RLBench: successes out of 30 on each of the 24 held-out tasks (720 in all), with one demonstration per episode. The same recipe trains two action heads, one predicting the 7-D end-effector increment (amendment G, the configuration reported in § 4.1 ) and one predicting the next absolute end-effector pose. Program: the effect program. Relative: the effect written as displacements from each entity’s anchor. Reference: the same recipe trained on the observation alone. IP: Instant Policy with one and two demonstrations. Replay: the demonstration’s own absolute-pose actions executed through the same IK (replaying the increments instead completes 616), with CloseJar and OpenJar using the gripper state from the grasp record. Every evaluation uses the clock pointer,

K_{\text{exec}}{=}4

and one sampling seed, and counts an infeasible pose as a skipped step.

variant	without	with	without	with	without	with
	successes		hold		still
program	160	163	0.283	0.185	338	332
relative	123	134	0.596	0.518	387	383
reference	108	121	0.380	0.236	316	300
program $-$ reference	52	42

Appendix

Table 22: RLBench increment variants without and with the fallback for rejected poses (amendment H), on the 24 held-out tasks with 720 episodes per variant. Hold: share of executed steps at which no pose was reachable and the arm held in place. Still: episodes in which no entity moved by 1 cm. Per-task counts will be released with the code.

run (V100)	Panda	IIWA	Kinova3	UR5e	$\Sigma$ bowl	following	3090
PEWAM	26/10	20/18	20/6	16/6	82	0.78	81, 0.73
W8	28/20	18/15	22/11	2/1	70	0.33	68, 0.38
v14 (weights) replicate	23/11	18/10	15/6	8/0	64	n/a	86

Appendix

Table 23: Re-evaluation on three V100s.

\Sigma

bowl and

\Sigma

bottle are the sums over the four arms, as in Table 2 ; following is the mean over the four in-distribution commanded rotations. The last column repeats the corresponding 3090 number. Following is undefined for the weight-encoded replicate, whose realised rotation is negative for every commanded rotation (

-60

-70

-49

and

-77^{\circ}

for

+20

-20

+45

and

-45^{\circ}

Figure 11: One program executed on four arms without retraining (episode 227, side view). The episodes take 100, 99, 103 and 162 steps, as in the evaluation records.

Figure 12: From a demonstration to a program (bowl task, episode 201). Left to right: the demonstration as recorded; the same frame with the robot removed, showing the nine keypoints of every entity; and the program built from them, consisting of the object’s track, its terminal state and the two contact points.

Figure 13: Figure 3 with five moments per episode: the program, then the execution at its start, one third, two thirds and end. Each row shows the outcome stored in its evaluation record.

Figure 14: The lowest-numbered failure of each held-out task in the same record (episodes 211, 400 and 450), drawn as in Figure 13 . Each runs to the 400-step limit.

Figure 15: The contact points rotated to five commanded angles on episode 225, seen from above: the programmed contacts, the frame at which the gripper closes, eight steps later, and the end. In this re-run the $-20^{\circ}$ rotation succeeds, while its record fails; the other four rotations reproduce their records.

Figure 16: The program and the same program reversed, on episode 204, both started from the demonstration’s initial state. The forward program puts the bowl on the stove; under the reversed program the arm reaches the bowl and leaves it in place.

Panel of Figure 17 — Figure 17: Three sampling seeds on one program. Top: (a) the grasp azimuth each seed realises on episode 224, where all three succeed and seed 2 takes the bowl from the other side; (b) the median of the three pairwise azimuth differences for each of the 29 episodes with a grasp under all three seeds. Bottom: episode 224 under each seed, seen from above.

Panel of Figure 18 — Figure 18: A 5 cm push at step 28 of episode 206. Top: the bowl’s distance to the programmed end over time. Despite the push, the effect program succeeds in 66 steps (95 undisturbed). Open-loop replay of the demonstration’s actions succeeds in 93 steps when undisturbed, but after the push it ends 8.0 cm from the programmed end after 400. Bottom: the two pushed episodes, seen from above. We re-ran these episodes for this figure to record the trajectory; they reproduce the outcomes and step counts of the records behind Table 14 .

Figure 19: The forward query. Left: three consecutive windows of episode 200, showing the executed end-effector path given to the model (orange), the object’s realised path (white) and its predicted path (magenta: the mean of eight draws, with the individual draws faint). Right: predicted against realised displacement for each window of four episodes.

Figure 20: A program from pixels (bowl task, episode 200, the first episode of the paired comparison of § 4.2 ). Top: the rendered frame, the SAM 2 masks, the oriented boxes fitted from pixels against the simulator’s, and the object centre tracked from pixels against state (median error 0.61 cm for the bowl, at most 1.51). Bottom: the pixel program executed. This episode succeeds in 80 steps from pixels and in 318 from state; over the 30 paired episodes, 26 succeed from pixels and 27 from state.

Figure 21: LIBERO-Object, the three held-out objects (§ 4.1 ): the lowest-numbered success of each in the record of Table 16 , and below it the same episode under the reference, which is trained without LIBERO-Object data and fails on all three. The program is drawn on the episode’s start scene. These reruns reproduce every outcome of the table’s records.

Figure 22: For Meta-World and RLBench, the first episode in the catalogue on which the baseline fails and the effect program succeeds. Each row shows five frames of one rollout, first to last, taken from that method’s own recording, with the outcome stored in its own record. Meta-World: the reference, given no task specification, descends to the block and retreats; the relative variant lifts the block onto the shelf. RLBench: Instant Policy, given segmented point clouds and one demonstration, hovers over the lid; the effect program places the lid on the jar.

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 223 papers

Apr 23, 2026

Post-training is essential for turning pretrained generalist robot policies into reliable task-specific controllers, but existing human-in-the-loop pipelines remain tied to physical execution: each correction requires robot time, scene setup, resets, and operator supervision in the real world. Meanwhile, action-conditioned world models have been studied mainly for imagination, synthetic data generation, and policy evaluation. We propose Human-in-the-World-Model (Hi-WM), a post-training framework that uses a learned world model as a reusable corrective substrate for failure-targeted policy improvement. A policy is first rolled out in closed loop inside the world model; when the rollout becomes incorrect or failure-prone, a human intervenes directly in the model to provide short corrective actions. Hi-WM caches intermediate states and supports rollback and branching, allowing a single failure state to be reused for multiple corrective continuations and yielding dense supervision around behaviors that the base policy handles poorly. The resulting corrective trajectories are then added back to the training set for post-training. We evaluate Hi-WM on three real-world manipulation tasks spanning both rigid and deformable object interaction, and on two policy backbones. Hi-WM improves real-world success by 37.9 points on average over the base policy and by 19.0 points over a world-model closed-loop baseline, while world-model evaluation correlates strongly with real-world performance (r = 0.953). These results suggest that world models can serve not only as generators or evaluators, but also as effective corrective substrates for scalable robot post-training.

Yaxuan Li, Zhongyi Zhou, Yefei Chen +5

Current Robotics · Tsinghua University · Peking University +1

Robot Policy Learning Robot Policy Adaptation Robot Skill Learning World Models for Robotics Action-Conditioned World Models Human-in-the-Loop AI+1

Ask PDF

Aug 27, 2026

We introduce Riemann-1.0, a fully causal autoregressive World Action Model for embodied intelligence. Riemann-1.0 jointly models multi-view visual observations, robot states, and embodiment-specific actions within a unified causal autoregressive sequence, representing robot actions and world evolution as causal state transitions. Unlike existing WAMs based on joint generation, video-first prediction, or decoupled modeling paradigms, Riemann-1.0 unifies online robot policy execution and action-conditioned world simulation within a single model, enabling it to function as both an executable robot policy and a multi-embodiment visual world simulator. To scale embodied experience across heterogeneous data sources, we further develop a progressive embodied pretraining framework that unifies learning from egocentric human videos, handheld-gripper demonstrations, and heterogeneous robot trajectories under a shared World Action Modeling objective. Built upon 200K+ hours of interaction data, Riemann-1.0 progressively transfers large-scale embodied experience into executable robot manipulation capabilities. Riemann-1.0 achieves state-of-the-art performance across both simulation benchmarks and real-world manipulation tasks. It achieves success rates of 94.3% on RoboTwin2.0, 99.0% on LIBERO, and 62.6% on the long-horizon compositional benchmark RoboCasa-365, outperforming the previous best method by 8.4% On long-horizon real-world manipulation tasks, Riemann-1.0 achieves a Success Rate (SR) of 85.0% and a Progress Success Rate (PSR) of 94.4%, exceeding the strongest open-source baseline by 15% in SR. These results demonstrate that unified World Action Modeling together with progressive embodied pretraining effectively transforms large-scale embodied experience into generalizable robot manipulation capabilities.

Haofeng Sun, Jiangbo Pei, Fei Kang +13

Riemann Dynamics · Research Intern at Riemann Dynamics

World Models for Robotics Long-Horizon Robotic Manipulation Cross-Embodiment Robot Learning

Ask PDF

Aug 13, 2026

We present DreamX-Phi 1.0, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm

\mathrm{SE}(3)

transformations into attention via PRoPE-style geometric encoding, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight depth branch for scene-level geometry and use SAM3 masks with a frozen V-JEPA teacher to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~~1 and second place on Track~~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.

DreamX Team, Rui Chen, Xiangxiang Chu +7

Action-Conditioned Video Generation Robotic Manipulation Video World Models

Ask PDF

bin-picking

hand-insert

door-unlock

$\Sigma/150$

effect program

terminal state only

reference, observation only

joint-state channel