Dynamic Manipulation with World-Action Models via Counterfactual Planning

Sunwoo Park; Wonbin Lee; Seonghyun Jin; Youngmin Kim; Jangho Park; Jong Chul Ye

cs.ROSep 27, 2026

Dynamic Manipulation with World-Action Models via Counterfactual Planning

Authors: Sunwoo Park, Wonbin Lee, Seonghyun Jin, Youngmin Kim, Jangho Park, Jong Chul Ye

Organizations: Kim Jaechul Graduate School of AI, KAIST, Korea

Abstract

World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar robot-target configuration. The resulting plan is connected to the robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data. Project page: https://methoder00.github.io/DPP/

Figures & tables

Figure 1: Predict, don’t chase. DPP pairs the initial robot state with the predicted interaction position to invoke learned manipulation capabilities rather than repeatedly chasing the moving target.

Figure 2: Target-response collapse. Predicted grip shift : before & after

Figure 3: DPP pipeline. An initial rollout predicts interaction timing while available actions begin execution. The canonical planning stage uses the familiar robot context and predicted target position. An activation bridge connects its trajectory to ongoing execution, after which current-state planning uses fresh observations and current robot-state input while retaining the predicted interaction position as its planning target.

Backbone / Model	Inference Method	Additional Dyn. Training	Success $\uparrow$		Contact $\uparrow$		RC $\uparrow$
Backbone / Model	Inference Method	Additional Dyn. Training	T1	T2	T1	T2	T1	T2
FastWAM	Refresh-24 (Default)	X	10.00	33.33	8.67	80.73	56.52	71.68
	Refresh-8	X	13.00	38.54	12.67	78.65	68.97	68.88
	Future-Conditioned (DPP bootstrap)	X	18.33	34.38	21.33	78.13	69.32	63.36
	Future-Conditioned (Oracle)	X	27.00	48.96	27.33	84.90	77.70	72.99
ImageWAM	Refresh-16 (Default)	X	11.33	39.58	11.33	76.56	61.97	70.62

Table 1: RoboTwin2.0-Dyn results. Each method uses 300 T1 and 192 T2 trials. Success follows the official task predicate; Contact is a separate post hoc diagnostic and may be lower than Success (Appendix E.1 ).

Figure 4: Real-world results and DPP execution. Each method uses 30 trials per task. Bars: success; Task A line: human-evaluated gripper–object contact. Reactive replans every 10 steps; Future-Cond. uses DPP counterfactual observations at $+16$ steps. Task B’s starred methods share a policy trained on anticipation-oriented static demonstrations; Hybrid uses reactive inference.

Figure 5: Complex target motion. Each method uses 100 trials per motion family. R: FastWAM Refresh-24; F: $+16$ -step oracle observations; DPP: FastWAM with the nonlinear extension and a declared motion-family prior.

Variant	SR (%)	$\Delta$ SR	Contact (%)	$\Delta$ Contact	RC	$\Delta$ RC
Full DPP	83.33	—	94.67	—	94.21	—
Component ablations
w/o initial execution	50.33	$-33.00$	46.67	$-48.00$	83.96	$-10.25$
w/o canonical-state conditioning	51.67	$-31.67$	47.33	$-47.33$	77.96	$-16.25$
w/o current-state planning	80.00	$-3.33$	92.33	$-2.33$	92.82	$-1.39$
Oracle diagnostics

Table 2: FastWAM ablations and Oracle diagnostics on Task 1.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Task type	Task / variants	Evaluation objective
T1	Bimanual Grasp	grab_roller	Coordinated two-arm interception of one moving target.
T2	Pick-and-Place	move_pillbottle_pad ; place_empty_cup ; place_phone_stand	Transitioning from grasp of a moving object to release at a moving placement target.
T3	Bottle Picking	pick_dual_bottles ; pick_diverse_bottles	Object- and arm-specific timing and future target positions.
T4	Bell Clicking	click_bell	Contact-based interaction-time estimation.
T5	Block Handover	handover_block	Integrating a dynamic stage into a multi-subtask sequence.
T6	Block Stacking	stack_blocks_two	Chaining multiple subtasks involving multiple moving objects.

Appendix

Table 3: Task suite and evaluation objectives. T1 and T2 are the main-text tasks; T3 contains two RoboTwin variants.

Task	Subtask / interaction	Detection condition in predicted actions
T1	Bimanual grasp	First bilateral gripper-closing event.
T2	Grasp	First sustained closing event of the active-arm gripper.
	Release	First sustained gripper-opening event after the predicted grasp and full closure.
T3	Each bottle grasp	First sustained gripper-closing event, detected independently for each assigned arm and object.
T4	Bell press	First action whose predicted TCP height satisfies the press condition relative to the calibrated tabletop.
T5	Initial block grasp	First sustained closing event of the giving-arm gripper. Subsequent handover uses native current-observation replanning toward fixed destinations.

Appendix

Table 4: Task-specific interaction-time detection from predicted actions. The table describes planning predicates, not simulator success conditions.

Method	T1	T2	T3	T4	T5	T6
	Grasp	Pick–Place	Bottle Picking	Bell	Handover	Stacking
FastWAM (Refresh-24)	10.00	33.33	1.50	3.00	11.00	0.00
FC-FastWAM (Oracle)	27.00	48.96	24.00	11.00	58.00	8.00
DynamicWAM	52.00	53.13	–	5.00	24.00	–
PUMA	50.33	9.38	–	4.00	19.00	–
$\pi_{0.5}$	9.00	13.02	3.50	0.00	12.00	0.00

Appendix

Table 5: Success (%) across the task suite. Higher is better. Trials per method are 300 for T1, 192 for T2, 200 for T3 (100 per Bottle Picking variant), and 100 per remaining additional task. – denotes omitted PUMA/DynamicWAM results on tasks absent from DOMINO. DPP uses the FastWAM backbone.

Figure 6: Target-response collapse under matched target relocation. Directional grip-response gain (top) and its change from prefix 0 (bottom). Maps show all 63 target cells; reported means use the same 33 noncentral cells in the fixed static-support mask. Gray denotes undefined gains. Color scales differ by row.

Figure 7: Contraction of predicted grip-point responses across execution prefixes. White circles are target offsets, blue circles are control-relative bilateral grip responses, and connecting segments preserve each matched pair. The blue hull is computed from mean predicted responses over the fixed 34-cell static-support set. Panel headings report its area relative to prefix 0. This measures contraction of predicted responses, rather than a change in physically reachable workspace.

Prefix	Mean gain	Direction cosine	Area retention	XY error (mm)	SR
0	0.897	0.999	100.00%	23.4	95.77%
8	0.806	0.998	77.82%	34.4	84.13%
16	0.636	0.997	46.24%	58.3	67.20%
32	0.225	0.930	6.12%	118.0	30.16%
48	0.078	0.649	1.47%	136.1	19.58%

Appendix

Table 6: Matched-prefix target-response and task-success summary. Gain and cosine use 33 noncentral support cells; area and error use all 34 support cells. SR uses all 63 grid cells, with three repeats per cell.

Prefix	SR (%)	Gain
0	98.41	0.885
8	94.18	0.817
16	78.31	0.722
32	36.51	0.274
48	18.52	0.063

Appendix

Table 7: Roller target response with ImageWAM. SR uses 189 trials per prefix; gain uses 48 noncentral static-support cells and three repeats.

	FastWAM		ImageWAM
Prefix	SR (%)	Gain	SR (%)	Gain
0	100.00	1.054	100.00	1.020
8	100.00	0.953	100.00	0.992
16	100.00	0.758	100.00	0.895
24	66.67	0.511	72.92	0.712
32	47.92	0.459	50.00	0.637

Appendix

Table 8: Dual-bottle target response through prefix 40. SR uses 48 shifted trials per model and prefix; gain averages both arms over 24 noncentral cells and two repeats. Prefix 48 is omitted because the bottles have already been grasped.

Figure 8: Depth-aware counterfactual observation synthesis with RT-rendered, OIDN-denoised observations. Editing starts from the selected observation. Source-target removal is followed by RGB inpainting and depth restoration using aligned background memory where available. Target reprojection uses scene depth to resolve visibility while preserving the selected robot context.

Figure 9: Real-robot target detection and counterfactual observation synthesis. A recorded trial shows (a) the bootstrap head observation, (b) its target-segmentation overlay, and (c) the synthesized head observation supplied to the first canonical-planning request. (d) Matched crops compare the bootstrap and synthesized target regions. All panels use recorded pipeline outputs; editing artifacts are retained.

Quantity	Setting
Prepayment start / end boundary	15 / 32 steps
Terminal continuation	10 steps
Added EE speed: prepayment / nominal Cartesian bridge	$0.10/0.20\,\mathrm{m/s}$
Total EE speed	$0.50\,\mathrm{m/s}$
EE acceleration / jerk	$5\,\mathrm{m/s^{2}}$ / $100\,\mathrm{m/s^{3}}$
Joint speed / acceleration	$2\,\mathrm{rad/s}$ / $20\,\mathrm{rad/s^{2}}$

Appendix

Table 9: Example controller settings from the main Task 1 constant-velocity experiments. Steps denote policy steps; prepayment boundaries are relative to the start of the corresponding planning phase. Cartesian limits apply to prepayment and nominal Cartesian bridges; the base canonical-to-current joint-space connection uses joint-motion checks without Cartesian-limit evaluation or trajectory IK. The canonical fallback retains joint checks but bypasses Cartesian rejection. These are controller settings, not certified physical safety bounds.

Panel of Figure 10 — Figure 10: Real-world experimental setup. (a) I2RT YAM leader–follower platform with an elevated head camera and wrist-mounted cameras. The follower arms are on the rear rail and the leader arms are in the foreground. (b) Task objects and the motor-driven apparatus used to generate target motion. The objects and apparatus are shown separately for identification.

Method	Success (%)	Successes
PUMA	86.67	260/300
DPP	99.67	299/300

Appendix

Table 10: DOMINO-style evaluation on T1.

Figure 11: Contact-stop versus interaction-aware driver handoff. A roller moves along world $-x$ at 3 mm per policy step. Each column pair compares DOMINO’s native driver with the RoboTwin2.0-Dyn T1 driver at the same policy step: before contact (47), first contact (48), and the latter’s grasp-triggered handoff (71). Colored borders mark the corresponding motion-to-physics transition frames in both conditions. Head and front views are recorded together within each execution; panels show the first recorded observation after the indicated action. The two conditions use the same scene, identical initial policy observations, and the DPP planner, but are independent executions rather than a replay of identical actions. Both satisfy the same native task-success criterion.

Timing component	Mean latency (s)
First-action activation	0.496
32-action chunk generation	0.495
Backbone inference, included above	0.487

Appendix

Table 11: Measured latency for FastWAM on an RTX 5090, using the six episodes in Figure 12 and their 71 generated chunks. First-action activation is measured from runtime start with the initial observation available and warm-up complete. Chunk generation includes adapter processing; backbone inference includes input preparation, denoising, decoding, and output materialization. These illustrative measurements are separate from the aggregate task-success evaluation.

Figure 12: Executable action queues in selected successful T1–T6 episodes. A positive queue means actions remain available for execution without waiting for further action generation. Action-consumption samples are connected linearly for readability, while queue replenishment and replacement retain their logged event times. Open circles mark the last executed action of each episode. Time includes initial and new-cycle preparation within the runtime; unaccepted planning candidates are excluded from queue counts.

Backbone / Model	Inference Method	Speed (cm/s)					All
		1.0–1.8	1.8–2.6	2.6–3.4	3.4–4.2	4.2–5.0
FastWAM	Refresh-24 (Default)	25.00	13.33	8.33	3.33	0.00	10.00
	Refresh-8	26.67	15.00	13.33	6.67	3.33	13.00
	Future-Conditioned (DPP bootstrap)	40.00	16.67	16.67	8.33	10.00	18.33
	Future-Conditioned (Oracle)	56.67	23.33	20.00	15.00	20.00	27.00
ImageWAM	Refresh-16 (Default)	26.67	11.67	11.67	6.67	0.00	11.33

Appendix

Table 12: T1 success by nominal object speed. Each bin contains 60 scenes; the All column aggregates 300 scenes and matches Table 1 . Method definitions follow the main table. Bold marks the highest rate in each column.

Backbone	Setting	Success $\uparrow$	Contact $\uparrow$	RC $\uparrow$
FastWAM + VP	Oracle-free	83.33%	94.67%	94.21
	Oracle-V	92.33%	95.33%	97.22
	Oracle-Obs	94.00%	98.00%	97.90
ImageWAM + VP	Oracle-free	78.67%	93.33%	93.43
	Oracle-V	90.00%	92.67%	96.11
	Oracle-Obs	93.67%	94.67%	96.89

Appendix

Table 13: T1 oracle ablation, grouped by backbone. All settings for both backbones use the same 300 scenes. The Oracle-free values match Table 1 . Oracle-V and Oracle-Obs use the same controller and evaluation protocol with oracle substitutions. Contact requires both gripper values below 0.5 and bilateral target contact for two consecutive policy steps. RC is averaged over all episodes, including failures.

Component	Base DPP	Nonlinear extension
Motion-hypothesis fitting	Bootstrap only	Causal, family-specific
Post-bootstrap pose accumulation	Disabled	Enabled
Late canonical planning	Disabled	Enabled
Current-State output spatial warping	Disabled	Conditional
Temporal Alignment	Enabled	Enabled

Appendix

Table 14: Motion-update settings for base DPP and the optional nonlinear motion extension. Conditional spatial warping corrects output actions, not the target hypothesis in the current-state planning input.

Visual	Initial	Canonical	Reduction	RGB MAE	Action RMSE
steps	(s)	(s)		(0–255 scale)
10	3.840	8.321	–	0.00	0.0000
6	2.581	5.870	30.5%	0.85	0.0042
3	1.726	3.226	59.3%	2.16	0.0232
2	1.396	2.577	67.3%	2.79	0.0239

Appendix

Table 15: ImageWAM visual denoising-step ablation. Action denoising remains fixed at ten steps. Latencies are means over five matched episodes; RGB and action errors are measured against the ten-step initial rollout.

Figure 13: ImageWAM visual denoising boundary on a representative difficult pair. Top rows compare predicted initial and canonical endpoints at fixed chunk indices for 10, 3, and 2 visual denoising steps. Bottom rows show absolute RGB differences from the ten-step reference, amplified by $4\times$ . Three steps retain the task-relevant geometry with small action deviation; the two-step endpoint exhibits the next visible degradation for a comparatively small additional latency reduction.

Group	Pairs	FastWAM	DynamicWAM	Ratio
Task 1	25	$8.61\pm 0.66$	$16.87\pm 0.80$	1.96
Task 2	25	$5.34\pm 1.63$	$8.18\pm 1.49$	1.53
Overall	50	$6.97\pm 2.06$	$12.53\pm 4.55$	1.80

Appendix

Table 16: Mean

\pm

standard deviation of bilateral TCP displacement over policy steps 1–24, measured in mm/action rather than physical velocity. Ratios are DynamicWAM/FastWAM ratios of mean displacement. FastWAM uses a 24-action refresh interval.

Approach	Intervention	Relation to DPP
SIDO ( Shin et al., 2026a )	Augment object poses and the associated action chunks during training.	Shares future-pose conditioning; DPP changes the frozen model’s planning observation instead.
AHEAD ( Syed et al., 2026 )	Train a latent predictor of future visual tokens for a frozen VLA.	Shares anticipation; DPP uses the WAM’s own rollout and a canonical robot context.
TEMPO ( Feng et al., 2026 )	Learn conditioning on motion summaries and proprioceptive history.	Addresses missing temporal information; DPP probes responsiveness across robot contexts.
F2F-AP ( Wei et al., 2026 )	Predict a flow-based observation representation one system delay ahead.	Shares motion anticipation; DPP places the target at a predicted task event.
RTC ( Black et al., 2025b )	Generate continuations asynchronously around committed actions.	Addresses chunk continuity; DPP must also connect a counterfactual trajectory to the live state.
AHA-WAM ( Cai et al., 2026 )	Couple a slower world planner with a faster feedback action model.	Shares asynchronous world-model control; DPP adds canonical querying at the interaction target.

Appendix

Table 17: Mechanistic comparison with the closest related work. A frozen backbone does not make a method training-free: auxiliary predictors or conditioning modules may still be trained.

Explore similar work

DynamicWAM: Dual-Path Motion Conditioning for World-Action Models in Dynamic Manipulation
Aug 1, 2026Yunfan Lou, Hewen Gao, Xiyu Zhu +6World Models for Robotics Robotic Manipulation
PKU-PI Lab · National University of Singapore · ePyBot Intelligence +2
PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
Oct 2, 2026Chunghyun Park, Beomjun Kim, Seungcheol Park +5Robotic Manipulation Embodied World Models
POSTECH · KAIST · RLWRLD +1
PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulation
Jul 2, 2026Peng Yun, Shouwang Huang, Hao Li +3World Models for Robotics World Models
vLAR Group, The Hong Kong Polytechnic University, Hong Kong SAR, China · Astribot, China

Aug 1, 2026J/K move · Enter open · S save

DynamicWAM: Dual-Path Motion Conditioning for World-Action Models in Dynamic Manipulation

Yunfan Lou, Hewen Gao, Xiyu Zhu, Zhuoran Qiao+5

PKU-PI Lab · National University of Singapore · ePyBot Intelligence+2

World Models for Robotics Robotic Manipulation Dynamic Robotic Manipulation

Open paper AskPDF arXiv

Dynamic manipulation requires robots to infer target motion and respond promptly, yet existing World-Action Models (WAMs) typically condition only on the current frame and execute large backbones synchronously, limiting motion awareness and responsive control in dynamic scenes. We propose DynamicWAM, a compact WAM for dynamic object manipulation with dual-path motion conditioning. DynamicWAM introduces history-flow conditioning, encoding temporally aligned optical-flow frames alongside the current observation through a frozen pretrained video VAE to preserve spatial motion structure, while injecting kinematic descriptors of displacement, duration, velocity, and acceleration into the action expert to provide motion magnitude and timing. The two complementary paths are fused through joint world-action attention. A distilled compact backbone and real-time chunking (RTC)-based asynchronous execution further enable responsive control. On DOMINO, DynamicWAM achieves a 38.2% success rate and a 53.2 manipulation score, outperforming all evaluated baselines. Across 12 real-world tasks spanning linear, circular, and compound target motion, it achieves a 46.7% average success rate, exceeding the strongest baseline by 22.9 percentage points.

Abstract

Explore similar work

DynamicWAM: Dual-Path Motion Conditioning for World-Action Models in Dynamic Manipulation

PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulation