World-Action models (WAMs) trained on static demonstrations often fail to manipulate moving targets even when they possess the required manipulation skills. We attribute this failure to target-response collapse: as execution advances, the policy becomes increasingly biased toward the learned continuation of its ongoing behavior and less responsive to target relocation. To bridge the gap between what the model has learned and what it can generate from the current context, we formulate dynamic manipulation as counterfactual planning by decoupling the context used for plan generation from the physical state used for execution. Our framework, Dynamic Predictive Planning (DPP), first uses the WAM's predictive rollout to estimate when an interaction is expected to occur, and combines this timing estimate with observed target motion to predict the target's future interaction position. DPP then constructs a counterfactual observation that places this predicted target position in a familiar robot context, allowing the model to invoke an existing manipulation skill rather than generate a recovery behavior from an unfamiliar robot-target configuration. The resulting plan is connected to the robot's actual state during execution. DPP enables real-time dynamic manipulation on a single consumer GPU without additional training on dynamic data. Experiments in simulation and on a real robot demonstrate consistent improvements across diverse target motions, with simulation performance surpassing all evaluated baselines, including methods additionally trained on dynamic data. Project page: https://methoder00.github.io/DPP/
Figures & tables
Figure 1: Predict, don’t chase. DPP pairs the initial robot state with the predicted interaction position to invoke learned manipulation capabilities rather than repeatedly chasing the moving target.
Figure 2: Target-response collapse. Predicted grip shift : before & after
Figure 3: DPP pipeline. An initial rollout predicts interaction timing while available actions begin execution. The canonical planning stage uses the familiar robot context and predicted target position. An activation bridge connects its trajectory to ongoing execution, after which current-state planning uses fresh observations and current robot-state input while retaining the predicted interaction position as its planning target.
Backbone / Model
Inference Method
Additional Dyn. Training
Success ↑
Contact ↑
RC ↑
T1
T2
T1
T2
T1
T2
FastWAM
Refresh-24 (Default)
X
10.00
33.33
8.67
80.73
56.52
71.68
Refresh-8
X
13.00
38.54
12.67
78.65
68.97
68.88
Future-Conditioned (DPP bootstrap)
X
18.33
34.38
21.33
78.13
69.32
63.36
Future-Conditioned (Oracle)
X
27.00
48.96
27.33
84.90
77.70
72.99
ImageWAM
Refresh-16 (Default)
X
11.33
39.58
11.33
76.56
61.97
70.62
Table 1: RoboTwin2.0-Dyn results. Each method uses 300 T1 and 192 T2 trials. Success follows the official task predicate; Contact is a separate post hoc diagnostic and may be lower than Success (Appendix E.1 ).
Figure 4: Real-world results and DPP execution. Each method uses 30 trials per task. Bars: success; Task A line: human-evaluated gripper–object contact. Reactive replans every 10 steps; Future-Cond. uses DPP counterfactual observations at +16 steps. Task B’s starred methods share a policy trained on anticipation-oriented static demonstrations; Hybrid uses reactive inference.
Figure 5: Complex target motion. Each method uses 100 trials per motion family. R: FastWAM Refresh-24; F: +16 -step oracle observations; DPP: FastWAM with the nonlinear extension and a declared motion-family prior.
Variant
SR (%)
Δ SR
Contact (%)
Δ Contact
RC
Δ RC
Full DPP
83.33
—
94.67
—
94.21
—
Component ablations
w/o initial execution
50.33
−33.00
46.67
−48.00
83.96
−10.25
w/o canonical-state conditioning
51.67
−31.67
47.33
−47.33
77.96
−16.25
w/o current-state planning
80.00
−3.33
92.33
−2.33
92.82
−1.39
Oracle diagnostics
Table 2: FastWAM ablations and Oracle diagnostics on Task 1.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Task type
Task / variants
Evaluation objective
T1
Bimanual Grasp
grab_roller
Coordinated two-arm interception of one moving target.
Table 3: Task suite and evaluation objectives. T1 and T2 are the main-text tasks; T3 contains two RoboTwin variants.
Task
Subtask / interaction
Detection condition in predicted actions
T1
Bimanual grasp
First bilateral gripper-closing event.
T2
Grasp
First sustained closing event of the active-arm gripper.
Release
First sustained gripper-opening event after the predicted grasp and full closure.
T3
Each bottle grasp
First sustained gripper-closing event, detected independently for each assigned arm and object.
T4
Bell press
First action whose predicted TCP height satisfies the press condition relative to the calibrated tabletop.
T5
Initial block grasp
First sustained closing event of the giving-arm gripper. Subsequent handover uses native current-observation replanning toward fixed destinations.
Appendix
Table 4: Task-specific interaction-time detection from predicted actions. The table describes planning predicates, not simulator success conditions.
Method
T1
T2
T3
T4
T5
T6
Grasp
Pick–Place
Bottle Picking
Bell
Handover
Stacking
FastWAM (Refresh-24)
10.00
33.33
1.50
3.00
11.00
0.00
FC-FastWAM (Oracle)
27.00
48.96
24.00
11.00
58.00
8.00
DynamicWAM
52.00
53.13
–
5.00
24.00
–
PUMA
50.33
9.38
–
4.00
19.00
–
π0.5
9.00
13.02
3.50
0.00
12.00
0.00
Appendix
Table 5: Success (%) across the task suite. Higher is better. Trials per method are 300 for T1, 192 for T2, 200 for T3 (100 per Bottle Picking variant), and 100 per remaining additional task. – denotes omitted PUMA/DynamicWAM results on tasks absent from DOMINO. DPP uses the FastWAM backbone.
Figure 6: Target-response collapse under matched target relocation. Directional grip-response gain (top) and its change from prefix 0 (bottom). Maps show all 63 target cells; reported means use the same 33 noncentral cells in the fixed static-support mask. Gray denotes undefined gains. Color scales differ by row.
Figure 7: Contraction of predicted grip-point responses across execution prefixes. White circles are target offsets, blue circles are control-relative bilateral grip responses, and connecting segments preserve each matched pair. The blue hull is computed from mean predicted responses over the fixed 34-cell static-support set. Panel headings report its area relative to prefix 0. This measures contraction of predicted responses, rather than a change in physically reachable workspace.
Prefix
Mean gain
Direction cosine
Area retention
XY error (mm)
SR
0
0.897
0.999
100.00%
23.4
95.77%
8
0.806
0.998
77.82%
34.4
84.13%
16
0.636
0.997
46.24%
58.3
67.20%
32
0.225
0.930
6.12%
118.0
30.16%
48
0.078
0.649
1.47%
136.1
19.58%
Appendix
Table 6: Matched-prefix target-response and task-success summary. Gain and cosine use 33 noncentral support cells; area and error use all 34 support cells. SR uses all 63 grid cells, with three repeats per cell.
Prefix
SR (%)
Gain
0
98.41
0.885
8
94.18
0.817
16
78.31
0.722
32
36.51
0.274
48
18.52
0.063
Appendix
Table 7: Roller target response with ImageWAM. SR uses 189 trials per prefix; gain uses 48 noncentral static-support cells and three repeats.
FastWAM
ImageWAM
Prefix
SR (%)
Gain
SR (%)
Gain
0
100.00
1.054
100.00
1.020
8
100.00
0.953
100.00
0.992
16
100.00
0.758
100.00
0.895
24
66.67
0.511
72.92
0.712
32
47.92
0.459
50.00
0.637
Appendix
Table 8: Dual-bottle target response through prefix 40. SR uses 48 shifted trials per model and prefix; gain averages both arms over 24 noncentral cells and two repeats. Prefix 48 is omitted because the bottles have already been grasped.
Figure 8: Depth-aware counterfactual observation synthesis with RT-rendered, OIDN-denoised observations. Editing starts from the selected observation. Source-target removal is followed by RGB inpainting and depth restoration using aligned background memory where available. Target reprojection uses scene depth to resolve visibility while preserving the selected robot context.
Figure 9: Real-robot target detection and counterfactual observation synthesis. A recorded trial shows (a) the bootstrap head observation, (b) its target-segmentation overlay, and (c) the synthesized head observation supplied to the first canonical-planning request. (d) Matched crops compare the bootstrap and synthesized target regions. All panels use recorded pipeline outputs; editing artifacts are retained.
Quantity
Setting
Prepayment start / end boundary
15 / 32 steps
Terminal continuation
10 steps
Added EE speed: prepayment / nominal Cartesian bridge
0.10/0.20m/s
Total EE speed
0.50m/s
EE acceleration / jerk
5m/s2 / 100m/s3
Joint speed / acceleration
2rad/s / 20rad/s2
Appendix
Table 9: Example controller settings from the main Task 1 constant-velocity experiments. Steps denote policy steps; prepayment boundaries are relative to the start of the corresponding planning phase. Cartesian limits apply to prepayment and nominal Cartesian bridges; the base canonical-to-current joint-space connection uses joint-motion checks without Cartesian-limit evaluation or trajectory IK. The canonical fallback retains joint checks but bypasses Cartesian rejection. These are controller settings, not certified physical safety bounds.
Figure 10: Real-world experimental setup. (a) I2RT YAM leader–follower platform with an elevated head camera and wrist-mounted cameras. The follower arms are on the rear rail and the leader arms are in the foreground. (b) Task objects and the motor-driven apparatus used to generate target motion. The objects and apparatus are shown separately for identification.
Method
Success (%)
Successes
PUMA
86.67
260/300
DPP
99.67
299/300
Appendix
Table 10: DOMINO-style evaluation on T1.
Figure 11: Contact-stop versus interaction-aware driver handoff. A roller moves along world −x at 3 mm per policy step. Each column pair compares DOMINO’s native driver with the RoboTwin2.0-Dyn T1 driver at the same policy step: before contact (47), first contact (48), and the latter’s grasp-triggered handoff (71). Colored borders mark the corresponding motion-to-physics transition frames in both conditions. Head and front views are recorded together within each execution; panels show the first recorded observation after the indicated action. The two conditions use the same scene, identical initial policy observations, and the DPP planner, but are independent executions rather than a replay of identical actions. Both satisfy the same native task-success criterion.
Timing component
Mean latency (s)
First-action activation
0.496
32-action chunk generation
0.495
Backbone inference, included above
0.487
Appendix
Table 11: Measured latency for FastWAM on an RTX 5090, using the six episodes in Figure 12 and their 71 generated chunks. First-action activation is measured from runtime start with the initial observation available and warm-up complete. Chunk generation includes adapter processing; backbone inference includes input preparation, denoising, decoding, and output materialization. These illustrative measurements are separate from the aggregate task-success evaluation.
Figure 12: Executable action queues in selected successful T1–T6 episodes. A positive queue means actions remain available for execution without waiting for further action generation. Action-consumption samples are connected linearly for readability, while queue replenishment and replacement retain their logged event times. Open circles mark the last executed action of each episode. Time includes initial and new-cycle preparation within the runtime; unaccepted planning candidates are excluded from queue counts.
Backbone / Model
Inference Method
Speed (cm/s)
All
1.0–1.8
1.8–2.6
2.6–3.4
3.4–4.2
4.2–5.0
FastWAM
Refresh-24 (Default)
25.00
13.33
8.33
3.33
0.00
10.00
Refresh-8
26.67
15.00
13.33
6.67
3.33
13.00
Future-Conditioned (DPP bootstrap)
40.00
16.67
16.67
8.33
10.00
18.33
Future-Conditioned (Oracle)
56.67
23.33
20.00
15.00
20.00
27.00
ImageWAM
Refresh-16 (Default)
26.67
11.67
11.67
6.67
0.00
11.33
Appendix
Table 12: T1 success by nominal object speed. Each bin contains 60 scenes; the All column aggregates 300 scenes and matches Table 1 . Method definitions follow the main table. Bold marks the highest rate in each column.
Backbone
Setting
Success ↑
Contact ↑
RC ↑
FastWAM + VP
Oracle-free
83.33%
94.67%
94.21
Oracle-V
92.33%
95.33%
97.22
Oracle-Obs
94.00%
98.00%
97.90
ImageWAM + VP
Oracle-free
78.67%
93.33%
93.43
Oracle-V
90.00%
92.67%
96.11
Oracle-Obs
93.67%
94.67%
96.89
Appendix
Table 13: T1 oracle ablation, grouped by backbone. All settings for both backbones use the same 300 scenes. The Oracle-free values match Table 1 . Oracle-V and Oracle-Obs use the same controller and evaluation protocol with oracle substitutions. Contact requires both gripper values below 0.5 and bilateral target contact for two consecutive policy steps. RC is averaged over all episodes, including failures.
Component
Base DPP
Nonlinear extension
Motion-hypothesis fitting
Bootstrap only
Causal, family-specific
Post-bootstrap pose accumulation
Disabled
Enabled
Late canonical planning
Disabled
Enabled
Current-State output spatial warping
Disabled
Conditional
Temporal Alignment
Enabled
Enabled
Appendix
Table 14: Motion-update settings for base DPP and the optional nonlinear motion extension. Conditional spatial warping corrects output actions, not the target hypothesis in the current-state planning input.
Visual
Initial
Canonical
Reduction
RGB MAE
Action RMSE
steps
(s)
(s)
(0–255 scale)
10
3.840
8.321
–
0.00
0.0000
6
2.581
5.870
30.5%
0.85
0.0042
3
1.726
3.226
59.3%
2.16
0.0232
2
1.396
2.577
67.3%
2.79
0.0239
Appendix
Table 15: ImageWAM visual denoising-step ablation. Action denoising remains fixed at ten steps. Latencies are means over five matched episodes; RGB and action errors are measured against the ten-step initial rollout.
Figure 13: ImageWAM visual denoising boundary on a representative difficult pair. Top rows compare predicted initial and canonical endpoints at fixed chunk indices for 10, 3, and 2 visual denoising steps. Bottom rows show absolute RGB differences from the ten-step reference, amplified by 4× . Three steps retain the task-relevant geometry with small action deviation; the two-step endpoint exhibits the next visible degradation for a comparatively small additional latency reduction.
Group
Pairs
FastWAM
DynamicWAM
Ratio
Task 1
25
8.61±0.66
16.87±0.80
1.96
Task 2
25
5.34±1.63
8.18±1.49
1.53
Overall
50
6.97±2.06
12.53±4.55
1.80
Appendix
Table 16: Mean ± standard deviation of bilateral TCP displacement over policy steps 1–24, measured in mm/action rather than physical velocity. Ratios are DynamicWAM/FastWAM ratios of mean displacement. FastWAM uses a 24-action refresh interval.
Approach
Intervention
Relation to DPP
SIDO ( Shin et al., 2026a )
Augment object poses and the associated action chunks during training.
Train a latent predictor of future visual tokens for a frozen VLA.
Shares anticipation; DPP uses the WAM’s own rollout and a canonical robot context.
TEMPO ( Feng et al., 2026 )
Learn conditioning on motion summaries and proprioceptive history.
Addresses missing temporal information; DPP probes responsiveness across robot contexts.
F2F-AP ( Wei et al., 2026 )
Predict a flow-based observation representation one system delay ahead.
Shares motion anticipation; DPP places the target at a predicted task event.
RTC ( Black et al., 2025b )
Generate continuations asynchronously around committed actions.
Addresses chunk continuity; DPP must also connect a counterfactual trajectory to the live state.
AHA-WAM ( Cai et al., 2026 )
Couple a slower world planner with a faster feedback action model.
Shares asynchronous world-model control; DPP adds canonical querying at the interaction target.
Appendix
Table 17: Mechanistic comparison with the closest related work. A frozen backbone does not make a method training-free: auxiliary predictors or conditioning modules may still be trained.
Dynamic manipulation requires robots to infer target motion and respond promptly, yet existing World-Action Models (WAMs) typically condition only on the current frame and execute large backbones synchronously, limiting motion awareness and responsive control in dynamic scenes. We propose DynamicWAM, a compact WAM for dynamic object manipulation with dual-path motion conditioning. DynamicWAM introduces history-flow conditioning, encoding temporally aligned optical-flow frames alongside the current observation through a frozen pretrained video VAE to preserve spatial motion structure, while injecting kinematic descriptors of displacement, duration, velocity, and acceleration into the action expert to provide motion magnitude and timing. The two complementary paths are fused through joint world-action attention. A distilled compact backbone and real-time chunking (RTC)-based asynchronous execution further enable responsive control. On DOMINO, DynamicWAM achieves a 38.2% success rate and a 53.2 manipulation score, outperforming all evaluated baselines. Across 12 real-world tasks spanning linear, circular, and compound target motion, it achieves a 46.7% average success rate, exceeding the strongest baseline by 22.9 percentage points.
Yunfan Lou, Hewen Gao, Xiyu Zhu +6
PKU-PI Lab · National University of Singapore · ePyBot Intelligence +2
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.
Manipulating fast and dynamically moving targets in unstructured 3D environments remains challenging for embodied AI. Existing visual-language-action models and world models struggle with accurate 3D geometry and physically meaningful forecasting. We propose PhysMani, a framework that couples a physics-principled 3D Gaussian world model with a future-aware action policy model. The world model learns a divergence-free Gaussian velocity field via online optimization for fast and physically grounded future dynamics prediction. The policy model integrates the predicted 3D scene future dynamics through a learnable token based cross-attention module. We introduce PhysMani-Bench, a dynamic manipulation benchmark with 16 tasks, and demonstrate a superior success rate over strong baselines in both simulation and real-world robot experiments.
Peng Yun, Shouwang Huang, Hao Li +3
vLAR Group, The Hong Kong Polytechnic University, Hong Kong SAR, China · Astribot, China