TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation
Authors: Haohan Geng, Sha Zhang, Weishuo Sun, Guanting Ye, Wenhao Yu, Yuang Xie, Qicheng Zeng, Peng Ye, Jiajun Deng, Jianmin Ji
Organizations: University of Science and Technology of China. · The Chinese University of Hong Kong. · University of Macau. · Huawei Technologies Co., Ltd.
Dynamic object manipulation requires robots to anticipate target motion while replanning fast enough to avoid executing outdated actions. Existing approaches either rely on fast reactive control without an explicit future target state or encode anticipated motion only implicitly. World Action Models (WAMs) offer a promising alternative by coupling future visual prediction with action generation, but scene-level video objectives may underrepresent target dynamics that are critical for manipulation, while iterative video-action denoising introduces substantial latency. We present TrajWAM, a trajectory-guided world action model for efficient dynamic robotic manipulation. Our key insight is to explicitly ground future visual imagination in predicted target-object motion and use the resulting future representation to condition action generation, while preserving this anticipation pathway after acceleration. Given a short history of object masks, a history-conditioned motion predictor forecasts future masks encoding the target object's trajectory. A lightweight Trajectory ControlNet then steers a pretrained video model toward trajectory-consistent futures. The resulting future representation conditions an action expert to predict executable action chunks. To enable one-step inference without breaking this interface, we introduce a decoupled distillation scheme that combines distribution matching for the video backbone, Trajectory ControlNet realignment, and action rollout distillation to support accurate one-step action prediction conditioned on the compressed future representation. Comprehensive evaluations in simulation and the real world show that TrajWAM achieves state-of-the-art performance on dynamic object manipulation tasks while enabling efficient one-step video-action inference. Project page: https://kevingeng3.github.io/TrajWAM/.
Figures & tables
Fig. 1: Motivation and performance comparison. (a) Existing WAMs may fail to accurately anticipate target-object motion, resulting in future predictions that deviate from the actual trajectory. TrajWAM uses predicted future object masks as trajectory guidance to generate motion-consistent futures for action prediction. (b) Success rate and inference latency on DOMINO. TrajWAM achieves state-of-the-art performance while maintaining efficient inference.
Fig. 2: Overview of TrajWAM. (a) Training pipeline. A language-conditioned segmentation model extracts the mask history of the target object, from which a motion predictor forecasts a sequence of future masks representing its anticipated trajectory. The predicted mask trajectory guides future-video generation through a Trajectory ControlNet, and the resulting trajectory-controlled video representation conditions the action branch through asymmetric video–action attention to predict an executable action chunk. (b) Inference pipeline. Target-object masks are updated online from incoming observations and maintained in a short history buffer. The predicted future mask trajectory guides the distilled one-step video model, whose future representation conditions the one-step action model for low-latency closed-loop control.
Fig. 3: Video distillation. (a) Video backbone distillation compresses the multi-step video generator into a fast generator using a frozen real-score function and a trainable fake-score function under distribution matching distillation. (b) ControlNet realignment adapts the Trajectory ControlNet to the frozen fast video model using latent reconstruction and diffusion-discriminator objectives.
Fig. 4: Action rollout distillation. The slow teacher and fast student receive the same observation, language instruction, trajectory condition, and initial noises. The trainable fast action model is optimized to match the action chunk generated by the slow multi-step teacher.
Fig. 5: Real-world evaluation setups. We evaluate TrajWAM on (a) conveyor-belt grasping with linear motion and (b) rotating-disc grasping with rotational motion. Both setups use two 6-DoF AgileX Piper arms, each equipped with a wrist-mounted Intel RealSense D435i camera, and a static D435i camera providing a global third-person view.
Fig. 6: Qualitative comparison of real-world conveyor-belt grasping. Snapshots are shown at 1 s intervals from T=0 to 4 s. TrajWAM conditions action generation on trajectory-guided future visual representations to anticipate target motion, successfully grasping and lifting the moving banana. In the illustrated trials, π0.5 and Fast-WAM fail to secure the object. Red crosses and the green check mark indicate grasp failure and success, respectively.
Method
SR (%) ↑
MS
Latency (ms)
OpenVLA [ 12 ]
1.54
6.10
173.6
π0 -FAST [ 36 ]
3.54
20.87
119.9
VLA-Adapter [ 37 ]
4.40
24.31
83.6
RDT-1B [ 38 ]
5.34
17.71
246.3
InternVLA-M1 [ 39 ]
5.40
27.57
183.8
Isaac-GR00T [ 40 ]
6.10
28.60
76.4
TABLE I: Comparison with SOTA methods on DOMINO.
Method
Low-Speed
Medium-Speed
High-Speed
Overall
Banana
Bottle
Tape
Plush toy
Banana
Bottle
Tape
Plush toy
Banana
Bottle
Tape
Plush toy
Succ.
SR (%)
Conveyor-belt
π0.5
15/20
18/20
13/20
19/20
9/20
10/20
8/20
16/20
3/20
0/20
5/20
14/20
130/240
54.2
Fast-WAM
14/20
6/20
11/20
18/20
5/20
2/20
3/20
8/20
1/20
0/20
1/20
2/20
71/240
29.6
TrajWAM (ours)
18/20
16/20
15/20
20/20
16/20
13/20
18/20
19/20
11/20
7/20
16/20
18/20
187/240
77.9
Rotating-disc
TABLE II: Real-world task success rates.
Fig. 7: Ablation of test-time mask sources on DOMINO. We evaluate success rate (left) and manipulation score (right) over 3,500 trials per source, with all model parameters fixed and only the observed masks replaced.
TABLE V: Evaluation subsets for the ablations in Table III . Task names follow the DOMINO identifiers.
Stage
Output channels
Spatial size
History input
4
120×160
Input block
48
120×160
Encoder stage 1
96
60×80
Encoder stage 2
192
30×40
Bottleneck
192
30×40
Decoder stage 1
96
60×80
TABLE VI: Trajectory predictor architecture.
Setting
Value
Training steps
5,000
Global batch size
1
Optimizer
AdamW
Learning rate
5×10−6
Learning-rate schedule
Linear warm-up, then constant
Warm-up steps
100
TABLE VII: Training settings for the action CD and DMD baselines.
Task
SR (%)
MS
adjust_bottle
51.00
67.17
beat_block_hammer
46.00
55.83
click_alarmclock
14.00
19.24
click_bell
4.00
17.28
dump_bin_bigbin
33.00
50.41
grab_roller
52.00
68.84
TABLE VIII: Per-task results of TrajWAM on DOMINO.
Fig. 8: Object-centric trajectory prediction on three DOMINO tasks and two real-world grasping sequences. Each row shows the current scene, four input masks, and four of the eight future predictions. Blue regions denote reference masks; orange contours denote predictions. Future overlays use the corresponding recorded RGB frames. References are dataset ground-truth masks for DOMINO and SAM3 segmentations for the real-world sequences. Time labels indicate offsets in source-video frames relative to t . All temporal panels within a row share a fixed crop; the current scene retains the full field of view.
Fig. 9: Video predictions on DOMINO training demonstrations. From top to bottom: Adjust Bottle, Click Bell, Move Pillbottle Pad, Place Bread Basket, and Place Object Scale. Each pair of rows shows the recorded sequence above the prediction, with time progressing from left to right. The leftmost column is the conditioning frame. The video generator uses one denoising step and ground-truth future object masks.
Fig. 10: Video predictions for real-world dynamic grasping. The first three examples show conveyor-belt grasping with a banana, a bottle, and a roll of tape; the last two show rotating-disc grasping with a plush toy and a banana. Each pair of rows compares the recorded sequence with the prediction at corresponding time points, ordered from left to right.