cs.ROOct 19, 2026

TrajWAM: An Efficient Trajectory-Guided World Action Model for Dynamic Robotic Manipulation

Authors: Haohan Geng, Sha Zhang, Weishuo Sun, Guanting Ye, Wenhao Yu, Yuang Xie, Qicheng Zeng, Peng Ye, 

Organizations: University of Science and Technology of China. · The Chinese University of Hong Kong. · University of Macau. · Huawei Technologies Co., Ltd.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Dynamic object manipulation requires robots to anticipate target motion while replanning fast enough to avoid executing outdated actions. Existing approaches either rely on fast reactive control without an explicit future target state or encode anticipated motion only implicitly. World Action Models (WAMs) offer a promising alternative by coupling future visual prediction with action generation, but scene-level video objectives may underrepresent target dynamics that are critical for manipulation, while iterative video-action denoising introduces substantial latency. We present TrajWAM, a trajectory-guided world action model for efficient dynamic robotic manipulation. Our key insight is to explicitly ground future visual imagination in predicted target-object motion and use the resulting future representation to condition action generation, while preserving this anticipation pathway after acceleration. Given a short history of object masks, a history-conditioned motion predictor forecasts future masks encoding the target object's trajectory. A lightweight Trajectory ControlNet then steers a pretrained video model toward trajectory-consistent futures. The resulting future representation conditions an action expert to predict executable action chunks. To enable one-step inference without breaking this interface, we introduce a decoupled distillation scheme that combines distribution matching for the video backbone, Trajectory ControlNet realignment, and action rollout distillation to support accurate one-step action prediction conditioned on the compressed future representation. Comprehensive evaluations in simulation and the real world show that TrajWAM achieves state-of-the-art performance on dynamic object manipulation tasks while enabling efficient one-step video-action inference. Project page: https://kevingeng3.github.io/TrajWAM/.

Figures & tables

Fig. 1: Motivation and performance comparison. (a) Existing WAMs may fail to accurately anticipate target-object motion, resulting in future predictions that deviate from the actual trajectory. TrajWAM uses predicted future object masks as trajectory guidance to generate motion-consistent futures for action prediction. (b) Success rate and inference latency on DOMINO. TrajWAM achieves state-of-the-art performance while maintaining efficient inference.
Fig. 2: Overview of TrajWAM. (a) Training pipeline. A language-conditioned segmentation model extracts the mask history of the target object, from which a motion predictor forecasts a sequence of future masks representing its anticipated trajectory. The predicted mask trajectory guides future-video generation through a Trajectory ControlNet, and the resulting trajectory-controlled video representation conditions the action branch through asymmetric video–action attention to predict an executable action chunk. (b) Inference pipeline. Target-object masks are updated online from incoming observations and maintained in a short history buffer. The predicted future mask trajectory guides the distilled one-step video model, whose future representation conditions the one-step action model for low-latency closed-loop control.
Fig. 3: Video distillation. (a) Video backbone distillation compresses the multi-step video generator into a fast generator using a frozen real-score function and a trainable fake-score function under distribution matching distillation. (b) ControlNet realignment adapts the Trajectory ControlNet to the frozen fast video model using latent reconstruction and diffusion-discriminator objectives.
Fig. 4: Action rollout distillation. The slow teacher and fast student receive the same observation, language instruction, trajectory condition, and initial noises. The trainable fast action model is optimized to match the action chunk generated by the slow multi-step teacher.
Fig. 5: Real-world evaluation setups. We evaluate TrajWAM on (a) conveyor-belt grasping with linear motion and (b) rotating-disc grasping with rotational motion. Both setups use two 6-DoF AgileX Piper arms, each equipped with a wrist-mounted Intel RealSense D435i camera, and a static D435i camera providing a global third-person view.
Fig. 6: Qualitative comparison of real-world conveyor-belt grasping. Snapshots are shown at 1 s intervals from T=0T=0 to 44 s. TrajWAM conditions action generation on trajectory-guided future visual representations to anticipate target motion, successfully grasping and lifting the moving banana. In the illustrated trials, π0.5\pi_{0.5} and Fast-WAM fail to secure the object. Red crosses and the green check mark indicate grasp failure and success, respectively.
MethodSR (%) ↑\uparrowMSLatency (ms)
OpenVLA [ 12 ]1.546.10173.6
π0\pi_{0} -FAST [ 36 ]3.5420.87119.9
VLA-Adapter [ 37 ]4.4024.3183.6
RDT-1B [ 38 ]5.3417.71246.3
InternVLA-M1 [ 39 ]5.4027.57183.8
Isaac-GR00T [ 40 ]6.1028.6076.4
TABLE I: Comparison with SOTA methods on DOMINO.
MethodLow-SpeedMedium-SpeedHigh-SpeedOverall
BananaBottleTapePlush toyBananaBottleTapePlush toyBananaBottleTapePlush toySucc.SR (%)
Conveyor-belt
π0.5\pi_{0.5}15/2018/2013/2019/209/2010/208/2016/203/200/205/2014/20130/24054.2
Fast-WAM14/206/2011/2018/205/202/203/208/201/200/201/202/2071/24029.6
TrajWAM (ours)18/2016/2015/2020/2016/2013/2018/2019/2011/207/2016/2018/20187/24077.9
Rotating-disc
TABLE II: Real-world task success rates.
Fig. 7: Ablation of test-time mask sources on DOMINO. We evaluate success rate (left) and manipulation score (right) over 3,500 trials per source, with all model parameters fixed and only the observed masks replaced.
VariantSR (%)MS
(a) Trajectory conditions (10 tasks)
Static20.9033.72
Black (all-zero)32.0044.32
Predicted47.3058.32
(b) One-step action distillation (12 tasks)
Undistilled (1-step inference)7.3915.82
TABLE III: Ablation studies on DOMINO.
Dynamic Interception (19)Dynamic Tracking (16)
Adjust BottleBeat Block Hammer
Dump Bin BigbinClick Alarmclock
Grab RollerClick Bell
Handover BlockMove Pillbottle Pad
Handover MicMove Stapler Pad
Hanging MugPlace Bread Basket
TABLE IV: Official task categories of DOMINO.
AblationTasks
(a) Trajectory conditions (10)click_bell , press_stapler , move_pillbottle_pad , place_shoe , rotate_qrcode , move_can_pot , grab_roller , handover_block , put_bottles_dustbin , place_object_basket
(b) Action distillation (12)click_bell , press_stapler , place_object_scale , place_shoe , move_pillbottle_pad , place_empty_cup , rotate_qrcode , move_can_pot , grab_roller , handover_block , put_bottles_dustbin , place_object_basket
(c) Adapter realignment (10)click_alarmclock , move_stapler_pad , place_mouse_pad , place_phone_stand , dump_bin_bigbin , move_pillbottle_pad , rotate_qrcode , place_a2b_left , grab_roller , place_object_basket
TABLE V: Evaluation subsets for the ablations in Table III . Task names follow the DOMINO identifiers.
StageOutput channelsSpatial size
History input4120×160120\times 160
Input block48120×160120\times 160
Encoder stage 19660×8060\times 80
Encoder stage 219230×4030\times 40
Bottleneck19230×4030\times 40
Decoder stage 19660×8060\times 80
TABLE VI: Trajectory predictor architecture.
SettingValue
Training steps5,000
Global batch size1
OptimizerAdamW
Learning rate5×10−65\times 10^{-6}
Learning-rate scheduleLinear warm-up, then constant
Warm-up steps100
TABLE VII: Training settings for the action CD and DMD baselines.
TaskSR (%)MS
adjust_bottle51.0067.17
beat_block_hammer46.0055.83
click_alarmclock14.0019.24
click_bell4.0017.28
dump_bin_bigbin33.0050.41
grab_roller52.0068.84
TABLE VIII: Per-task results of TrajWAM on DOMINO.
Fig. 8: Object-centric trajectory prediction on three DOMINO tasks and two real-world grasping sequences. Each row shows the current scene, four input masks, and four of the eight future predictions. Blue regions denote reference masks; orange contours denote predictions. Future overlays use the corresponding recorded RGB frames. References are dataset ground-truth masks for DOMINO and SAM3 segmentations for the real-world sequences. Time labels indicate offsets in source-video frames relative to tt . All temporal panels within a row share a fixed crop; the current scene retains the full field of view.
Fig. 9: Video predictions on DOMINO training demonstrations. From top to bottom: Adjust Bottle, Click Bell, Move Pillbottle Pad, Place Bread Basket, and Place Object Scale. Each pair of rows shows the recorded sequence above the prediction, with time progressing from left to right. The leftmost column is the conditioning frame. The video generator uses one denoising step and ground-truth future object masks.
Fig. 10: Video predictions for real-world dynamic grasping. The first three examples show conveyor-belt grasping with a banana, a bottle, and a roll of tape; the last two show rotating-disc grasping with a plush toy and a banana. Each pair of rows compares the recorded sequence with the prediction at corresponding time points, ordered from left to right.
More in this topic

Dynamic Robotic Manipulation

Dynamic robotic manipulation studies how robots act on moving or deformable objects and respond to fast-changing physical interactions.

See all 89 papers

Explore similar work

CardsList
Aug 1, 2026

DynamicWAM: Dual-Path Motion Conditioning for World-Action Models in Dynamic Manipulation

Dynamic manipulation requires robots to infer target motion and respond promptly, yet existing World-Action Models (WAMs) typically condition only on the current frame and execute large backbones synchronously, limiting motion awareness and responsive control in dynamic scenes. We propose DynamicWAM, a compact WAM for dynamic object manipulation with dual-path motion conditioning. DynamicWAM introduces history-flow conditioning, encoding temporally aligned optical-flow frames alongside the current observation through a frozen pretrained video VAE to preserve spatial motion structure, while injecting kinematic descriptors of displacement, duration, velocity, and acceleration into the action expert to provide motion magnitude and timing. The two complementary paths are fused through joint world-action attention. A distilled compact backbone and real-time chunking (RTC)-based asynchronous execution further enable responsive control. On DOMINO, DynamicWAM achieves a 38.2% success rate and a 53.2 manipulation score, outperforming all evaluated baselines. Across 12 real-world tasks spanning linear, circular, and compound target motion, it achieves a 46.7% average success rate, exceeding the strongest baseline by 22.9 percentage points.

Yunfan Lou, Hewen Gao, Xiyu Zhu +6

PKU-PI Lab · National University of Singapore · ePyBot Intelligence +2

World Models for RoboticsGoal-Conditioned Robotic ManipulationRobotic ManipulationDynamic Robotic Manipulation
AskPDF
May 31, 2026

τ0τ_0-WM: A Unified Video-Action World Model for Robotic Manipulation

Robotic manipulation requires models that generate executable actions while anticipating and evaluating their future consequences before physical execution. We present τ0τ_0-World Model (τ0τ_0-WM), a unified video-action world model that integrates policy learning, video prediction, and action evaluation within a single future-predictive framework. Built on a shared video diffusion backbone, τ0τ_0-WM provides two complementary interfaces. First, a video action model jointly predicts future visual latents and continuous action chunks from multi-view observations, language instructions, and robot state. Second, an action-conditioned video simulator rolls out candidate action chunks into multi-view futures and predicts dense task-progress scores. The model is trained on approximately 27,30027{,}300 hours of real-robot teleoperation, UMI-style interaction, egocentric human videos, and rollout or failure trajectories using modality-specific supervision masks. At inference time, τ0τ_0-WM uses test-time computation to sample action candidates, rank them with re-denoising consistency, and invoke simulator-based rectification for low-quality candidates. On challenging long-horizon and fine-grained robotic manipulation tasks, τ0τ_0-WM shows superior performance over other relevant baselines.

Pengfei Zhou, Shengcong Chen, Di Chen +17

1Shanghai Innovation Institute · 2AGIBOT Finch

Action-Conditioned Video GenerationLong-Horizon Robotic ManipulationRobot Skill LearningWorld Models for RoboticsWorld ModelsWorld Model-Based Planning+2
AskPDF
Aug 22, 2026

DELE-w0.5: Inferring Action from Future Latent State for Robotic Manipulation

World-Action Models (WAMs) build robot control on video-generation backbones, which jointly predict dense future visual trajectories and robot actions. We argue that video generation is an unnecessary intermediate objective for world-action modeling. For robotic manipulation, the goal of a world model is not to reproduce how the world looks at every intermediate moment, but to predict the state that the world will reach after an action is executed. The intermediate frames only describe the visual transition between physical states, which consumes substantial model capacity and computation, but do not directly specify the physical outcome that the robot action is intended to produce. In this paper, we propose DELE-w0.5, which infers robot actions from predicted future states without relying on video generation. Concretely, DELE-w0.5 infers the action sequence from its corresponding compact future latent state. The future latent state captures the action-relevant physical outcome of robot interaction and serves as an explicit bridge between world modeling and action generation. The core design principle of DELE-w0.5 is to model how the physical world changes under robot actions, rather than how its visual appearance evolves frame by frame. This formulation removes the high-dimensional visual redundancy introduced by dense video representations, and it therefore enables cheaper training and low-latency inference. Across 640 real-robot trials on four long-horizon manipulation tasks, our DELE-w0.5 achieves the best performance among all compared policies, attaining 62.5% overall full-task success and 81.3% macro ordered-stage progress. It outperforms the strongest baseline by 32.5 percentage points in full-task success and 20.1 percentage points in macro progress.

Fenghao Lei, Zhixiong Huang, Long Yang +5

DeepLeap Research, Shenzhen, China

Long-Horizon Robotic ManipulationWorld Models for RoboticsRobotic ManipulationWorld Action Models
AskPDF